# Architecture — httpjet Documentation > The httpjet workspace: 19 hj-* library crates under one binary, the frozen hj-core contract, the io_uring transport bridge, the request pipeline, and configuration generations. - **Canonical URL:** https://httpjet.net/docs/architecture - **Source Repository:** https://github.com/faratech/httpjet --- # Architecture httpjet is a Cargo workspace: **one binary crate** (`httpjet`) over **nineteen `hj-*` library crates**. The graph is acyclic, `hj-config` is the root, and `hj-core` is the small frozen seam every handler programs against. Two crates (`hj-acme`, `hj-ocsp`) only link in behind build features. Roughly 3,000 test functions span the workspace (3,019 green at v0.1.5). ## Workspace shape Responsibilities are split so the contract a handler depends on stays tiny while transports, protocols, gateways and the cache evolve behind it. Three libraries are completely isolated — `hj-log`, `hj-geo` and `hj-cache` take no `hj-*` dependency — while `hj-rewrite` depends only on `hj-compress` so `.htaccess` and XML expiry directives share one `A/M` time grammar. `hj-pagecache` is transport-agnostic and builds on `hj-cache`'s sharded LRU; its request-aware glue lives in the binary. Browse workspace on GitHub ↗ 1 binary + 19 libraries Rust 1.99.1 · edition 2024 · v0.1.5 Acyclic; hj-config at the root ≈3,000 test functions Vendored forks: monoio, monoio-rustls (real writev), dashmap, pprof ## Crate dependency graph `hj-config` parses the LiteSpeed XML tree into one normalized model; `hj-core` re-exports it so every downstream crate imports the configuration model *through* core — one import surface, never inverted. The binary is the only crate that sees everything. - hj-config XML tree → model · parse_bundle hj-core the frozen contract · ReqCtx · Router · budgets handlers & transports (depend on hj-core) hj-tls hj-h2 hj-lsapi hj-fastcgi hj-static hj-compress hj-proxy hj-acl hj-extension hj-http (ServeConfig) hj-pagecache (uses hj-cache) hj-cache sharded byte-LRU hj-acme · hj-ocsp --features acme / ocsp isolated (no hj-* deps) hj-rewrite hj-log hj-geo hj-cache httpjet uring transport · pipeline · lscache glue · admin config_transaction · resource_generation · waf · otel hj-config → hj-core → handlers/transports → the httpjet binary. Isolated crates converge at the binary; `hj-geo` feeds `hj-acl`'s `GeoRules`; the two feature-gated crates hang off `hj-tls`'s resolver handles. ## Crate map Crate Role hj-core Protocol-agnostic `Request`/`Response`/`Body`, `ReqCtx`, the vhost `Router`, shared budgets (`BodyBufferBudget` leases, `BandwidthThrottle`, `ResponseCompletion`, `body_deadline`), and `recycle` (`TieredPool` transient buffer and task pooling) — the seam every handler programs against. hj-config LiteSpeed XML → one normalized model (`$VAR` substitution, OLS-faithful value semantics, fail-closed defaults). `parse_bundle` is the in-memory variant used by the transactional API . hj-tls One rustls `ServerConfig` for TCP + QUIC: SNI resolver (no cert-name-match, like OLS), dual ECDSA/RSA chains, mTLS verifier with CRLs, generation-bound staple slots, certificate reload handles. hj-acme feature RFC 8555 HTTP-01/DNS-01 issuance and renewal (instant-acme), the 0700 `PrivateStore` with atomic snapshots, challenge registry, DNS webhook adapter with durable cleanup journal. hj-ocsp feature Bounded OCSP fetch + OpenSSL 3.x signature/issuer/freshness validation; status lifecycle with revocation enforcement. hj-http The shared `ServeConfig` connection-tuning struct derived from ` `. Its old hyper-H1/quinn-H3 adapters were removed when io_uring became the only transport. hj-h2 The **native HTTP/2 server**: one task per connection, send flow control, streaming bodies, one vectored write per wake, RFC 9113 validation, and the zero-allocation HPACK codec (`hj_h2::hpack`). hj-lsapi Native LSAPI wire codec (vendored `lsapidef.h` constants), UDS client pool with generation epochs, the `lsphp` supervisor and subreaper, the CGI env builder, and the fail-closed jail. hj-fastcgi Bounded FastCGI record codec, connection pool (poison-by-default), strict CGI response parser and streaming handler for explicit `fcgi`/`fastcgi` processors. hj-static Static files: `openat2(RESOLVE_BENEATH)` resolution, `fileETag` ETags, conditional requests, single + multipart ranges, precompressed `.br`/`.gz` siblings, `Body::File` with pinned inodes for sendfile. hj-cache The shared `ShardedCache` byte-weighted LRU primitive and `CacheCaps` sizing (the former mmap file cache was removed — it had no consumers). hj-pagecache The **origin page store**: sharded byte-LRU index with W-TinyLFU admission, keys + vary + tags + TTL/stale windows, the LSCache header classifier, the HJPC file tier with boot scan, dictionary recompression, single teardown path. hj-compress gzip/brotli/zstd response encoders, server-priority negotiation, streaming `CompressStream`, per-thread zstd contexts, and the per-vhost `PageDictRegistry` the page cache uses for internal dedup. hj-rewrite Apache mod_rewrite / `.htaccess`: dense-DFA prefilter, linear-first regex with `fancy-regex` fallback, access control, scoped Basic auth, header ops, LSCache directives, memo-eligibility classification. hj-proxy Reverse proxy (h1, `h2://`, `h2s://`), upstream groups with weighted selection and active health checks, passive circuit breaker, WebSocket relay, SSE streaming, upstream mTLS. hj-acl Allow/deny ACLs with trusted-proxy (`T`) marking, real-IP resolution and anti-spoofing, the sharded per-IP throttle, and `GeoRules` (geo/ASN resolved to CIDR sets at state build). hj-geo Offline GeoIP/ASN sources: the CidrList text format and binary-search interval sets. No runtime fetches. hj-log Non-blocking access + error logging: common/combined/JSON access lines, text or JSON error events with redaction, size rolling with day pruning, the syslog tap. hj-extension `ExtensionRegistry` for compile-time pre-handlers and response transforms (the shipped registry is empty). httpjet The binary: listener planning and adoption, the `uring/` transport, `pipeline/`, `lscache/` glue, the fast-path memo, metrics, admin listeners, config transactions and resource generations, WAF client, OTel, PHP slow log, memory trim. ## Transport paths The pure io_uring transport is the only serving transport. There is no runtime switch and no `URING=0` rollback build; reverting is a rebuild from an older revision. One monoio runtime runs per inherited or self-bound `SO_REUSEPORT` listener fd, pinned to its core when the worker count equals the CPU count (`--no-core-pinning` is the kill switch). > [Diagram: transport] One transport, three protocols, one pipeline. Cheap work stays on the monoio core: the H1 codec, the native h2 task, the QUIC driver, and the on-core fast path (finished-response memo, static files, page-cache hits). Anything else crosses uring::bridge into the tokio runtime that owns ServerState, the LSAPI pool, the page cache and reload state, then returns to the monoio core for framing and writes. Bodies ≤ 256 KiB are buffered; larger ones stream on H1, H2 and H3 alike. Ring setup is tunable but deliberately plain: `--uring-ring legacy` (default) uses flagless rings because the measured `coop` and `defer` variants were noise or a ~2% loss on the virtualized production guest; ring size stays at the default because ring memory is memlock-charged. Multishot accept is on by default (`--no-multishot-accept` reverts to one SQE per connection). ## The frozen contract — hj-core This is the seam that lets handlers compose without knowing the transport. Changing these signatures means updating every dependent, so they are treated as frozen: **hj-core** · the seam ``` enum Body { Empty, Full(Bytes), Stream(BoxBody), File(FileBody) } type Request = http::Request ; type Response = http::Response ; // every terminal handler (StaticFiles, Lsapi, FastCgi, ProxyHandler) trait Handler { async fn handle(&self, &mut ReqCtx, Request) -> Result ; } // every post-handler munger (promote → expires → sub_filter → compress → deny-CDN-cache → Alt-Svc) trait ResponseTransform { async fn transform(&self, &ReqCtx, &mut Response); } ``` `FileBody` may carry an open descriptor that pins the selected inode across a concurrent cache version swap, recompression or unlink — that is what lets sendfile and the file tier coexist safely. `ReqCtx` carries the per-request state: `server`/`vhost` (Arcs of the pinned generation), `peer_ip`/`client_ip` (post-forwarding, or `peer_unix`), `is_tls`, `protocol`, `trusted_proxy`, and an `env` vec that holds rewrite `[E=]` sets *and* CGI overrides. `Router::resolve(listener, key)` maps SNI/Host to a vhost. i Extending without touching handle() New terminal handlers `impl Handler` and register in `pipeline/mod.rs::dispatch()`; new response munging `impl ResponseTransform` and is pushed into `state.rs::build_transforms`. Out-of-tree code uses the same traits through the compile-time extension registry . ## The request pipeline `pipeline::handle()` is the integration point: vhost resolution → mTLS trust boundary → client-IP/ACL/geo/throttle → optional WAF inspection → extension pre-handlers → `dispatch()` → response transforms → access log, metrics and spans. `dispatch()` mirrors OpenLiteSpeed's ordering with the page-cache seams inserted once rewrite and access state are known: The terminals are intentionally *not* a uniform matcher registry. Their predicates differ (request headers for WebSocket, URI prefix for proxy contexts, script-path split for suffix routing), their fall-through rules differ, and their post-processing differs, so an explicit `if/return` chain keeps the security-relevant ordering legible. Two fall-throughs matter: a proxy context that cannot resolve its ext-processor falls through to static, while a script whose handler has no pool **hard-fails 503 and must never fall through** — falling through would serve the raw PHP source. ! Load-bearing detail A rewrite to `index.php` keeps the **original** `REQUEST_URI`: the resolved script is pinned separately via the `LsapiScript` request extension, with `SCRIPT_NAME`/`QUERY_STRING` pushed into `ctx.env`. That is what makes XenForo-style front-controller routing correct — and why the page cache must key by the *original* URI, never the rewrite-resolved path. The cache seams are inert unless `serve` runs with `--page-cache`: `lscache::cache_lookup` runs after rewrite/ACL but before any backend (a hit short-circuits dispatch), and `lscache::cache_store` runs after `finalize_response` at each terminal return. Both key off a single `CacheCtx` built once per dispatch. ## Module organization Path Contents `crates/httpjet/src/uring/` The transport: `mod.rs` (accept loops, sniff, TLS), `codec.rs` (H1), `h3.rs` (QUIC driver), `bridge.rs`, `ktls.rs` + `ktls_policy.rs`, `proxy_protocol.rs`, `request_body.rs` (decompression), `worker_group.rs` (listener generations). `crates/httpjet/src/pipeline/` `mod.rs` (`handle`/`dispatch`/`finalize_response`), `rewrite_glue`, `proxy_glue` (incl. `ProxyHandler`), `htaccess_apply`, `suffix_routing`, `scoped_auth`, `fast_memo`, `response_util`, and the in-file e2e/security regression tests. `crates/httpjet/src/lscache/` `mod.rs` (`CacheCtx`, `cache_lookup`/`cache_store`), `hit.rs` (hit-response construction + precompress), `singleflight.rs`. `crates/httpjet/src/{admin*,config_transaction,resource_generation,serving_generation,listener_plan,tcp_candidate}.rs` The read-only status API, the authenticated write API, the publication coordinator, and the listener/resource generation model behind SIGHUP and transactional reloads. `crates/httpjet/src/{acme_runtime,ocsp_runtime,otel,waf,extensions,metrics,telemetry,phpslow,memtrim}.rs` Feature runtimes and observability. `crates/hj-h2/src/server/` `state`/`recv`/`send` + `serve()`; HPACK is the `hpack` submodule. `crates/hj-rewrite/src/` `rules/{parse,eval}`, `htaccess/{parse,mod_access,auth_scope,cache,php,scope_index}`, `auth.rs` (htpasswd). `crates/hj-lsapi/src/` `proto/{frame,builder}`, `pool.rs`, `supervisor`, and the security-critical `jail`. ## Configuration generations Live state is published as immutable *generations* behind an `ArcSwap`. A SIGHUP or an API transaction builds a complete candidate (config, rewrite caches, health schedulers) with background work inactive, validates it, then publishes it atomically under one coordinator; requests already in flight finish on the generation they pinned at acceptance. > [Diagram: generations] Candidate → published → retired. A rejected candidate never touches the serving generation. Trust changes (TLS material, mTLS policy, PROXY protocol) retire the old listener group and drain its connections under their pinned policy; boot-owned resources (listener topology, PHP pools, cache stores, privileges) are restart_required. The full ownership inventory and the transactional replacement sequence are on Admin API & reload . ## Code conventions **Formatting:** rustfmt-clean (default settings, edition 2024); `cargo fmt --all --check` is a CI gate. - **Errors:** `thiserror` types at system boundaries; internal invariant violations panic with a message rather than pretending to recover. - **unsafe:** rare, and never in HPACK. The real sites are `hj-h2`'s pin projection and writev/iovec plumbing, the validated fixed-size mmaps in `hj-pagecache/diskstore.rs` and `hj-lsapi/pool.rs`, and small libc shims — each with its safety invariant documented above the block. - **Async:** tokio tasks for independent work, `FuturesUnordered` for multiplexing with coordinated shutdown, no per-stream spawn on hot paths. - **Hashing:** `rustc-hash` only for internal maps keyed by trusted values (h2 stream ids); anything keyed by untrusted input keeps SipHash. - **Allocator & pooling:** mimalloc globally, with a background `mi_collect` trim that only runs above an RSS threshold (`--mimalloc-trim-*`). Because production sets `MIMALLOC_PURGE_DELAY=0` so freed pages return to the OS immediately, transient per-request buffers and task boxes are pooled in `hj_core::recycle` (`Recycled ` and `BODY_BUFS` `TieredPool >`) and per-core TLS write pools to keep steady-state page faults at zero. > [Diagram: recycle-pools] Zero-fault buffer and task recycling (v0.1.2–v0.1.5). Under MIMALLOC_PURGE_DELAY=0, unpooled transient allocations in sparse size classes fault back in on every request. hj_core::recycle pools H2 stream futures, bridge futures, tiered body buffers (frozen into ref-counted Bytes via Bytes::from_owner and returned when the last slice drops), LSAPI read/BEGIN buffers, and per-core 64 KiB TLS record buffers.