# httpjet — Complete Technical Documentation (v0.1.5) Canonical URL: https://httpjet.net/docs/ Source Repository: https://github.com/faratech/httpjet ============================================================================== PAGE: Overview (https://httpjet.net/docs/) SUMMARY: httpjet is a from-scratch Rust web server and drop-in LiteSpeed replacement: the same XML configuration, native HTTP/2 and HTTP/3, LSAPI PHP, .htaccess rewriting, reverse proxy and FastCGI gateways, and an opt-in origin page cache. ============================================================================== httpjet · documentation · v0.1.5 # httpjet A from-scratch **Rust web server** that is a **drop-in LiteSpeed replacement**: it reads the same `conf/httpd_config.xml` tree, speaks native LSAPI to `lsphp`, evaluates the same `.htaccess` files, and adds native HTTP/2 and HTTP/3, reverse-proxy and FastCGI gateways, an opt-in LSCache-protocol page cache, optional ACME and OCSP, and a loopback admin API — all on one io_uring transport. resolve SNI / Host → vhost acl identity · throttle · geo · WAF rewrite .htaccess · auth cache lookup → hit? dispatch ws · proxy · php · fcgi · static finalize headers · compress store cache write one pipeline per request, whatever the wire protocol — every page tells you where it sits on this rail GitHub · faratech/httpjet ↗ Drop-in LiteSpeed XML config HTTP/1.1 · native HTTP/2 · HTTP/3 Native LSAPI → lsphp · FastCGI Apache .htaccess · Basic auth Opt-in origin page cache ACME · OCSP · OTel (build features) ## What httpjet is httpjet is a complete origin server on a memory-safe Rust core, built so that an existing LiteSpeed Web Server (LSWS 6.3.5 compatibility target) or OpenLiteSpeed deployment becomes an httpjet deployment by pointing `--root` at the tree it already has. Compatibility is the feature; everything httpjet adds on top is an explicit opt-in that LiteSpeed simply ignores. ### Compatible surface - LiteSpeed XML: server, tuning, listeners, vhosts, contexts, `mime.properties` - Native LSAPI wire protocol to any LSAPI-capable `lsphp` - mod_rewrite , access control, Basic auth, `Header`, `ErrorDocument`, LSCache directives - LSCache protocol (`X-LiteSpeed-Cache-Control` / `-Tag` / `-Vary` / `-Purge`) - LiteSpeed `fileETag=28` ETags, `.html`→PHP dispatch order ### httpjet-native (opt-in) - Per-client throttle, GeoIP/ASN rules, PROXY protocol , Unix-socket listeners - Weighted upstream groups with active health checks; `h2://` / `h2s://` upstreams; FastCGI - Per-context body/timeout/bandwidth overrides, `sub_filter`, request decompression - ACME HTTP-01/DNS-01, OCSP stapling , kTLS/sendfile, OpenTelemetry - Loopback status + transactional config API , WAF sidecar , compile-time extensions i Where it runs httpjet serves WindowsForum.com on an x86_64 origin and MoonTimeNow.com + ThisDayInHistory.ai on an active ARM64 origin behind Cloudflare, over the pure io_uring transport with the page cache's file tier live. It is early-stage software (semantic version **0.1.5**, Rust 1.99.1+, edition 2024, GPL-3.0-only) built around these production deployments — test on alternate ports before trusting it elsewhere. See Capabilities & limits for the honest matrix. ## How the pieces fit Every wire protocol lands in the same request pipeline. Click a node to jump to its page. > [Diagram: pipeline] The request pipeline. pipeline::handle() resolves the vhost and the client's identity, then dispatch() mirrors OpenLiteSpeed's ordering with the WAF, extension, page-cache lookup and store seams inserted at fixed points. Each terminal finishes through finalize_response before the transforms run. ## By the numbers Headline figures from the checked-in benchmark record — every number names its regime (see Performance for methodology and the raw CSVs). ~908 k rps h2 static, dedicated-host loopback — vs OpenLiteSpeed's ~793k (2026-08-27) 44.8 k rps PHP/LSAPI over h1, PGO loopback — vs LiteSpeed's ~38.5k (2026-06) 101 × cache hit vs full PHP render of the same page (19,912 vs 196 rps) 146 /146 h2spec conformance on the native HTTP/2 stack, over TLS, three consecutive runs 0.2 % httpjet's share of production request latency — ~138 µs/req; PHP owns the rest 3,000 + test functions across 20 crates; full workspace suite green at v0.1.5 ## Project status | Aspect | State (2026-10-24) | | Version | **0.1.5** — semantic versioning adopted 2026-09-17; MSRV Rust 1.99.1, edition 2024; dependencies current (0.1.x sweep: tokio 1.54, rustls 0.23.46, rustls-webpki 0.103.16 + webpki-roots 1.0.10, fancy-regex 0.20, parking_lot 0.13, zstd 0.14.3). | | Configuration | LiteSpeed-compatible XML is the authority; httpjet-native directives are opt-in add-ons. `check --strict` is the deploy gate. | | Transport | Pure io_uring (monoio) for H1/H2/H3 — the *only* transport; there is no epoll fallback and no Windows/macOS port. | | PHP | Native LSAPI; the operator supplies `lsphp`. Production runs the persistent external pool with zero-downtime generation reloads. | | Page cache | Off by default; LSCache-protocol opt-in, private tier, per-URL cookie vary, tmpfs file tier, per-vhost dictionaries, generation-scoped namespaces. Live in production. | | Optional features | `acme`, `ocsp`, `otel`, `ktls`, `profiling` — build features that still need explicit runtime opt-in. | | Security posture | Whole-codebase review rounds through 2026-10-23 landed, plus the October bug-bash campaign (rounds 1–16 across 0.0.6–0.1.1; see audit posture ); fail-closed defaults throughout. | | Release | Source builds on github.com/faratech/httpjet , generic systemd units, and a hardened OCI example. Signed public packages and a supported-platform matrix are still release work. | ## Read next The docs follow the request: how the server is structured, the protocols it speaks, how it runs PHP and proxies, who gets in, the page cache, and how it is configured, operated and measured. Quickstart → Build, validate a tree with `check --strict`, serve on loopback. Architecture → 20 crates, the frozen `hj-core` seam, the io_uring bridge, generations. Capabilities → Supported, optional, partial, unsupported — and the fixed bounds. Page cache → Keys, tiers, admission, purge, the file tier, capsules. CLI reference → Every subcommand and flag, filterable. Migrating → The validation-first path off LiteSpeed. GitHub ↗ Browse the source tree, benchmark harnesses, and issues at `faratech/httpjet`. ============================================================================== PAGE: Quickstart (https://httpjet.net/docs/quickstart) SUMMARY: Build httpjet from source, validate a LiteSpeed-compatible configuration tree with check --strict, and serve it on loopback in ten minutes. ============================================================================== # Quickstart From a source checkout to a running origin in a few minutes: build the binary, validate a LiteSpeed-compatible tree with `check --strict`, serve it on loopback, and verify. httpjet reads the same `conf/httpd_config.xml` layout LiteSpeed uses, so an existing tree needs no migration step — but validate it first. ## Prerequisites | Requirement | Why | | Linux with **io_uring** | The monoio/io_uring transport is the only transport. Container runtimes must allow `io_uring_setup`/`io_uring_enter`/`io_uring_register` (see OCI ). | | **Rust 1.99.1+**, edition 2024 | The workspace MSRV. A C toolchain is needed for the aws-lc-rs crypto backend and mimalloc. | | OpenSSL 3.x headers + pkg-config | Only for `--features ocsp`. | | An LSAPI-capable `lsphp` | Only for PHP. Static, proxy and FastCGI deployments need no PHP and no LiteSpeed installation. | i Release status No signed binary packages are published yet; the source build below is the install path. The repository's PGO and deploy scripts are tailored to one production deployment — do not treat them as a general installer. ## 1 · Build the binary Clone the public repository from github.com/faratech/httpjet and compile the `httpjet` binary crate in release mode: **build** · cargo ``` git clone https://github.com/faratech/httpjet.git cd httpjet # release build of just the binary crate (the workspace is large — always scope with -p) cargo build --release -p httpjet # → target/release/httpjet # optional build features (each still needs explicit runtime opt-in): cargo build --release -p httpjet --features acme,ocsp,otel ``` A debug build (`cargo build -p httpjet`) compiles faster but runs ~5× slower — fine for iteration, meaningless for any measurement. For reproducible results use `--locked`. ## 2 · Validate a configuration tree httpjet is configured by a *server root* containing `conf/httpd_config.xml`, optional `conf/mime.properties`, and the per-vhost XML files it references. The checked-in `examples/litespeed/` tree is a minimal working example: one plain listener on `127.0.0.1:8080` and one static vhost (`example.test`) with scripting and symlinks off. **check** · config lint ``` # parse the tree and print a summary (check is also the default subcommand) target/release/httpjet --root "$PWD/examples/litespeed" check # strict: lint topology and exit non-zero on findings — the deploy gate target/release/httpjet --root "$PWD/examples/litespeed" check --strict ``` `check --strict` hard-errors on a docroot shared by two vhosts, a vhost whose docroot is missing, and a vhost with no exact `vhostMap` domain; it warns on an apex mapped without its `www.` twin, TLS over a Unix socket, PROXY protocol on a public bind, geo rules without a database, ignored LiteSpeed anti-abuse elements, generic CGI contexts, and every other directive that would parse but do nothing. It also prints each vhost's fast-memo verdict (`fast_memo: eligible [vary: …]`) and a `vhost-map-fingerprint` for comparing check output over time. Always use an absolute `--root`. ! Parsed ≠ equivalent httpjet accepts LiteSpeed XML leniently so real trees load. A warning means the directive is *not* enforced — review every one before assuming parity. The capability matrix lists what is implemented. ## 3 · Serve on loopback The XML holds persistent policy; `serve` flags hold operational overrides (addresses, workers, PHP wiring, cache budgets). The example is HTTP-only, PHP-off and cache-off, so it is safe next to an existing server. **serve** · development ``` target/release/httpjet --root "$PWD/examples/litespeed" serve \ --http-addr 127.0.0.1:8080 --https-addr "" \ --workers 1 --no-php ``` To try TLS and HTTP/3, add a secure listener with `keyFile`/`certFile` to the XML, set ` 1 `, and pass `--https-addr 127.0.0.1:8443`; the same port must be reachable over TCP (h1/h2) and UDP (h3). See TLS, ACME & OCSP . ! Keep test state isolated Use alternate HTTP/HTTPS/metrics/admin ports, a separate `--php-socket` if PHP is on, low `--php-children`, and a separate `--page-cache-store-path`. Never point a test instance at a production LSAPI socket — two supervisors on one socket corrupt each other's worker accounting. Test pools default their control and generation files *beside* their own socket for exactly this reason. ## 4 · Verify **verify** · curl ``` printf 'hello from httpjet\n' > examples/litespeed/example/public/index.html curl -i -H 'Host: example.test' http://127.0.0.1:8080/ # h2c prior knowledge on the same plain port curl -i --http2-prior-knowledge -H 'Host: example.test' http://127.0.0.1:8080/ # Prometheus metrics (loopback-only by design) curl -s http://127.0.0.1:9090/metrics | grep -E '^httpjet_(requests_total|active_connections)' ``` Expect a LiteSpeed-style ETag (`"size-mtime-inode;;;"`), `Vary: Accept-Encoding` on compressible bodies, and — once a listener is secure and QUIC is on — an `Alt-Svc` header advertising h3. ## 5 · Add PHP (optional) With an LSAPI-capable `lsphp` configured in the XML (` ` or an `lsapi` `extProcessor` plus a `scriptHandler`), drop `--no-php` and let httpjet spawn its own pool on a private socket. For zero-downtime deploys, run the pool as a separate process instead: **two processes** · external pool ``` # terminal 1 — the persistent PHP pool httpjet --root /abs/root lsphp --php-socket /run/httpjet-test/php.sock --php-children 4 # terminal 2 — the web tier dials it and never spawns lsphp itself httpjet --root /abs/root serve --http-addr 127.0.0.1:8080 --https-addr "" \ --lsphp-external /run/httpjet-test/php.sock \ --lsphp-generation-file /run/httpjet-test/php.sock.generation ``` How the pool, the generation file and `lsphp-reload` fit together is on LSAPI & PHP . ## Where to go next - Configuration — the XML reference, including the httpjet-native directives. - CLI reference — every `serve` flag, grouped and filterable. - Operations — systemd socket activation, the PHP pool, deploy gates and rollback. - Page cache — enabling `--page-cache` safely. - Migrating from LiteSpeed — the validation-first path for a real tree. - github.com/faratech/httpjet — browse the Rust workspace source, example configs, and issue tracker on GitHub. ============================================================================== PAGE: Architecture (https://httpjet.net/docs/architecture) SUMMARY: The httpjet workspace: 19 hj-* library crates under one binary, the frozen hj-core contract, the io_uring transport bridge, the request pipeline, and configuration generations. ============================================================================== # Architecture httpjet is a Cargo workspace: **one binary crate** (`httpjet`) over **nineteen `hj-*` library crates**. The graph is acyclic, `hj-config` is the root, and `hj-core` is the small frozen seam every handler programs against. Two crates (`hj-acme`, `hj-ocsp`) only link in behind build features. Roughly 3,000 test functions span the workspace (3,019 green at v0.1.5). ## Workspace shape Responsibilities are split so the contract a handler depends on stays tiny while transports, protocols, gateways and the cache evolve behind it. Three libraries are completely isolated — `hj-log`, `hj-geo` and `hj-cache` take no `hj-*` dependency — while `hj-rewrite` depends only on `hj-compress` so `.htaccess` and XML expiry directives share one `A/M` time grammar. `hj-pagecache` is transport-agnostic and builds on `hj-cache`'s sharded LRU; its request-aware glue lives in the binary. Browse workspace on GitHub ↗ 1 binary + 19 libraries Rust 1.99.1 · edition 2024 · v0.1.5 Acyclic; hj-config at the root ≈3,000 test functions Vendored forks: monoio, monoio-rustls (real writev), dashmap, pprof ## Crate dependency graph `hj-config` parses the LiteSpeed XML tree into one normalized model; `hj-core` re-exports it so every downstream crate imports the configuration model *through* core — one import surface, never inverted. The binary is the only crate that sees everything. - hj-config XML tree → model · parse_bundle hj-core the frozen contract · ReqCtx · Router · budgets handlers & transports (depend on hj-core) hj-tls hj-h2 hj-lsapi hj-fastcgi hj-static hj-compress hj-proxy hj-acl hj-extension hj-http (ServeConfig) hj-pagecache (uses hj-cache) hj-cache sharded byte-LRU hj-acme · hj-ocsp --features acme / ocsp isolated (no hj-* deps) hj-rewrite hj-log hj-geo hj-cache httpjet uring transport · pipeline · lscache glue · admin config_transaction · resource_generation · waf · otel hj-config → hj-core → handlers/transports → the httpjet binary. Isolated crates converge at the binary; `hj-geo` feeds `hj-acl`'s `GeoRules`; the two feature-gated crates hang off `hj-tls`'s resolver handles. ## Crate map Crate Role hj-core Protocol-agnostic `Request`/`Response`/`Body`, `ReqCtx`, the vhost `Router`, shared budgets (`BodyBufferBudget` leases, `BandwidthThrottle`, `ResponseCompletion`, `body_deadline`), and `recycle` (`TieredPool` transient buffer and task pooling) — the seam every handler programs against. hj-config LiteSpeed XML → one normalized model (`$VAR` substitution, OLS-faithful value semantics, fail-closed defaults). `parse_bundle` is the in-memory variant used by the transactional API . hj-tls One rustls `ServerConfig` for TCP + QUIC: SNI resolver (no cert-name-match, like OLS), dual ECDSA/RSA chains, mTLS verifier with CRLs, generation-bound staple slots, certificate reload handles. hj-acme feature RFC 8555 HTTP-01/DNS-01 issuance and renewal (instant-acme), the 0700 `PrivateStore` with atomic snapshots, challenge registry, DNS webhook adapter with durable cleanup journal. hj-ocsp feature Bounded OCSP fetch + OpenSSL 3.x signature/issuer/freshness validation; status lifecycle with revocation enforcement. hj-http The shared `ServeConfig` connection-tuning struct derived from ` `. Its old hyper-H1/quinn-H3 adapters were removed when io_uring became the only transport. hj-h2 The **native HTTP/2 server**: one task per connection, send flow control, streaming bodies, one vectored write per wake, RFC 9113 validation, and the zero-allocation HPACK codec (`hj_h2::hpack`). hj-lsapi Native LSAPI wire codec (vendored `lsapidef.h` constants), UDS client pool with generation epochs, the `lsphp` supervisor and subreaper, the CGI env builder, and the fail-closed jail. hj-fastcgi Bounded FastCGI record codec, connection pool (poison-by-default), strict CGI response parser and streaming handler for explicit `fcgi`/`fastcgi` processors. hj-static Static files: `openat2(RESOLVE_BENEATH)` resolution, `fileETag` ETags, conditional requests, single + multipart ranges, precompressed `.br`/`.gz` siblings, `Body::File` with pinned inodes for sendfile. hj-cache The shared `ShardedCache` byte-weighted LRU primitive and `CacheCaps` sizing (the former mmap file cache was removed — it had no consumers). hj-pagecache The **origin page store**: sharded byte-LRU index with W-TinyLFU admission, keys + vary + tags + TTL/stale windows, the LSCache header classifier, the HJPC file tier with boot scan, dictionary recompression, single teardown path. hj-compress gzip/brotli/zstd response encoders, server-priority negotiation, streaming `CompressStream`, per-thread zstd contexts, and the per-vhost `PageDictRegistry` the page cache uses for internal dedup. hj-rewrite Apache mod_rewrite / `.htaccess`: dense-DFA prefilter, linear-first regex with `fancy-regex` fallback, access control, scoped Basic auth, header ops, LSCache directives, memo-eligibility classification. hj-proxy Reverse proxy (h1, `h2://`, `h2s://`), upstream groups with weighted selection and active health checks, passive circuit breaker, WebSocket relay, SSE streaming, upstream mTLS. hj-acl Allow/deny ACLs with trusted-proxy (`T`) marking, real-IP resolution and anti-spoofing, the sharded per-IP throttle, and `GeoRules` (geo/ASN resolved to CIDR sets at state build). hj-geo Offline GeoIP/ASN sources: the CidrList text format and binary-search interval sets. No runtime fetches. hj-log Non-blocking access + error logging: common/combined/JSON access lines, text or JSON error events with redaction, size rolling with day pruning, the syslog tap. hj-extension `ExtensionRegistry` for compile-time pre-handlers and response transforms (the shipped registry is empty). httpjet The binary: listener planning and adoption, the `uring/` transport, `pipeline/`, `lscache/` glue, the fast-path memo, metrics, admin listeners, config transactions and resource generations, WAF client, OTel, PHP slow log, memory trim. ## Transport paths The pure io_uring transport is the only serving transport. There is no runtime switch and no `URING=0` rollback build; reverting is a rebuild from an older revision. One monoio runtime runs per inherited or self-bound `SO_REUSEPORT` listener fd, pinned to its core when the worker count equals the CPU count (`--no-core-pinning` is the kill switch). > [Diagram: transport] One transport, three protocols, one pipeline. Cheap work stays on the monoio core: the H1 codec, the native h2 task, the QUIC driver, and the on-core fast path (finished-response memo, static files, page-cache hits). Anything else crosses uring::bridge into the tokio runtime that owns ServerState, the LSAPI pool, the page cache and reload state, then returns to the monoio core for framing and writes. Bodies ≤ 256 KiB are buffered; larger ones stream on H1, H2 and H3 alike. Ring setup is tunable but deliberately plain: `--uring-ring legacy` (default) uses flagless rings because the measured `coop` and `defer` variants were noise or a ~2% loss on the virtualized production guest; ring size stays at the default because ring memory is memlock-charged. Multishot accept is on by default (`--no-multishot-accept` reverts to one SQE per connection). ## The frozen contract — hj-core This is the seam that lets handlers compose without knowing the transport. Changing these signatures means updating every dependent, so they are treated as frozen: **hj-core** · the seam ``` enum Body { Empty, Full(Bytes), Stream(BoxBody), File(FileBody) } type Request = http::Request ; type Response = http::Response ; // every terminal handler (StaticFiles, Lsapi, FastCgi, ProxyHandler) trait Handler { async fn handle(&self, &mut ReqCtx, Request) -> Result ; } // every post-handler munger (promote → expires → sub_filter → compress → deny-CDN-cache → Alt-Svc) trait ResponseTransform { async fn transform(&self, &ReqCtx, &mut Response); } ``` `FileBody` may carry an open descriptor that pins the selected inode across a concurrent cache version swap, recompression or unlink — that is what lets sendfile and the file tier coexist safely. `ReqCtx` carries the per-request state: `server`/`vhost` (Arcs of the pinned generation), `peer_ip`/`client_ip` (post-forwarding, or `peer_unix`), `is_tls`, `protocol`, `trusted_proxy`, and an `env` vec that holds rewrite `[E=]` sets *and* CGI overrides. `Router::resolve(listener, key)` maps SNI/Host to a vhost. i Extending without touching handle() New terminal handlers `impl Handler` and register in `pipeline/mod.rs::dispatch()`; new response munging `impl ResponseTransform` and is pushed into `state.rs::build_transforms`. Out-of-tree code uses the same traits through the compile-time extension registry . ## The request pipeline `pipeline::handle()` is the integration point: vhost resolution → mTLS trust boundary → client-IP/ACL/geo/throttle → optional WAF inspection → extension pre-handlers → `dispatch()` → response transforms → access log, metrics and spans. `dispatch()` mirrors OpenLiteSpeed's ordering with the page-cache seams inserted once rewrite and access state are known: The terminals are intentionally *not* a uniform matcher registry. Their predicates differ (request headers for WebSocket, URI prefix for proxy contexts, script-path split for suffix routing), their fall-through rules differ, and their post-processing differs, so an explicit `if/return` chain keeps the security-relevant ordering legible. Two fall-throughs matter: a proxy context that cannot resolve its ext-processor falls through to static, while a script whose handler has no pool **hard-fails 503 and must never fall through** — falling through would serve the raw PHP source. ! Load-bearing detail A rewrite to `index.php` keeps the **original** `REQUEST_URI`: the resolved script is pinned separately via the `LsapiScript` request extension, with `SCRIPT_NAME`/`QUERY_STRING` pushed into `ctx.env`. That is what makes XenForo-style front-controller routing correct — and why the page cache must key by the *original* URI, never the rewrite-resolved path. The cache seams are inert unless `serve` runs with `--page-cache`: `lscache::cache_lookup` runs after rewrite/ACL but before any backend (a hit short-circuits dispatch), and `lscache::cache_store` runs after `finalize_response` at each terminal return. Both key off a single `CacheCtx` built once per dispatch. ## Module organization Path Contents `crates/httpjet/src/uring/` The transport: `mod.rs` (accept loops, sniff, TLS), `codec.rs` (H1), `h3.rs` (QUIC driver), `bridge.rs`, `ktls.rs` + `ktls_policy.rs`, `proxy_protocol.rs`, `request_body.rs` (decompression), `worker_group.rs` (listener generations). `crates/httpjet/src/pipeline/` `mod.rs` (`handle`/`dispatch`/`finalize_response`), `rewrite_glue`, `proxy_glue` (incl. `ProxyHandler`), `htaccess_apply`, `suffix_routing`, `scoped_auth`, `fast_memo`, `response_util`, and the in-file e2e/security regression tests. `crates/httpjet/src/lscache/` `mod.rs` (`CacheCtx`, `cache_lookup`/`cache_store`), `hit.rs` (hit-response construction + precompress), `singleflight.rs`. `crates/httpjet/src/{admin*,config_transaction,resource_generation,serving_generation,listener_plan,tcp_candidate}.rs` The read-only status API, the authenticated write API, the publication coordinator, and the listener/resource generation model behind SIGHUP and transactional reloads. `crates/httpjet/src/{acme_runtime,ocsp_runtime,otel,waf,extensions,metrics,telemetry,phpslow,memtrim}.rs` Feature runtimes and observability. `crates/hj-h2/src/server/` `state`/`recv`/`send` + `serve()`; HPACK is the `hpack` submodule. `crates/hj-rewrite/src/` `rules/{parse,eval}`, `htaccess/{parse,mod_access,auth_scope,cache,php,scope_index}`, `auth.rs` (htpasswd). `crates/hj-lsapi/src/` `proto/{frame,builder}`, `pool.rs`, `supervisor`, and the security-critical `jail`. ## Configuration generations Live state is published as immutable *generations* behind an `ArcSwap`. A SIGHUP or an API transaction builds a complete candidate (config, rewrite caches, health schedulers) with background work inactive, validates it, then publishes it atomically under one coordinator; requests already in flight finish on the generation they pinned at acceptance. > [Diagram: generations] Candidate → published → retired. A rejected candidate never touches the serving generation. Trust changes (TLS material, mTLS policy, PROXY protocol) retire the old listener group and drain its connections under their pinned policy; boot-owned resources (listener topology, PHP pools, cache stores, privileges) are restart_required. The full ownership inventory and the transactional replacement sequence are on Admin API & reload . ## Code conventions **Formatting:** rustfmt-clean (default settings, edition 2024); `cargo fmt --all --check` is a CI gate. - **Errors:** `thiserror` types at system boundaries; internal invariant violations panic with a message rather than pretending to recover. - **unsafe:** rare, and never in HPACK. The real sites are `hj-h2`'s pin projection and writev/iovec plumbing, the validated fixed-size mmaps in `hj-pagecache/diskstore.rs` and `hj-lsapi/pool.rs`, and small libc shims — each with its safety invariant documented above the block. - **Async:** tokio tasks for independent work, `FuturesUnordered` for multiplexing with coordinated shutdown, no per-stream spawn on hot paths. - **Hashing:** `rustc-hash` only for internal maps keyed by trusted values (h2 stream ids); anything keyed by untrusted input keeps SipHash. - **Allocator & pooling:** mimalloc globally, with a background `mi_collect` trim that only runs above an RSS threshold (`--mimalloc-trim-*`). Because production sets `MIMALLOC_PURGE_DELAY=0` so freed pages return to the OS immediately, transient per-request buffers and task boxes are pooled in `hj_core::recycle` (`Recycled ` and `BODY_BUFS` `TieredPool >`) and per-core TLS write pools to keep steady-state page faults at zero. > [Diagram: recycle-pools] Zero-fault buffer and task recycling (v0.1.2–v0.1.5). Under MIMALLOC_PURGE_DELAY=0, unpooled transient allocations in sparse size classes fault back in on every request. hj_core::recycle pools H2 stream futures, bridge futures, tiered body buffers (frozen into ref-counted Bytes via Bytes::from_owner and returned when the last slice drops), LSAPI read/BEGIN buffers, and per-core 64 KiB TLS record buffers. ============================================================================== PAGE: Capabilities & limits (https://httpjet.net/docs/capabilities) SUMMARY: What httpjet supports today, what is optional behind build features, what is deliberately unsupported, and the fixed operational bounds. ============================================================================== # Capabilities & limits What the current source tree does, what it does only behind a build feature, what is a tested subset, and what is deliberately unsupported. **Supported** means an implemented, tested code path — not directive-for-directive parity with LiteSpeed, NGINX, Apache or Caddy. Optional features stay off unless explicitly enabled at build *and* at runtime. supported implemented + tested optional build feature + runtime opt-in partial tested subset — review the boundary unsupported deliberate, documented ## Capability matrix | Area | Status | Scope and important limits | | HTTP/1.1 | supported | Keep-alive, pipelining, chunked bodies; strict framing; configured header/URL/body/timeout/request-count limits; 103 Early Hints; single + multipart ranges; `CONNECT` answered 405. → | | HTTP/2 | supported | Native TLS h2 and h2c prior knowledge; flow-controlled buffered, file and streaming responses; h2spec 146/146. No server push; no `Upgrade: h2c`. → | | HTTP/3 | supported | QUIC over UDP through the same pipeline; TCP *and* UDP HTTPS ports must both be published. Connection migration disabled. → | | TLS | supported | rustls (aws-lc-rs) TLS 1.2/1.3, SNI, ALPN, dual ECDSA+RSA chains, certificate reload, optional/required client certificates, CRLs. → | | Automatic certificates | optional | HTTP-01 and DNS-01 behind `--features acme` + CLI opt-in. One managed account/identifier set per process; DNS-01 needs an operator-supplied webhook controller. → | | OCSP stapling | optional | `--features ocsp`, OpenSSL 3.x, an explicit responder that covers every issuer. Not for CAs that retired OCSP (Let's Encrypt). → | | kTLS / sendfile | partial | Plain-TCP H1 file bodies use `sendfile(2)` always; a `--features ktls` build adds TLS 1.3 H1 sendfile only when `--ktls=auto` finds a physical NIC with active `tls-hw-tx-offload`. H2/H3/UDS/TLS 1.2 keep userspace paths. → | | Virtual hosts | supported | Listener + exact Host/SNI maps, OLS wildcard domain patterns (`*.example.test` and `example.*` with exact > longest-match precedence), and a bare `*` fallback. | | Static content | supported | Conditional requests, LiteSpeed ETags, ranges, in-memory small-file promotion, precompressed `.br`/`.gz` siblings, traversal-safe opens. → | | Reverse proxy | supported | HTTP/1 upstreams plus explicit `h2://`/`h2s://`; request/response streaming, SSE; upstream mTLS. → | | WebSocket | partial | HTTP/1 upgrade and raw relay. Relay does not support TLS, H2 or Unix-socket upstreams. → | | Upstream groups | supported | Primary-first, smooth weighted round-robin, weighted least-active, passive breaker, optional connect/GET/HEAD probes. No automatic replay of unsafe requests. → | | PHP / LSAPI | supported | Native LSAPI; owned pool or separately supervised external pool with generation tracking and candidate-first reload. → | | FastCGI | supported opt-in | Explicit `fcgi`/`fastcgi` processor + suffix routing over TCP or Unix sockets; the app is externally supervised; bodies buffered. → | | CGI execution, uWSGI | unsupported | Design contracts exist; nothing is routable. A `cgi` handler without an explicit FastCGI processor fails closed (never serves source). → | | Rewrites & `.htaccess` | partial | Common rewrite, access, auth, header (`set`/`append`/`add`/`merge`/`unset`/`setifempty`), expiry (`ExpiresActive`/`ExpiresDefault`/`ExpiresByType`), cache and env directives. `Header edit` and `RequestHeader` are ignored. Run `check --strict`. → | | Compression | supported | Dynamic gzip/brotli/zstd + precompressed siblings. Gzip *request* decoding on by default; brotli/zstd request decoding opt-in and bounded. → | | Origin page cache | supported opt-in | Public/private tiers, vary (including per-URL cookie vary), purge tags, stale policies, single flight, admission, RAM/file tiers, dictionaries, restart scan, generation-scoped keys. A first-class route-level *microcache policy* is specified, not implemented. → | | Access controls | supported | Peer ACLs, trusted-proxy client IP, GeoIP/ASN CIDR rules, per-IP rate limit, body limits, Basic auth (bcrypt, APR1 `$apr1$`, MD5-crypt `$1$` and `{SHA}` htpasswd in constant time; bcrypt cost capped at 14, files confined to the docroot). → | | WAF | external opt-in | Bounded request-inspection sidecar over Unix socket (peer-UID verified) or loopback TCP, fail-closed by default. No embedded ModSecurity/CRS engine. → | | Logs and metrics | supported | Combined or JSON access logs, text or JSON error logs with redaction, syslog tap, Prometheus text metrics, request ids, telemetry TSV, PHP slow log. → | | OpenTelemetry | optional | OTLP traces and metrics behind `--features otel` + `HTTPJET_OTEL=1`. Bounded queues; no payload export. → | | Runtime APIs | opt-in | Loopback read-only status API; separately authenticated transactional config API. Boot-owned resources stay restart-required. → | | Extensions | compile-time only | Trusted Rust crates register read-only pre-handlers and response transforms. No dynamic ABI, no in-process sandbox. → | | Deployment | examples | Generic systemd units (socket activation, boot gate, PHP pool) and a hardened OCI example that still needs an io_uring-permitting runtime. → | ## Deliberately unsupported or deferred - **Generic CGI process execution and uWSGI** are not routable; their fail-closed design gates are documented. FastCGI, LSAPI or HTTP proxying cover application paths. - **HTTP/2 server push** is not implemented — use `Link` preload plus 103 Early Hints. - **HTTP trailers** (inbound h2/h3 trailing HEADERS, chunked trailers) are discarded and not forwarded through gateways. LSAPI has no trailer concept. - **h2c is prior-knowledge only**; the HTTP/1.1 `Upgrade: h2c` handshake was removed by RFC 9113 and is not implemented. - **QUIC connection migration** is disabled: listener steering does not yet preserve connection ownership across an address change. - **No epoll/tokio transport fallback**, no Windows/macOS. Linux io_uring is mandatory. - **No native dynamic-module ABI.** Untrusted inspection belongs behind the bounded sidecar contract. - **LiteSpeed anti-abuse elements** (`lsrecaptcha`, `censorshipControl`, permission masks, `cgiRlimit`) are parsed and warned about, never enforced. - **Legacy crypt and plaintext htpasswd schemes** (traditional DES, `$5$`/`$6$` `crypt()`, and plaintext entries) verify *false* with a parse-time warning — use bcrypt (`htpasswd -B`), APR1 (`$apr1$`), MD5-crypt (`$1$`) or `{SHA}`. - **Cross-node cache peers** (`--cache-peer`, peer fill/purge fan-out) were removed in August 2026; the loopback purge and readiness endpoints remain. - **Multipath QUIC** was assessed and deferred (June 2026). ## Build-time feature gates The default binary includes the serving engine, LSAPI, FastCGI, proxy, cache, logging, metrics, the WAF sidecar client and an empty extension registry. These features add dependencies and require an explicit build — and still nothing happens until the matching runtime switch is set: | Cargo feature | Adds | Runtime opt-in | | `acme` | `hj-acme`: HTTP-01 / DNS-01 issuance and renewal | `--acme-directory … --acme-domains … --acme-store … --acme-accept-terms` | | `ocsp` | `hj-ocsp`: fetch, validate, staple (OpenSSL 3.x) | `--ocsp-responder URL` (+ `--ocsp-required`) | | `otel` | OTLP HTTP/protobuf tracing and metrics exporters | `HTTPJET_OTEL=1` (+ `HTTPJET_OTEL_METRICS=1`) | | `profiling` | Loopback pprof CPU profile endpoint | `--profile-token` policy; loopback only regardless | | `ktls` | Kernel TLS on the io_uring TLS path | `--ktls=auto|on|off`; `auto` requires a physical NIC with `tls-hw-tx-offload` | | `allocount` | Allocation-measurement harness | Diagnostic builds only | ## Operational bounds Request limits come from the LiteSpeed ` ` block and per-context overrides; inspect the effective values with `check --strict` and `serve --help`. Fixed or separately configured bounds worth knowing: | Bound | Value | | Page-cache single object | 100 MiB — larger responses serve uncached | | Page-cache budgets | Independent index (`--page-cache-mem`), file tier (`--page-cache-disk-mem`), hot tier (`--page-cache-hot-mem`), object TTL and stale-window caps; persistent files need an integrity key (`--page-cache-integrity-key`; loaded through a private-file validator — single link, owner root or the serving uid, no group/other bits — with `--page-cache-integrity-required` to refuse booting the file tier without one) | | Bridge buffering | Bodies ≤ 256 KiB buffered, larger streamed (H1/H2/H3) | | Request body budget | Process-wide byte budget with leases; usage, capacity and rejections exported as `httpjet_body_buffer_*` | | Request decompression | 16 MiB decoded ceiling; 17 MiB codec workspace reservation for br/zstd; zstd window ≤ 2 24 | | Multipart ranges | 8 ranges / 8 MiB of part bytes, else a full 200 | | `.htaccess` chain depth | 64 directories, then a deny-all sentinel | | sub_filter buffering | 4 MiB default (`subFilterMaxBody`); beyond it the stream passes through raw | | WAF sidecar | Body ≤ 1 MiB inspected (`--waf-body-max`), total timeout ≤ 30 s, concurrency ≤ 4096, 4 KiB response head, 64 MiB process-wide serialization budget | | Admin APIs | Read: 4 KiB/32 header fields, 4 connections, 5 s deadlines, 1 MiB response. Write: 8 KiB/32 fields, 1 MiB body, ≤128 vhosts, ≤64 XML nesting, 4 connections, one candidate build, 5 s lock / 10 s wait | | Active health probes | Intervals/timeouts 1 s–1 h, rise/fall 1–100, process-wide cap of 16 concurrent probes | | FastCGI | 65,535-byte records; separately bounded PARAMS aggregate, response head, STDERR, body buffering, pool size and deadlines | | ACME | 1–100 identifiers, ≤128 live challenges, 1 MiB snapshots and CA bodies, 120 s per attempt | | OCSP | ≤128 identities, ≤4 concurrent fetches, 64 KiB responses, 7-day staple validity cap | | OpenTelemetry | 2048 queued spans + one 256-span batch, 1 s interval, 2 s export timeout, 3 s shutdown bound | | Loopback-only endpoints | `/__hj_cache_purge`, `/__hj_cache_ready` (raw loopback peer), metrics `:9090`, admin listeners | Defaults are starting points, not sizing recommendations. Worker count, open-file limit, body budget, cache caps, PHP/backend concurrency and health/WAF timeouts must be sized together for the deployment. ## Reload boundary (summary) SIGHUP and the transactional API replace application state without dropping in-flight requests: vhost maps, rewrites, document roots, contexts, reloadable request settings, TLS material on an unchanged listener topology, and the trust policy of an already-enabled QUIC endpoint. Listener addresses/roles, QUIC enablement, Unix-listener topology, PHP topology, cache-store identity, process user/namespace policy and other boot-owned resources are **restart-required**; the write API says so instead of partially applying. Details: Admin API & reload . ## Reproducible checks **verify the tree** · what CI runs ``` cargo fmt --all -- --check cargo check --locked -p httpjet --bin httpjet cargo run --locked -p httpjet -- --root "$PWD/examples/litespeed" check --strict cargo test --locked --workspace ``` Protocol, cache, ACME, OCSP, OTel, FastCGI, WAF, admin and OCI pages carry their narrower acceptance commands. Every test uses synthetic configuration, temporary files and ephemeral loopback ports. ============================================================================== PAGE: Protocol stacks (https://httpjet.net/docs/protocols) SUMMARY: HTTP/1.1, native HTTP/2 with zero-allocation HPACK, HTTP/3 over QUIC, sendfile and kTLS egress, PROXY protocol, and Unix-socket listeners on one io_uring transport. ============================================================================== # Protocol stacks httpjet speaks **HTTP/1.1**, a **from-scratch native HTTP/2** stack, and **HTTP/3 over QUIC** through one unconditional **io_uring / monoio** transport. TCP and QUIC converge behind one rustls configuration, every protocol hands requests to the same pipeline, and file bodies leave over `sendfile(2)` where the kernel allows it. resolve acl rewrite cache dispatch finalize store ## Overview All serving lives in the binary's `uring/` module. HTTP/1.1 is a custom monoio + `httparse` codec; HTTP/2 and cleartext h2c use the native `hj-h2` stack through its monoio serve path; HTTP/3 is `uring/h3.rs`, a `quinn-proto` QUIC loop over io_uring UDP with `quinn-udp` socket metadata (GSO/GRO). Protocol selection happens at the connection boundary: ALPN over TLS, a non-destructive preface sniff in cleartext, and QUIC on the UDP listener. > [Diagram: connection-lifecycle] A TCP connection's life. PROXY-protocol stripping (if the listener opts in) happens before the TLS handshake or the cleartext sniff, so every later stage sees the proxied client as the peer. ALPN picks the TLS stack; the 24-byte h2 preface picks the cleartext one. H2 connections drain with GOAWAY on shutdown. HTTP/1.1 · custom monoio codec HTTP/2 · native `hj-h2` h2c · prior-knowledge sniff HTTP/3 · `quinn-proto` on io_uring HPACK · RFC 7541, zero-alloc TLS · rustls / aws-lc-rs · optional kTLS ## Connection front doors Two per-listener features sit before protocol detection. Both are httpjet-native — LiteSpeed has no listener-level equivalent — and both change *who httpjet thinks the client is*. ### PROXY protocol v2 / v1 A listener with ` 1 ` requires every connection to open with a PROXY header. The header is consumed and the claimed client address installed before the TLS handshake on secure listeners and before the H1/h2c discriminator on plain ones. Missing header → the connection is closed (fail-closed both ways). Since 2026-09-16 a header that claims a loopback or unspecified source is **refused at the parser**: those addresses feed the mTLS exemption and the loopback purge gate, so accepting them would let a proxy hop forge local trust. ! The listener opt-in is the trust decision Unlike OpenLiteSpeed's server-level trusted-peer ACL, httpjet binds PROXY trust to the listener (nginx/HAProxy semantics). Enabling it on an internet-facing bind lets anyone forge their client address; `check` warns on wildcard or public binds. Point such a listener only at your load balancer or a local hop. ### HTTP over Unix sockets A listener whose address is `unix:/path/sock` (or `uds:`), or the `--http-uds` flag, serves plain HTTP over a Unix-domain socket: H1 and h2c prior knowledge, one accept core (SO_REUSEPORT does not exist for AF_UNIX), socket mode 0600 group-writable, socket-activated fds adopted by path. A stale socket file is unlinked only after a `connect(2)` proves it dead, and cleanup is bound to the owned directory and socket identity. Peers have no address: ACLs, the per-IP throttle and `REMOTE_ADDR` see a fabricated loopback value and the access log renders `unix:`. TLS over UDS is unsupported and linted. ## HTTP/1.1 & h2c detection The H1 codec parses with `httparse`, enforces strict framing (single `Content-Length`, conflicting-length rejection, bounded incomplete reads), supports keep-alive and pipelining, honors `maxKeepAliveReq` (0 or 1 disables keep-alive, matching OpenLiteSpeed), decodes chunked bodies, and streams large responses from the bridge without buffering. Request-body deadlines extend on forward progress up to an absolute ceiling, while response egress enforces a minimum send-progress rate. Since the 2026-10-06 review it also enforces the RFC 9112 request-target and Host rules at intake: only origin-form targets (plus `*` for OPTIONS) and http/https absolute-form (reduced to origin-form, its authority replacing `Host`); `CONNECT` answers **405**, authority-form and a raw `#` answer 400. HTTP/1.1 **requires `Host`**, and any Host must be UTF-8 and pass the same authority validation the h2/h3 `:authority` rule uses (HTTP/1.0 may omit it). A header line terminated by a bare LF is rejected 400 before the head is materialized, and refusals that leave request bytes unread perform a lingering close so the peer reads the error status cleanly. Cleartext HTTP/2 by prior knowledge has no upgrade handshake — the client simply sends the connection preface. httpjet detects it *non-destructively*: the bytes already read are handed to whichever stack wins, so h1 and h2 both parse the connection as if no router sat in front. **uring** · ALPN + cleartext routing ``` // over TLS: ALPN decides the TCP stack match alpn { b"h2" => hj_h2::server::serve_local_with_prefix(io, prefix, service, cfg).await, _ => handle_h1_bridged(io, prefix, service).await, // http/1.1 or none } // cleartext: sniff the 24-byte preface, pass the observed bytes along match sniff_cleartext(&mut stream).await? { Proto::Http2(prefix) => hj_h2::server::serve_local_with_prefix(stream, prefix, service, cfg).await, Proto::Http1(prefix) => handle_h1_bridged(stream, prefix, service).await, } ``` Beyond framing, the H1 path implements the newer semantics end to end: - **103 Early Hints** — the same preload/preconnect `Link` matcher the h2 stack uses emits an interim `103` on h1 (batched into the same write as the final head when both are ready); `101` upgrades bypass it; canonical/alternate links are excluded. - **Ranges, single and multipart** — RFC 9110 multi-range sets are parsed, clamped, coalesced and framed as `multipart/byteranges`, capped at 8 ranges / 8 MiB of part bytes with a full-200 fallback. An invalid component poisons the whole header (→ 200); an unsatisfiable set is 416. - **Request decompression** — gzip bodies decode by default; brotli/zstd opt in with `--request-decompression-extra`. Bounded and buffered; see Compression . - **Bandwidth pacing** — a per-connection token bucket (burst 2× rate) paces `Body::Full` as 64 KiB slices and streams per chunk when `bandwidthLimit` is set; a matched context overrides the server-wide rate. - **Explicit `CONNECT` rejection** (405) and stalled-sendfile backoff with a 60 s zero-progress bound. ## Native HTTP/2 `hj-h2` is httpjet's own HTTP/2 server. Every h2 connection — ALPN `h2` over TLS and cleartext h2c — lands here through the monoio local serve path. The design targets the CDN→origin shape: few long-lived connections, many concurrent streams, so it optimizes one connection's multiplexing rather than spreading work across many. - TLS conn ALPN h2 / h2c ONE connection task FuturesUnordered⟨stream⟩ stream 1 · handler stream 2 · handler … stream N HPACK enc / dec send flow control · deadlines one vectored writev / wake wire zero-copy Bytes bridge dispatch → shared pipeline → monoio writes One connection task multiplexes every stream and emits a single vectored write per wake. Handlers cross the bridge into the shared pipeline; framing stays on the monoio core. **One task per connection.** All streams multiplex through `FuturesUnordered`; a 100-stream connection costs one task. - **Send-side flow control** honors per-stream and connection windows; **frame, header-block and body deadlines** are enforced on the monoio loop, and a stalled response stream waiting on withheld flow-control credit that delivers under 16 KiB per `connTimeout` window is reset with `RST_STREAM(CANCEL)`. - **Streaming bodies** — LSAPI, proxy, SSE and async file bodies stream without buffering; the response-completion hook keeps spans open through the final flush. - **One vectored write per wake**; large bodies are zero-copy from page-cache `Bytes`. The vendored `monoio-rustls` fork adds a real `writev`, flushes all queued TLS records in one socket write, and pools per-record buffers per core so records do not page-fault. - **Inbound RFC 9113 validation** with graceful GOAWAY drain; `CONTINUATION` without an open header block and stream-id reuse are connection errors; over-cap decoded header blocks become a **stream error** (`RST_STREAM(REFUSED_STREAM)`, decoding the block with emission suppressed so the HPACK dynamic table stays in sync on pooled CDN connections); the time-refilled rapid-reset budget (`httpjet_h2_rapid_reset_drops_total`), HTTP/2 Bomb pattern (CVE-2026-49975) and no-forward-progress cases are bounded. - **Egress pacing** is per connection at the OutQueue flush with the server-wide rate (a per-response override is ill-defined under multiplexing). HTTP/3 is not paced. ⎁ Measured The native stack passes **146 of 146** h2spec 2.6.0 cases over TLS (three consecutive runs is the gate). h2c is informative-only: generic §3.5/2 is structurally unsatisfiable on a port that also sniffs HTTP/1.1, so it records 145/146 with that one documented exception. scripts/h2spec_check.sh · scripts/ci-h2spec.py ## HPACK Header compression is RFC 7541 HPACK, implemented as the `hpack` submodule of `hj-h2` (once a standalone crate, folded in so the codec and the connection state evolve together). It is **zero-allocation** — fields are decoded and encoded by slicing frame bytes in place — and pure safe Rust with `#![forbid(unsafe_op_in_unsafe_fn)]`. Both the decoder and the round-trip are fuzz targets ( Conformance ). ## HTTP/3 & QUIC HTTP/3 is `uring/h3.rs`: `quinn-proto` for QUIC state, `quinn-udp` for socket metadata (with a live GSO segment cap and QUIC DATAGRAM disabled), UDP driven from monoio/io_uring, incremental request-stream parsing, QPACK encode/decode (charging decoded head bytes to both `total_req_bytes` and the global body budget), and the same bridge into the pipeline. Small, cached and sub-threshold bodies are buffered and sent whole; large, SSE and proxy bodies stream — HEADERS goes out immediately and DATA follows as the backend produces it, with backpressured writeback and send-stall stream resets (`httpjet_h3_send_stall_resets_total`). Retained response memory is bounded per connection (2026-09-06). The process-wide connection budget (shared with TCP accepts) is charged at `Connected` — after address validation — so spoofed Initials cannot hold budget for the idle timeout; until then a connection costs only one of 256 per-core handshake slots, and once 64 connections on a core are handshaking an unvalidated Initial receives a stateless **Retry** (token-bearing Initials are still accepted). A full budget closes with `H3_EXCESSIVE_LOAD`; handshakes are closed after 10 s. Discovery is via **Alt-Svc** on TLS responses (the last response transform), so an h2 client learns it can upgrade to QUIC. Connection migration is disabled because listener steering does not yet preserve ownership across an address change. Each connection pins its trust generation and mTLS requirement at acceptance; a trust reload replaces quinn's server configuration only for future handshakes while the bound UDP workers and established connection IDs survive. > [Diagram: h3-admission] HTTP/3 UDP intake, anti-amplification and stream admission. Unvalidated Initial packets pass through per-core handshake slot caps (256 max, stateless Retry above 64 concurrent handshakes) before charging the process-wide maxConnections budget at Connected. Decoded QPACK headers charge both total_req_bytes and BodyBufferBudget. ! Publish UDP too The same numeric HTTPS port must be reachable over TCP (h1/h2) *and* UDP (h3). Publishing `8443/tcp` alone means no HTTP/3, and the Alt-Svc header will advertise a port clients cannot reach. ## File egress: sendfile & kTLS An uncached `Body::File` keeps its pinned inode and byte range across the tokio→monoio bridge. On **plain TCP HTTP/1** it is sent with bounded nonblocking `sendfile(2)`; unsupported descriptor/filesystem combinations resume at the exact offset with positional io_uring reads, and short files or client aborts close the connection rather than completing an incorrect `Content-Length`. A stalled sendfile backs off exponentially and gives up after 60 s of zero progress. A `--features ktls` build extends this to **TLS 1.3 H1**: after the rustls handshake the socket is upgraded to kernel TLS, so the same sendfile writer feeds ciphertext without a userspace AEAD copy. A peer KeyUpdate queued during a transfer is applied before the next response writes, fencing TX rekey from sendfile. Policy is `--ktls=auto|on|off`: `auto` (default in feature builds) resolves the IPv4 default-route interface, reads its driver and features via ethtool ioctls, refuses known virtual NICs, and enables kTLS only when `tls-hw-tx-offload` is active — otherwise it logs one reason and stays on rustls. TLS 1.2, H2, H3 and UDS keep their userspace paths. ⎁ Measured Plaintext sendfile: 5,000 × 1 MiB at c=50 → **4,852 req/s**, 10.08 ms mean, zero failures. Software kTLS on a virtual GVE NIC (offload `off [fixed]`): **2,999.7 req/s off vs 2,944.9 on** — the expected-negative result that is why `auto` refuses virtual NICs. loopback · 2026-09-09 · `bench-results/httpjet-sendfile-*.csv`, `httpjet-ktls-sendfile-*.csv` ## TLS in one paragraph `hj-tls` builds **one** `rustls::ServerConfig` for TCP and QUIC with aws-lc-rs as the crypto provider. SNI maps directly to configured chains (no cert-name-match, like OLS), with dual ECDSA+RSA chains per name; client verification is optional mTLS against the *listener* CA with CRL enforcement; TLS 1.3 offers exactly `TLS_AES_128_GCM_SHA256`; resumption uses single-use tickets with 0-RTT off. Certificates, ACME and OCSP have their own page: TLS, ACME & OCSP . ============================================================================== PAGE: TLS, ACME & OCSP (https://httpjet.net/docs/certificates) SUMMARY: Certificate selection, mutual TLS with CRLs, resumption policy, opt-in ACME HTTP-01 and DNS-01 issuance, and opt-in OCSP stapling. ============================================================================== # TLS, ACME & OCSP How httpjet selects certificates, verifies clients, and resumes sessions — plus the two opt-in certificate services: **ACME** issuance and renewal (HTTP-01 and DNS-01, `--features acme`) and **OCSP stapling** (`--features ocsp`). Both are default-off at build *and* runtime, and both publish into the same generation-bound resolver slots that file-based certificates and SIGHUP use. resolve acl rewrite cache dispatch finalize store ## TLS configuration `hj-tls` builds **one** `rustls::ServerConfig` for TCP *and* QUIC via `build_server_config_alpn(cfg, listener, alpn)`, on the aws-lc-rs provider. A certificate rotation, verifier change or CRL update therefore lands on both transports at once. | Listener XML | Meaning | | `secure`, `keyFile`, `certFile` | The listener's default identity (PEM). `certChain=1` appends `CACertFile` as the served chain. | | per-vhost ` ` | The SNI identity for that vhost's domains — server *chain* material, never a client-verify root. Configure an ECDSA and an RSA pair for the same names to serve modern clients the smaller chain (ECDSA is chosen when the client offers ECDSA signature algorithms). | | `clientVerify` 0–3, `CACertFile`, `verifyDepth` | Mutual TLS: 0 none · 1 optional · 2 required · 3 optional without CA check. The verifier is built from the **listener** CA only. Positive chain verdicts are memoized off the hot path. | | `crlFile` httpjet | PEM CRL or bundle enforced by the webpki verifier. Absent = no revocation checking. Loaded at config build; rotating means a reload of the trust generation. | | `enableStapling` | Parsed for compatibility; OCSP fetching needs the `ocsp` feature and `--ocsp-responder` (see below). `check` says so. | ### Handshake posture - **SNI without cert-name-match** — the SNI name maps directly to configured material, mirroring OpenLiteSpeed; there is no re-validation that the leaf's SANs match the SNI. Managed ACME identities are exact-SNI overlays and do not replace the no-SNI listener fallback. - **TLS 1.3 offers exactly `TLS_AES_128_GCM_SHA256`** (the RFC 8446 mandatory suite); TLS 1.2 keeps its full set. rustls negotiates by client order, and without this pin OpenSSL-family clients pick slower suites; the transcript moves to hardware SHA-NI. - **Resumption & certified-peer idle window** — single-use TLS 1.3 tickets, 0-RTT off, including on client-cert-verifying listeners. Cloudflare does not re-offer origin tickets, so `httpjet_tls_handshakes_resumed_total` stays low behind it. Ticket volume is tunable with `--tls-session-tickets` (rustls default 2); production mints **0** — each ticket was two extra handshake records plus a cached copy of CF's client-cert chain for a resumption that never comes. Instead, `--trusted-proxy-idle-secs` extends the idle window for connections that presented a verified client certificate (reducing full handshakes per request by 28% on x86 and 78% on ARM in production). Resumption is *disabled* whenever OCSP is enabled (status must be enforced per handshake). - **mTLS trust boundary** — on a listener with `clientVerify=2`, a plaintext request for an mTLS-protected vhost is refused from the request head unless it arrives from loopback; `--no-mtls` relaxes the requirement for local testing only. - **Reload** — file-based certificates reload through `CertReloadHandle` on SIGHUP or via the write API ; a failed reload keeps the certificates whose reload failed. kTLS templates apply to new handshakes only. ! Two CAs, two roles A per-vhost `CACertFile` is chain material sent *to* clients. The listener `CACertFile` is the root that verifies *client* certificates. Treating the vhost chain CA as a client-verify root would silently widen the mTLS trust boundary; httpjet keeps them distinct. ## ACME certificate automation --features acme Build with `cargo build --locked -p httpjet --features acme`. The manager is *native CLI*, not pretend LiteSpeed XML: nothing happens unless all four required flags are present. One manager, account and identifier set is supported per process; its domain and TLS/HTTP vhost mappings are boot-frozen. HTTP-01 DNS-01 / wildcard Local test CA **serve** · HTTP-01 ``` httpjet --root /abs/root serve … \ --acme-directory https://YOUR-CA/directory \ --acme-domains example.test,www.example.test \ --acme-store /abs/private/acme \ # pre-created, mode 0700, owned by the service uid --acme-accept-terms \ --acme-bootstrap # only for first issuance without a configured certificate ``` Requires public TCP port 80 and an enabled TLS listener. The raw challenge route `/.well-known/acme-challenge/ ` runs before URI normalization, redirects, rewrites, auth, cache and backends; only plaintext GET on the validated authority answers (404 for unknown tokens, 405 otherwise, always `Cache-Control: no-store`); tokens are redacted in the access log. No exemption is granted for forwarded-scheme headers, and the historical path-prefix bypass (#259) stays closed. **serve** · DNS-01 ``` httpjet --root /abs/root serve … \ --acme-directory https://YOUR-CA/directory \ --acme-domains 'example.test,*.example.test' \ # quote the wildcard --acme-store /abs/private/acme --acme-accept-terms \ --acme-dns-webhook https://YOUR-CONTROLLER/acme \ --acme-dns-zones example.test \ --acme-dns-token-file /abs/private/dns-controller.token # regular file, 0600, 16..4096 printable chars ``` No port 80 requirement and no HTTP challenge exemption. There is **no built-in Cloudflare or provider integration**: you run a controller that implements the webhook contract below with a credential scoped to the listed zones. A managed wildcard matches exactly one leftmost label — request the apex separately. **controller contract** · POST, Bearer token, application/json ``` {"version":1,"operation":"present","name":"_acme-challenge.example.test","value":" "} {"version":1,"operation":"ready", "name":"…","value":"…"} → {"ok":true,"ready":true|false} {"version":1,"operation":"cleanup","name":"…","value":"…"} → {"ok":true} ``` Operations must be idempotent: `present` adds the exact value without replacing the RRset; `cleanup` removes only that value and succeeds if it is already absent (apex and wildcard share one owner name). Replies over 16 KiB, redirects, non-2xx and unexpected fields are rejected; each call has a 5 s deadline; a `ready:true` returned without real DNS visibility is not a valid production implementation. `--acme-test-mode` permits literal-loopback HTTP directories and alternate HTTP ports and requires loopback serving addresses. `--acme-test-ca-root` and `--acme-test-issuer-root` replace trust roots without disabling verification; `--acme-dns-test-root` does the same for the DNS controller. These are for the Pebble fixture only: **acceptance** · Pebble v2.10.1 ``` cargo test --locked -p hj-acme cargo build --locked -p httpjet --features acme python3 -B scripts/acme-pebble-test.py --binary target/debug/httpjet \ --pebble /path/to/pebble --dns /path/to/pebble-challtestsrv \ --pebble-source /path/to/pebble-v2.10.1 --bootstrap [--dns01] ``` ### Issuance, recovery and activation > [Diagram: acme-http01] HTTP-01 issuance. Every durable step precedes the network step that depends on it: the account key before registration, the certificate key and CSR before finalize, and the issued pair in the same atomic snapshot as account/order state. Only a fully validated pair is published to the TLS resolvers. - **Durable before network.** A lost `newOrder` response is intrinsically ambiguous, so a marker stops automatic reissuance: recover the order URL from the CA's account records and restart once with `--acme-reconcile-order `. Never delete the store as a generic retry. - **Bounded CA traffic.** Only GET/HEAD/POST at the directory's origin, no redirects, no environment proxies, 1 MiB bodies, 5 s connect / 10 s request, 120 s per attempt, persisted exponential backoff with account jitter plus `Retry-After`. - **Private store.** An operator-provisioned 0700 directory owned by the effective uid; every ancestor is checked with pinned no-follow descriptors; an exclusive lock prevents two managers sharing a store; snapshots are 0600 single-link regular files written pending→fsync→rename→dirsync, capped at 1 MiB; an unfinished pending file is discarded on reopen, never promoted. - **Activation.** The validated `CertifiedKey` is published to the TCP, kTLS and QUIC resolver overlays as an exact-SNI identity. File-based SIGHUP updates a separate slot and cannot erase or mix managed generations. Boot revalidates the persisted pair; an expired one is rejected and needs operator recovery. - **Bootstrap.** With `--acme-bootstrap` the default TLS identity starts *absent*: handshakes for hosts without a configured certificate fail until issuance succeeds. No self-signed placeholder is generated; client-auth policy is unchanged. - **Renewal** is scheduled with one third of the remaining lifetime in reserve plus jitter. > [Diagram: acme-dns01] DNS-01 with durable cleanup. The exact owner/value cleanup intent is journaled before any DNS mutation, so a failed, cancelled or timed-out presentation is always recoverable; cleanup replays after attempts, at shutdown and at startup, and blocks new issuance until it succeeds. Snapshots bind cleanup to the provider endpoint and sorted zone scope — changing those while reusing the store fails closed. Scope today: DNS identifiers only (ASCII A-labels), one account/group per process, same-origin ACME endpoints, no EAB, contact management or ARI. Production at windowsforum.com still renews through certbot; the manager is an adoption path, not a forced replacement. ## OCSP stapling --features ocsp Build with `--features ocsp` (OpenSSL 3.x development headers and pkg-config at build time, the shared libraries at runtime). Pass `--ocsp-responder https://ocsp.example.net/status` to `serve`, optionally with `--ocsp-required`. The explicit CLI opt-in applies to every certificate on the process's TLS listeners regardless of the XML `enableStapling` boolean, and the one responder must serve every enabled issuer (per-issuer routing is not implemented). ! Not for Let's Encrypt Let's Encrypt retired its OCSP service on 2025-08-06 and publishes CRLs instead. There is no responder to fetch from — do not enable this manager for current Let's Encrypt certificates. It exists for other CAs. > [Diagram: ocsp-policy] Status lifecycle. Fetching never happens during a handshake. A verified GOOD response fills a generation-bound staple slot and is refreshed before expiry with jitter; a verified REVOKED result suppresses that leaf's handshakes even across a same-leaf reload; malformed or transient failures preserve a still-valid staple and otherwise fall to the optional/required policy. - **Trust.** Each managed leaf must include its immediate issuer in the configured chain; responses must be signed by that issuer or a responder it directly authorizes for OCSP signing. Arbitrary trusted-root responders, unsigned responses, AIA fetching, redirects, encoded bodies and verification-bypass flags are all refused. SHA-256 CertIDs; RSA PKCS#1 v1.5 and ECDSA signatures with SHA-256/384/512. - **Freshness.** `thisUpdate`, `producedAt` and `nextUpdate` are required; a staple is never valid past seven days from `thisUpdate` or the leaf/issuer/responder validity; expiry uses wall *and* monotonic time so a clock step back cannot extend a staple. - **Transport.** Percent-encoded GET when the full URL is ≤ 255 bytes, else binary POST; DNS pinned per fetch; private destinations rejected except literal loopback under `--ocsp-test-mode`. - **Policy.** Optional (default): cold or expired status serves without a staple unless the certificate has Must-Staple. Required: new handshakes are rejected until a valid response arrives — including at startup and for new certificate generations, so plan availability. Status is memory-only and resets on restart. **acceptance** · real-server fixtures on isolated listeners ``` cargo test --locked -p hj-ocsp cargo test --locked -p hj-tls --features ocsp ocsp cargo build --locked -p httpjet --features acme,ocsp python3 -B scripts/ocsp-runtime-test.py --binary target/debug/httpjet # required policy python3 -B scripts/ocsp-runtime-test.py --binary target/debug/httpjet --optional python3 -B scripts/ocsp-runtime-test.py --binary target/debug/httpjet --optional --must-staple ``` ## Kernel TLS The staged `--ktls` path (feature `ktls`) upgrades a completed TLS 1.3 rustls handshake to kernel TLS so H1 file bodies leave through `sendfile(2)`. It is hardware-aware and off on virtual NICs by default; see Protocol stacks → File egress for the policy and the measured (negative-on-GVE) receipt. ============================================================================== PAGE: LSAPI & PHP (https://httpjet.net/docs/lsapi) SUMMARY: The native LSAPI wire codec, the lsphp supervisor, the zero-downtime external pool with generation tracking, the CGI environment, and the fail-closed jail. ============================================================================== # LSAPI & PHP httpjet speaks the **native LSAPI wire protocol** to an operator-configured `lsphp` executable — no FastCGI shim, no translation layer. `hj-lsapi` holds the wire codec, a generation-aware UDS client pool, the `lsphp` supervisor, the CGI environment builder, and a fail-closed jail. The production shape is a **persistent external pool** so the web tier restarts without cold-starting PHP. resolve acl rewrite cache dispatch finalize store ## Overview When suffix routing maps a request to a PHP handler (` ` or an `lsapi` `extProcessor` bound by a `scriptHandler`, or a `.htaccess` `SetHandler application/x-httpd-php`), httpjet talks LSAPI directly to `lsphp`. PHP is optional: static, proxy and FastCGI deployments need no LSAPI executable. - client HTTP/1·2·3 HTTP httpjet LSAPI handler · CGI env LSAPI · UDS lsphp pool — capped by --php-children / maxConns php #1 php #2 … php #N streamed response · php-slow log · lsapi_ttfb histogram Native LSAPI over a Unix socket. The pool may be owned by the web process (spawn mode) or run as a separate `httpjet lsphp` process (external mode, the production shape). Native LSAPI · vendored `lsapidef.h` Operator-supplied `lsphp` Generation-tracked UDS pool Candidate-first zero-downtime reload Fail-closed jail ## The LSAPI wire codec `proto/{frame,builder}` implement the protocol straight from the vendored php-src `lsapidef.h` constants (BSD-3-Clause), so frames match byte-for-byte what `lsphp` reads. One sharp edge deserves stating precisely: every env key/value and known header value is length-prefixed with a big-endian `u16` that **includes the trailing NUL**, so the largest safe field is `u16::MAX − 1` bytes. A longer field would wrap the prefix and desync `lsphp`'s parser for every subsequent frame; the builder rejects it. **proto** · the NUL-inclusive length prefix ``` // each env/header field: [u16 BE = byte_len + 1] [bytes…] [0x00] const MAX_FIELD: usize = (u16::MAX as usize) - 1; // 65534 ``` The response-header parser is one of the eight fuzz targets (`lsapi_resp_header`). Request bodies are scanned before transmission and refused if they embed an `lsphp` `BEGIN_REQUEST` frame signature, negative env counts are rejected at the builder, and the repository ships a verified `packaging/lsphp/lsapilib-hardening.patch` for upstream `lsapilib.c`. If a worker dies before emitting an `LSAPI_RESP_END` packet, the response body is aborted rather than silently truncated. Inbound tracing context reaches PHP as `HTTP_TRACEPARENT` in the unknown-header table when OpenTelemetry is on. ## The UDS connection pool Requests reach `lsphp` over a Unix-domain socket fronted by a pool that carries a **generation epoch**. In owned mode the epoch is process-local; in external mode it is the 16-byte generation file below, mapped read-only. A bodyless idempotent request whose pooled connection dies before the send gets one transparent retry on a *fresh* dial, bounded by the processor's `retryTimeout`. A fresh-dial `EAGAIN` under backlog pressure is retried and counted (`httpjet_lsapi_eagain_*`). Accepted connections keep TCP keepalive so a vanished client cannot pin pool permits. Attribution is what makes generation retirement safe: for every connection the pool identifies the accepting worker through `UNIX_DIAG` peer-inode lookup plus `/proc`, pins it with a pidfd, and — after the response completes — retires a stale-generation worker through that pidfd rather than by pid (workers call `setsid()` and are not reliably in the master's tree). Failures to attribute are counted, never guessed. Metric Meaning `httpjet_lsapi_generation_advances_total` Epoch advances this web process observed. `httpjet_lsapi_stale_idle_drops_total` / `_stale_checked_out_drops_total` Idle sockets discarded / checked-out sockets refused re-pooling after an advance. `httpjet_lsapi_stale_worker_retire_signals_total` / `_failures_total` Old-generation workers retired via pinned pidfd after their response completed / signals that failed. `httpjet_lsapi_worker_attribution_failures_total` Connections whose worker could not be pinned. `httpjet_lsapi_acquire_seconds` · `_dispatch_seconds` · `_ttfb_seconds` Pool contention · backend call time · request-committed-to-first-byte (worker pickup + render start). ## The lsphp supervisor In spawn mode (`serve` without `--lsphp-external`) `hj-lsapi` spawns and supervises `lsphp` itself: the listen socket is passed on **fd 0**, inherited socket-activation listeners are kept out of PHP children via `FD_CLOEXEC` and `close_range(..., CLOSE_RANGE_CLOEXEC)`, `setsid()` detaches the pool from any controlling terminal, `PR_SET_PDEATHSIG` ties a worker's life to its parent, and the `pre_exec` closure between fork and exec calls only async-signal-safe syscalls. The process is a subreaper (reaping all adopted PHP orphan zombies) and tracks each generation family by an injected `LSAPI_Z_HTTPJET_GENERATION_MARKER` plus pidfds. The config's `LSAPI_CHILDREN` env is stripped so `--php-children` / `maxConns` is the only cap (values below 2 normalize to 2). Two refusals are unconditional (2026-09-16): the supervisor **never runs workers as root** (resolve_credentials rejects uid/gid 0 even if suEXEC would compute it) and **refuses to spawn a setuid `lsphp` binary**. `--php-max-process-time-ms` sets a total per-request deadline: exceeded → 504. In spawn mode the monitor also kills and restarts the worker; under `--lsphp-external` there is no monitor in the web process, so only the 504 applies and a wedged worker is not auto-reclaimed here — keep the value conservative. ! Packaging boundary httpjet ships no PHP runtime. The executable, its extensions, `php.ini`, opcache settings and writable application paths are operator-managed. For the pool unit's sandboxing the repository ships a **validated hardening runbook** (`systemd/httpjet-lsphp.service.d/hardening.conf`: a bounded capability set, the standard Protect*/Restrict* family, and `MemoryDenyWriteExecute=no` because opcache JIT needs RWX, with the opcache helper steps run via `+`-prefixed ExecStartPre/Reload) — validated against a live pool, but not applied by any unit by default; adopting it is the operator's call. ## Zero-downtime deploys: the external pool Spawn mode couples PHP's lifetime to the web process — a binary deploy cold-starts opcache. The `httpjet lsphp` subcommand instead runs a **persistent pool as its own systemd unit** (`httpjet-lsphp.service`, `Type=notify`, socket-activated through `httpjet-lsphp.socket`); the web tier dials it with `serve --lsphp-external --lsphp-generation-file ` and never spawns workers itself. Both halves then deploy independently: ### Web tier deploy `systemctl restart httpjet` - systemd holds the :80/:443 fds — no RST for new or queued connections - PHP workers and opcache stay warm ### PHP recycle - `systemctl reload httpjet-lsphp` → `httpjet lsphp-reload` - candidate-first: a new family must answer a real, marker-attributed LSAPI request before promotion - the systemd-owned listener is never closed; queued dials survive > [Diagram: lsphp-reload] Candidate-first generation reload. The supervisor promotes a new family only after a marker-attributed response, then advances the shared 16-byte epoch. Web pools observe the epoch, drop idle sockets, and retire each in-flight stale worker only after its response completes, through a pidfd pinned before the epoch moved. The old family is drained with bounded forced cleanup. external pool wiring (see Operations for the units) ``` # the PHP half — its own unit, its own identity httpjet --root /abs/root lsphp --php-socket /run/httpjet/php.sock --php-children 256 \ --control-socket /run/httpjet/lsphp-control.sock \ --generation-file /run/httpjet/lsphp.generation # the serving half — dial, never spawn; restart freely httpjet --root /abs/root serve … --lsphp-external /run/httpjet/php.sock \ --lsphp-generation-file /run/httpjet/lsphp.generation # opcache/PHP recycle without dropping the listener httpjet lsphp-reload --control-socket /run/httpjet/lsphp-control.sock --timeout-secs 90 ``` ! The generation file is the trust anchor The 16-byte file (epoch + promoted-family marker fingerprint) is how a web tier tells *its* pool from a foreign one. Supplying `--lsphp-generation-file` is fail-closed: startup aborts if it is missing or malformed. Its parent directory must not be group/other-writable. Test pools default their control and generation files *beside their own socket* so they can never advance the production epoch — which would drop the live web tier's pooled connections and retire its workers against a foreign marker. ! Never restart the socket unit during a reload `httpjet-lsphp.socket` is the continuity boundary: keeping it open preserves the listener inode and its 8192-deep accept queue. Replace the supervisor by restarting only `httpjet-lsphp.service`; it reapplies the XML backlog to the inherited fd with `listen(2)`. Verify with `stat` of the socket inode before/after, a 16-byte generation file, and `ss -xln` Send-Q = 8192; `LIVE=1 scripts/lsphp-reload-test.sh` is the live gate. ## The CGI environment The env builder assembles the CGI/LSAPI environment per request. The load-bearing split is *what was requested* vs *what runs*: the `LsapiScript` request extension pins a resolved `SCRIPT_FILENAME` while the **original** `REQUEST_URI` is kept, with `SCRIPT_NAME`/`QUERY_STRING` pushed into `ctx.env` and upserted over the builder's defaults. That is what makes front-controller routing (every page → `/index.php`) correct. | Variable | What it holds | | `REQUEST_URI` | The original request URI, preserved through rewrites. | | `SCRIPT_FILENAME` / `SCRIPT_NAME` | The resolved on-disk script and its URI-space name. | | `REMOTE_ADDR`, `SERVER_NAME`, `HTTPS`, `CONTENT_LENGTH`, `HTTP_AUTHORIZATION`, `PHP_VALUE`, `PHP_ADMIN_VALUE`, … | Server-computed or special-env only. Since 2026-09-16 a `.htaccess` `[E=…]`/`SetEnv` or processor `env` entry that names one of these is **denied** (shared deny-list with FastCGI) — a directory cannot override the server's view of the client or inject PHP ini overrides through the regular environment. | | `HTTP_*` | Request headers after name/value validation, under one forwarding rule (2026-10-06, applied identically to the CGI env table, the LSAPI wire header block and FastCGI params): the `Proxy` header and underscore-named headers never pass, and forwarding-authority names (`CF-Connecting-IP`, `X-Forwarded-*`, `X-Real-IP`, `True-Client-IP`, `Forwarded`) pass only from a peer that is *both* a trusted proxy and — on a client-cert listener — presented one. `HTTP_TRACEPARENT` when OTel is enabled. | | `REMOTE_USER` | Set by a successful Basic auth realm. | | special-env section | `php_value` / `php_admin_value` / `php_flag` / `php_admin_flag` from `.htaccess`, carried on the LSAPI special-env wire path with the right permission level — plain `lsphp` ignores a regular CGI `PHP_VALUE`. | ## The jail PHP execution is fenced by a config-gated jail — suEXEC (` ` with `uidMin`/`gidMin` floors), chroot (`chrootMode` 0–2, per-vhost `chrootPath`) and optional Linux namespaces (`mount`, `net`, `uts`, `ipc`; `pid` is accepted but deliberately not honored). Resolution has two outcomes and no third: - suEXEC off, non-root process, or no per-vhost isolation → the **all-None** case: the default `"php"` pool, single-pool behavior preserved byte-for-byte. - A configured jail that turns out unsafe — uid 0, a uid below the floor, a bad chroot, an out-of-range `setUIDMode`/`chrootMode` (hard error since 2026-09-16), or a chroot without credentials to run inside it (hard error since 0.0.5) → httpjet **disables PHP for that vhost and returns 503**. It never runs unjailed, and it never falls through to the static handler. ! Source-disclosure guard A script that resolves to a PHP handler is identified as PHP even when no pool can serve it. Falling through to static would hand back the raw `.php` source, so an unavailable pool is a 503, always. Operators sharing one pool across vhosts should read the "shared PHP trust boundary" note in the repository's security docs: workers in one pool share credentials, so per-vhost isolation is a suEXEC decision, not a default. ## Request bodies LSAPI needs a concrete `m_reqBodyLen` before the body is sent, so chunked / no-`Content-Length` uploads are collected first. `collect_to_cap` enforces the effective `maxReqBodySize` *as frames arrive* — the instant the accumulated size would exceed the cap it returns `PayloadTooLarge` (413) — and every buffered byte is charged to the process-wide body budget through an owned lease that is released on completion or on connection teardown (H2/H3 teardown leaks were closed in the 2026-09-06 review). ⎁ Measured PHP/LSAPI over HTTP/1.1: **~44.8k req/s vs LiteSpeed's ~38.5k at c=40** on the same hardware (LiteSpeed's best, ~41.2k, needed c=60). In production the render inside PHP is ~90% of request wall time; pool acquire is p99.9 loopback · PGO · 2026-06 · production telemetry 2026-08-30 ============================================================================== PAGE: Proxy & gateways (https://httpjet.net/docs/gateways) SUMMARY: Reverse proxy over h1/h2/h2s, weighted upstream groups with active health checks, WebSocket and SSE relay, and the opt-in FastCGI gateway. ============================================================================== # Proxy & gateways Everything that hands a request to another process: the **reverse proxy** (HTTP/1, `h2://`, `h2s://`) with **weighted upstream groups and active health checks**, **WebSocket** and **SSE** relay, and the opt-in **FastCGI** gateway. LSAPI/PHP has its own page . Generic CGI execution and uWSGI are specified but deliberately not routable. resolve acl rewrite cache dispatch finalize store ## How a request reaches a backend Three routes select a gateway, in dispatch order: a **WebSocket map** (` `) when the request carries an upgrade, a **proxy ``** whose URI prefix matches and whose `handler` names a `proxy` extProcessor, and a **rewrite `[P]`** target (deferred so it can still participate in the page cache). Script suffixes (`scriptHandler`) route to LSAPI or FastCGI. A proxy context that cannot resolve its processor falls through to static; a script handler that cannot resolve its pool hard-fails 503. | Target form | Protocol to the backend | | `host:port`, `ip:port`, `uds:///path`, `http://…` | HTTP/1.1 with keep-alive pools | | `https://…` | HTTP/1.1 over TLS; backend certificate verified; optional client cert (`clientCertFile`/`clientKeyFile`) | | `h2://host:port` httpjet | HTTP/2 prior knowledge (cleartext); multiplexes requests on one backend connection | | `h2s://host[:443]` httpjet | HTTP/2 over TLS, ALPN fail-closed | Pool keys are scheme-scoped, so an h1 and an h2 target at the same address never share a connection. Forwarded requests to h2 upstreams are re-absolutized (the h2 client derives `:scheme`/`:authority` from the URI). Proxied *responses* may not assert `X-LiteSpeed-Purge`/`-Cache-Control`/`-Vary` — those are stripped so a remote backend cannot inject cache policy. Named upstreams receive a W3C `traceparent` when tracing is on. ! [P] targets are screened Since 2026-09-16 a rewrite `[P]` URL that resolves (after DNS — `/etc/hosts` pins cannot bypass the screen, and concurrent DNS lookups are bounded so slow resolvers fail fast) to loopback, link-local or a Unix socket is refused unless it matches an operator-configured extProcessor address. Forwarding to a foreign `[P]` origin also withholds client `Authorization` and `Cookie` headers. This closes the class where a `.htaccess` author reaches the local trust plane (metrics, admin, purge endpoints) or leaks origin credentials through the server. ## Upstream groups & selection An `extProcessor` with several ` ` children is a group. Without the options below it keeps **ordered primary/failover** behavior with a passive circuit breaker (3 consecutive dial failures mark a peer bad; one successful dial clears it — nginx `max_fails` semantics). With them it becomes a load-balanced group: **httpd_config.xml** · a weighted group with active probes ``` application proxy 127.0.0.1:9001 127.0.0.1:9002 weighted-round-robin 2,1 GET /ready application.internal 10 2 2 3 200 ``` > [Diagram: proxy-selection] Selection. Eligibility is the intersection of active health and the passive breaker; the policy then picks among eligible peers. A pre-head dial failure marks the peer and moves to the next eligible one; nothing replays a request body that has already been sent. > [Diagram: upstream-health] Peer states. A health-enabled peer starts cold (ineligible) until rise consecutive successes; fall failures take it down; probe success restores it without user traffic. Passive breaker state is independent. All-ineligible → 503. - **Policies:** `primary-first` (default), `weighted-round-robin` (smooth WRR), `weighted-least-active` (active ÷ weight, smooth-WRR tie-break). These options on non-proxy processors, unknown policies and invalid health settings fail config loading rather than silently falling back. - **Probes:** `connect` tests transport only; `GET`/`HEAD` use the configured protocol, validate TLS, bound the response head, compare the status exactly, follow no redirects, consume no body, and carry no cookies, authorization or body. Dedicated connections, a process-wide cap of 16 concurrent probes, skipped missed ticks; they never consume application pool permits. - **All peers in a group share a protocol**; do not put credentials, paths or queries in group authorities. A `Host` override does not override TLS identity. - **Reloads:** candidate generations start no probe tasks; publication starts the new scheduler and retires the old one. Unchanged identities keep their group state; changed endpoints, weights, policies or TLS identities start fresh. Server- and vhost-scoped processors are distinct. - **Reservations** survive the response head: streaming bodies and WebSocket relays release the peer's active count on completion, error or drop. Per-peer metrics carry scope, group name and ordinal labels (never URLs or credentials): `httpjet_proxy_peer_{healthy,probes_total,health_transitions_total,active_requests,selections_total,prehead_failures_total}`, plus `httpjet_proxy_failover_total` and the `httpjet_phase_upstream_seconds` latency histogram. The read-only status API exposes the same peer health. ## Streaming, SSE and timeouts Upstream responses are **never buffered** (`respBuffer` is parsed but always streams — `check` warns). Server-sent events therefore stay real-time through compression (the per-chunk `CompressStream`) and through H1, H2 and H3 alike. A backend connect timeout is `initTimeout`; the response-head deadline is the processor's default or a context's `responseTimeout` override; a backend timeout is a 504. Request bodies stream to h1/h2 upstreams; a context `maxReqBodySize` is enforced before dispatch. ## WebSocket relay A ` ` map (`uri` → `address`) or an upgrade request on a proxy context performs the HTTP/1 handshake with the backend and then relays raw frames both ways. The relay supports **plain TCP HTTP/1 upstreams only** — not TLS, H2 or Unix-socket targets. Maps carry a stable `websocket: ` target identity scoped to their vhost, so named-target trace propagation applies; the client span covers the handshake, not the potentially unbounded relay. TCP keepalive keeps a vanished peer from pinning fds and permits forever. ## FastCGI gateway opt-in An external processor declared ` fcgi ` or `fastcgi` is an `ExtKind::FastCgi` — never silently converted into an HTTP proxy or an LSAPI worker. Only an explicit `cgi` script handler whose named processor is FastCGI dispatches through the gateway; a `cgi` mapping with a missing or different processor **fails closed** and is never served as static source. httpjet does not spawn the application: it connects to the configured TCP or Unix endpoint of an independently supervised process. **vhost XML** · FastCGI suffix routing ``` my_fcgi fcgi uds:///run/my-app/fcgi.sock 32 5 30 fcgi cgi my_fcgi ``` > [Diagram: fastcgi-flow] One request through hj-fastcgi. The pipeline hands over an already-validated script target; the gateway never derives an executable path from the URI. Connections are poisoned by default and only a clean, exact END_REQUEST makes one reusable. Multiplexing request ids on one connection is disabled. - **Wire bounds:** FastCGI version 1, nonzero request id, known record types, the protocol's 65,535-byte record limit; PARAMS lengths are rejected before allocation when either the field bound or the configured aggregate cap would be exceeded. - **Pool:** caps all sockets including idle and in-progress dials under one acquisition deadline; poison/drop releases capacity immediately; clean idle sockets expire oldest-first. - **CGI variables** are server-derived; request headers become `HTTP_*` after validation; processor `env` cannot override protected identity/routing variables (`SCRIPT_FILENAME`, `DOCUMENT_ROOT`, `REMOTE_ADDR`, `SERVER_NAME`, `CONTENT_LENGTH`, `HTTP_AUTHORIZATION`, …). The same deny-list applies to `.htaccess` `[E=]` for LSAPI. - **Bodies** are buffered under the shared budget so the exact STDIN length is known; declared underflow/overflow, framing errors and cancellation poison the connection. - **Responses:** strict CRLF head, repeated end-to-end headers preserved, one valid `Status`, CGI `Location` default, folding rejected, app hop-by-hop headers and origin cache-control headers (`X-LiteSpeed-*`) stripped, and streams that end without an `END_REQUEST` frame abort the body; STDERR is byte-capped and control-escaped into the error log, never the client. Known follow-ups: explicit CGI context-directory routing (today: script-handler suffix mappings only), a codec fragmentation corpus, and a maintained real-application interoperability fixture. ## Conventional CGI and uWSGI not routable Both remain disabled by design until a demonstrated workload exists. Existing `type=cgi`, `rails`, `appserver` and unknown processor types never activate either; `check --strict` keeps warning for configured CGI contexts. The design contracts are strict on purpose: CGI would require an absolute execution root distinct from the docroot, a fixed non-root identity, wall/CPU/memory/process/output limits, an executable allowlist, direct `execve` with no shell or PATH lookup, inode pinning, and bounded pipes with process-group termination; uWSGI would be a separate crate over an explicit endpoint with modifier-0 packets only, no multiplexing, and no connection reuse until a maintained responder proves the response boundary. Proposed identities are `type=cgi-exec` and `type=uwsgi`. ## Reproducible checks **tests** · loopback fixtures only ``` cargo test --locked -p hj-config -p hj-proxy cargo test --locked -p hj-proxy --test upstream_groups # h1/h2 weighted distribution, cold start, down/recovery, all-down 503 cargo test -p hj-fastcgi # synthetic loopback responder + clean-connection reuse cargo test -p httpjet --bin httpjet ``` To return a group to legacy selection, remove `loadBalancePolicy`, `addressWeights` and `healthCheck` and reload; removing only the health check keeps the configured policy. ============================================================================== PAGE: Rewrite & .htaccess (https://httpjet.net/docs/rewrite) SUMMARY: The Apache mod_rewrite engine: per-directory loading, the dense-DFA prefilter, directives and flags, access control, Basic auth, header operations, and the fast-path memo. ============================================================================== # Rewrite engine & .htaccess `hj-rewrite` is httpjet's Apache **mod_rewrite** / `.htaccess` engine — depending only on `hj-compress` so `.htaccess` and XML expiry directives share one `A/M` time grammar. It prefers a linear-time regex engine and falls back to **`fancy-regex`** only for the lookaround and backreferences real `.htaccess` files use. The same crate enforces per-directory access control, scoped Basic auth, header operations, expiry rules and the LSCache directives. resolve acl rewrite cache dispatch finalize store ## Overview The rewrite stage runs the vhost's inline ` ` rules first, then the `.htaccess` chain for the resolved directory — OpenLiteSpeed's order. One entry point, `evaluate()`, returns exactly one outcome per pass: `Rewritten`, `Redirect`, `Proxy`, `Forbidden`, `Gone` or `Unchanged`. > [Diagram: rewrite-eval] One evaluation pass. A rewrite that crosses into another directory reloads the destination chain and evaluates again; a terminal flag stops the walk; an exhausted chain returns Unchanged and dispatch falls through to suffix routing. ## Per-directory loading An `HtaccessCache` loads each directory's file (name from `accessFileName`, default `.htaccess`; bare file names only — `/` and `..` are rejected) with **mtime invalidation**, so an edit takes effect on the next request without a reload. The parsed form holds the directive list, compiled regexes, scoping blocks, a per-file memo of which access rules match, the auth realm map and the memo-eligibility verdict. The chain from docroot to the request's directory is bounded at **64** levels; past that a deny-all sentinel applies. Since 2026-09-16 an **unreadable or non-UTF-8 `.htaccess` also fails closed** to that sentinel rather than being skipped. Vhost-level switches: ` 0` disables the whole chain (even with `autoLoadHtaccess`), while `autoLoadHtaccess` alone loads files without enabling general overrides. Candidate configuration generations own their own htaccess, rewrite-outcome and UA caches, so a reload never reintroduces stale results. ## The dense-DFA prefilter Rather than test every rule's regex per request, the eligible patterns compile once into **one fully-determinized dense DFA** (`regex_automata::dfa::dense`) searched a single time to learn *which* rules could match. Only those run for real. A dense DFA holds no lazy transition state, so the search is **cacheless**: no per-search cache allocation and no cross-thread pool lock — one mutex fewer on the request path at high concurrency. **hot path** · prefilter, then evaluate ``` let hits = prefilter.search(path); // which rules *could* match — one DFA pass for rule in hits { if let Some(m) = rule.regex.find(path) { return rule.apply(m); } // substitute + flags } Outcome::Unchanged ``` ## Linear vs backtracking matching A flagged rule runs through a `CompiledRegex` that prefers `regex_automata::meta::Regex` with an explicit thread-local cache keyed per pattern, and falls back to `fancy-regex` only for constructs the linear engine cannot express. The split is exact, not approximate: a pattern the linear engine accepts has identical match semantics under both, so the choice is invisible. | Pattern shape | Engine | Why | | Anchors, classes, alternation, repetition | `regex_automata::meta` | Linear time; per-pattern thread-local cache, no shared lock. | | Lookahead `(?=…)` / `(?!…)`, lookbehind | `fancy-regex` | Not expressible as a finite automaton. | | In-pattern backreference `\1` | `fancy-regex` | Requires backtracking. | A regex that errors at match time **fails closed toward the rule's polarity**: a deny that cannot evaluate counts as matched; an allow that cannot evaluate is withheld (hardened 2026-09-06 alongside the spoofing checks). ## Rewrite-outcome cache & UA classification For chains whose inputs are *path-cacheable*, the whole rewrite decision is memoized for `--rewrite-outcome-ttl-ms` (default 1000 ms — which also bounds `-f`/`-d` filesystem staleness and lets `.htaccess` edits land within a second; 0 disables). The key is vhost/scheme/method/host/path/query plus any keyable header variables the chain reads. Counters: `httpjet_rewrite_outcome_{hits,misses,uncacheable}_total`. `--rewrite-ua-classify` (default off, a deploy-time decision) keys User-Agent-reading chains by the *bitmap of matching UA RewriteConds* instead of the raw UA string, when parse-time analysis proves the outcome depends on the UA only through those matches. Real-world UA diversity collapses onto a handful of entries (`httpjet_rewrite_ua_classify_entries`). Since 2026-09-06 header-dependency collection continues past the UA classification so a chain that also reads Origin or Accept keys correctly. ## The fast-path memo and chain eligibility The on-core **finished-response memo** (default on; `--no-fast-memo` / `HJ_FAST_MEMO=0`) serves repeat cookieless static GETs without running the pipeline at all — per thread, 1 s TTL, 64 MiB/worker, only for 200 full bodies without `Set-Cookie`, and only after each distinct key has passed the full pipeline once. Whether a docroot's chain is eligible is a **parse-time verdict** (`MemoClass`, fail-closed) that `httpjet check` prints per vhost. > [Diagram: memo-eligibility] Memo eligibility. A chain that reads anything the memo key cannot carry — client/server addresses, protocol, IP/env/host access operands, REDIRECT_STATUS, %{HTTP_COOKIE}, %{TIME*} — is ineligible and pays the full pipeline every time. Eligible chains memoize with a VARY-SET of every header the chain reads (raw Origin, UA bitmap …), so an 800-line forum chain still memoizes. ⎁ Measured Same-binary A/B on the forum's 806-line docroot chain (`/robots.txt`, c=100): **h2 137k → 248k req/s (+81%)**, **h1 107k → 153k (+43%)**. The byte-identity gate `scripts/fast_memo_parity_test.sh` diffs a UA × Origin × Accept-Encoding × scheme matrix memo-on vs memo-off (192 cells, zero differences at the last three review rounds). PGO · 2026-08-28 · parity re-run 2026-09-16 Interactive · `.htaccess` & `fast_memo` readiness inspector Paste or edit an `.htaccess` snippet to inspect `MemoClass` eligibility, `VARY-SET` headers, regex engine split, and directive support: RewriteEngine On SetEnvIfNoCase User-Agent "SemrushBot|AhrefsBot|MJ12bot" bad_bot=1 RewriteCond %{ENV:bad_bot} =1 RewriteRule ^ - [F,L] Header always set Access-Control-Allow-Origin "https://windowsforum.com" env=CORS_OK RewriteCond %{HTTP:Origin} ^https://(www\.)?windowsforum\.com$ [NC] RewriteRule ^ - [E=CORS_OK:1] RewriteCond %{REQUEST_FILENAME} -f [OR] RewriteCond %{REQUEST_FILENAME} -d RewriteRule ^ - [L] RewriteRule ^ index.php [L] Mirrors the parse-time `MemoClass` analysis printed by `httpjet --root /usr/local/lsws check`. ## SetEnvIf & lazy headers `SetEnvIf` / `SetEnvIfNoCase` evaluate with **lazy header access**: the engine hands each rule a callback that materializes headers only if the rule references one, and the attribute compare is case-insensitive without allocating a lowercased copy. A request matching no rule pays nothing. Since 2026-09-16, `SetEnvIf`/`[E=]` cannot set server-computed CGI names (`REMOTE_ADDR`, `HTTPS`, `CONTENT_LENGTH`, …) — those writes are denied. **.htaccess** · SetEnvIf ``` SetEnvIfNoCase User-Agent "bot|crawler" is_bot=1 SetEnvIf Request_URI "\.(?:png|jpg|css|js)$" is_asset=1 RewriteCond %{ENV:is_bot} =1 RewriteRule ^/expensive/ - [E=no-cache:1] # [E=no-cache] also bypasses the page cache ``` ## Front-controller routing XenForo, WordPress and most PHP frameworks rewrite *every* URL to `/index.php`. The engine keeps the **original `REQUEST_URI`** and pins the resolved script separately (`SCRIPT_FILENAME` via the `LsapiScript` extension, `SCRIPT_NAME`/`QUERY_STRING` into the env). PHP sees the front controller; everything downstream still sees the URL the visitor asked for — which is why the page cache keys by the original URI . ## Access control ` ` / ` ` / ` ` / ` ` / ` ` scoping carries `Order` / `Allow from` / `Deny from` and `Require`, ANDed across every enclosing block. A ` ` deny applies to the directory **and everything beneath it**. `Require all granted` grants; `Require all denied` and every other predicate (`ip`, `host`, `env`, `expr`, `method`, …) **fails closed to deny** rather than being approximated — a shipped auth file that httpjet cannot honor 403s rather than opening up. Directory patterns accept mapped filesystem paths as well as the docroot-relative convention. ## HTTP Basic auth `AuthType Basic` + `AuthName` + `AuthUserFile` + `Require valid-user | user …` protect a subtree. Since the 2026-09-06 review the model separates parsed policy from the effective realm: requirements resolve in order across scopes, a metadata-only section cannot erase an inherited requirement, an explicit `Require` replaces it, exact usernames are preserved, and the realm is enforced against the original, rewritten, index, script/PATH_INFO and canonical filesystem targets **before cache lookup**, with a served-target backstop. Unsupported or incomplete active auth fails closed. Valid credentials set `REMOTE_USER`; failures return 401 with the challenge. **.htaccess** · a protected subtree ``` AuthType Basic AuthName "Staging" AuthUserFile /srv/example.test/conf/htpasswd Require valid-user ``` htpasswd hashes: **bcrypt** (`$2y$`/`$2b$`/`$2a$`, `htpasswd -B`, cost capped at 14 — a higher cost is refused before hashing), **APR1-MD5** (`$apr1$`), **MD5-crypt** (`$1$`) and `{SHA}`, verified in constant time; an unknown user pays dummy bcrypt/MD5 work at the file's recorded cost, so user enumeration costs nothing, and Basic-auth attempts are charged per client before hashing (repeated `Authorization` headers are refused). Plaintext entries and traditional DES / `$5$` / `$6$` `crypt()` hashes verify *false* with a parse-time warning. The `AuthUserFile` must resolve **inside the vhost docroot** — or a directory listed in `--auth-user-file-roots` — checked on the opened descriptor's real path (symlinks followed, `.`/`..` rejected in relative paths, read capped at 1 MiB, a planted FIFO cannot block the open); an unreadable or out-of-root file is a 500 with an error-log entry, never a silent 401, and `check` warns about out-of-root paths. Authenticated static responses get `Cache-Control: private, no-store` from automatic expiry when no explicit policy exists (#414). ## Directives reference | Directive | Status | Notes | | RewriteEngine · RewriteBase | supported | Toggle and base-path resolution. | | RewriteCond · RewriteRule | supported | Flags: `L END R R=code F G P PT E= N NC QSA QSD NE C S=n`, `OR` on conditions; `-f -d -l -s -x` file tests. Variables: `REQUEST_URI REQUEST_FILENAME REQUEST_METHOD REQUEST_SCHEME QUERY_STRING THE_REQUEST HTTPS HTTP_HOST HTTP_USER_AGENT HTTP_COOKIE HTTP_REFERER HTTP_ACCEPT REMOTE_ADDR REMOTE_PORT SERVER_ADDR SERVER_NAME SERVER_PORT SERVER_PROTOCOL DOCUMENT_ROOT SCRIPT_FILENAME`, `%{HTTP:Name}`, `%{ENV:NAME}`, `%{LA-U:…}`/`%{LA-F:…}`. `%{TIME*}` and map lookups yield the empty string. Unknown flags are ignored, as Apache does. | | SetEnvIf · SetEnvIfNoCase | supported | Lazy header access; server-computed CGI names are protected. | | Order · Allow from · Deny from | supported | Apache accumulation per scope; `env=`, host and CIDR operands (host/env/IP operands make the chain memo-ineligible). | | Require | partial | `all granted`, `valid-user`, `user …`. Everything else fails closed to deny. | | AuthType · AuthName · AuthUserFile | supported | Basic only; bcrypt, APR1 (`$apr1$`), MD5-crypt (`$1$`) and `{SHA}` hashes. | | Header set / add / append / merge / unset / setifempty | supported | `always`/`onsuccess`, case-insensitive `env=`/`env=!` guards, `%{NAME}e` interpolation, ` `/` `/` ` scopes (merged in Apache order: directory-level across the chain, then ` ` against the resolved target file, then ` `). `merge` deduplicates trimmed comma-separated values. Applied in `finalize`, before cache storage and compression. CR/LF rejected. | | Header echo | partial | Parses without error; emits nothing. | | Header edit / edit* / note · RequestHeader * | ignored | Line ignored. See the header-directive compatibility inventory in the source tree (`docs/header-directive-compatibility.md`) for the exact boundary and the executable fixtures. | | Header condition expr= / value expr= | partial | Only a conjunction of `%{REQUEST_URI} =~ / !~` is evaluated; other expressions are not. Value expressions are not implemented. | | ErrorDocument | supported | Path, inline and external forms; never clobbers an application's own JSON error body. | | DirectoryIndex | supported | Including the `Disabled` form. | | SetHandler · AddHandler · AddType · ForceType · RemoveHandler · RemoveType | supported | `application/x-httpd-php` forces PHP handling / records the mapping; other handlers and `Remove*` record a non-PHP reset; `AddType`/`ForceType` override response `Content-Type`. | | CacheLookup · CacheEnable · CacheDisable · CacheKeyModify | supported | The LSCache directives driving the page cache ; `CacheKeyModify` strips query parameters from the key. | | php_value · php_admin_value · php_flag · php_admin_flag | supported | Per-request PHP ini overrides on the LSAPI special-env path. | | | partial | Common scopes honored; ` ` only for the modeled REQUEST_URI shape; ` ` bodies are evaluated. | | FileETag · H2EarlyHints · esi · AddOutputFilterByType · AllowOverride · Options | no-op | Recognized and dropped. | | ExpiresActive · ExpiresDefault · ExpiresByType | supported | Scoped across directory, ` ` and ` ` blocks; shares `hj-compress`'s `A/M` time grammar and overrides the vhost XML ` ` defaults when active. | Unrecognized directives are ignored leniently so vendor `.htaccess` files remain loadable. The planned first step for the header family is making unsupported forms visible in `check --strict` before adding more mutation behavior. ## Per-context overrides & sub_filter Alongside the per-directory chain, XML contexts rewrite *responses* and override limits for a URI prefix (reference: Configuration → Contexts ): - **`subFilter`** — literal `SEARCH => REPLACEMENT` substitution between the expires and compress stages, 200 responses only, un-encoded bodies of matching `subFilterTypes` (default `text/html`). Buffered bodies and streams both cap filtered output at `subFilterMaxBody` (4 MiB default); a stream past the cap passes through raw, never half-filtered. `Content-Length` is recomputed, ETag dropped, Last-Modified off unless kept. sub_filter paths bypass the page cache (whose hits re-enter transforms), while eligible static responses may use the fast-path memo (which stores and replays finalized bytes without re-filtering). - **`maxReqBodySize`** — per-context body cap enforced at dispatch for all terminals (413). - **`responseTimeout`** (proxy paths) and **`bandwidthLimit`** — override the backend deadline and the server-wide egress rate. ## Reproducible checks **tests** ``` cargo test -p hj-rewrite # parser, evaluator, access, auth, cache directives cargo test -p hj-rewrite --test header_compat # characterization fixtures for the Header boundary cargo test -p hj-rewrite --test redirect_repro bash scripts/fast_memo_parity_test.sh # memo-on vs memo-off byte identity on a real chain ``` ============================================================================== PAGE: Security & access control (https://httpjet.net/docs/security) SUMMARY: Client identity, network ACLs, throttling, GeoIP/ASN rules, PROXY protocol trust, request decompression limits, the body budget, the WAF sidecar, and the PHP jail. ============================================================================== # Security & access control Everything that decides **who the client is** and **who may ask**: the trusted-proxy identity model, network ACLs, the opt-in per-IP throttle, GeoIP/ASN rules from an offline source, PROXY-protocol trust, the mTLS boundary, request-body and decompression budgets, the WAF sidecar seam, and the fail-closed PHP jail. The recurring rule is **fail closed**: a control that cannot evaluate acts restrictive, never permissive. resolve acl rewrite cache dispatch finalize store ## Client identity Every decision that depends on "the client's IP" starts from one resolved value, produced at connection admission and stored in `ReqCtx.client_ip`: > [Diagram: client-ip] From socket peer to client identity. Forwarded headers are evidence, not identity: they count only from a peer an allow rule marks T; from anyone else they are attacker-controlled data and ignored. Every later gate — ACL, geo, throttle, mTLS exemption, logs, the cache key — reads the resolved value. | `useIpInProxyHeader` | Forwarded headers are believed… | | 0 | never | | 1 / 4 | always (only safe when nothing but your proxy can reach the listener) | | 2 (default) | only from trusted peers (`allow … T`); `CF-Connecting-IP` recognized on that path | | 3 | as 2, for IP purposes only | The admin write API classifies `useIpInProxyHeader` and the whole ` ` block as **restart-required**: trust must not change under a running generation without a listener transaction. ## Network ACLs ` ` evaluates `allow`/`deny` CIDR lists most-specific-wins; a trailing `T` on an *allow* rule marks a trusted proxy. ` ` globs are never served, and a malformed glob is a hard start/reload error rather than a warn-and-skip. ` ` mode `2` (owner-restricted) fails closed to never-follow. Static opens use `openat2(RESOLVE_BENEATH)` — plus `RESOLVE_NO_SYMLINKS` when policy denies symlinks — and refuse non-regular files so a planted FIFO cannot park an executor thread; a kernel without `openat2` fails closed to 404. ## Per-client-IP throttle ` ` (or `--per-ip-rate`) caps requests per client IP per fixed window (`perIpRateWindowSecs`, default 1 s); excess requests get 429. The implementation is a sharded fixed-window counter — 64 shards, 4096 tracked IPs per shard — counted exactly once per request at the fast-path serve points and at dispatch. ! Off by default, and fail-open Default 0 (disabled): a shared NAT egress aggregates many real visitors behind one address, so limits are strictly opt-in. Under internal state pressure (a distinct-IP flood) the throttle evicts expired windows and then *admits untracked* — degrading fairness, never availability. `httpjet_throttle_{allowed,rejected}_total` show what it does. ## GeoIP / ASN rules Country and network membership come from `hj-geo`, which has no runtime network access: point ` ` at a CidrList text file and refresh it on your own schedule. Ranges load into sorted, merged interval sets queried by binary search; IPv4-mapped IPv6 addresses are judged in v4 space. **geo source** · CidrList format ``` # country country US 203.0.113.0/24 198.51.100.0/24 country DE 2001:db8::/32 # asn asn 13335 203.0.113.0/24 ``` `geoAllow`/`geoDeny` (ISO-2 labels) and `asnAllow`/`asnDeny` resolve to CIDR sets at state build: **deny first**; an allow list with no deny list is a **fail-closed whitelist**; an unknown label or a missing database is a hard build error, not a silent no-op; `check` warns on geo lists without a database. Both funnels (on-core and bridged) judge the *resolved* client IP, so CDN-fronted visitors are evaluated by their real address. ## PROXY protocol trust A listener with `proxyProtocol=1` requires a PROXY v2/v1 header on every connection, consumed before TLS or the h1/h2c sniff. The listener opt-in *is* the trust decision — never enable it on a public bind (`check` warns). Headers claiming a loopback or unspecified source are refused at the parser (2026-09-16), because those addresses would otherwise unlock the mTLS exemption and the loopback purge/ready endpoints. Details: Protocol stacks . ## TLS & mTLS - **mTLS** — `clientVerify` 1–3 attaches a webpki verifier to the *listener* CA; level 2 requires a client certificate (e.g. Cloudflare Authenticated Origin Pulls). A plaintext request for an mTLS vhost is refused unless from loopback. `--no-mtls` is for local testing only — always confirm the live flags with `systemctl show -p ExecStart` before reasoning about the boundary. - **Revocation** — `crlFile` adds PEM CRL enforcement; absent means none, stated plainly. - **Cipher and resumption discipline** — one TLS 1.3 suite, single-use tickets, 0-RTT off; resumption disabled when OCSP is on. TLS stream splitting was made memory-safe in the 2026-09-06 round. - **One config, two transports** — a certificate, verifier or CRL change cannot land on TCP and miss QUIC. See TLS, ACME & OCSP . ## Request limits & resource fences | Fence | Enforcement | | Header and URL size, connection counts, timeouts | From ` `: `maxReqURLLen`, `maxReqHeaderSize`, `maxConnections`, `connTimeout`, `keepAliveTimeout`, `maxKeepAliveReq`. Incomplete request reads are bounded (2026-09-06); h2 frame/body deadlines apply on both loops. | | Body size | `maxReqBodySize` server-wide, per-context override; 413 on every terminal. Chunked/decoded growth is reserved before allocation. | | Body budget | A process-wide byte budget with owned leases attached to the allocation (including cloned/sliced `Bytes`), released on completion or connection teardown. `httpjet_body_buffer_{bytes,capacity_bytes,rejected_total}`. | | Request decompression | Gzip by default; br/zstd opt-in. Decoded ceiling 16 MiB, 17 MiB codec workspace reservation, zstd window ≤ 2 24 , one coding only; 503 when the reservation fails, 413 past `maxReqBodySize`. → | | Egress bandwidth | Per-connection token bucket (`bandwidthLimit`, context override); H2 paces per connection; H3 unpaced. | | Ranges · sub_filter · `.htaccess` depth | 8 ranges / 8 MiB; 4 MiB buffer then raw pass-through; 64 levels then deny-all. | | H2 / H3 memory | Stream and connection limits, HTTP/2 Bomb mitigation, bounded H3 retained response memory, body-reservation leases released on reset/EOF/error. | | Page cache | 100 MiB object cap; index/file/hot budgets; identity guard; integrity-keyed persistent containers (key loaded through the same private-file validator as the admin/profiling tokens — `O_NOFOLLOW`, single link, owner root or the serving uid, no group/other bits, checked on the opened descriptor; `httpjet_pagecache_integrity_active` on :9090, and `--page-cache-integrity-required` makes an uninstallable key a boot failure); purge endpoint loopback-only and POST-only (other methods get 405 + `Allow`). | | Reload safety | A broken config on SIGHUP keeps the old generation; out-of-range isolation modes are hard errors; the write API rejects rather than partially applies. | | Worker pinning | Core indices clamped to `CPU_SETSIZE`. | ## WAF inspection seam httpjet embeds no rules engine. `--waf-sidecar-addr` enables a bounded HTTP inspection call over a Unix socket (`unix:/path`, peer UID verified via `SO_PEERCRED` against `--waf-sidecar-uid`) or loopback TCP (`ip:port`) that runs after URI normalization and body limits but **before** rewrite, cache lookup, WebSocket/proxy dispatch, LSAPI, FastCGI and static serving — so a warm cache hit is still inspected. The default failure policy is **closed** (503); `--waf-fail-open` must be chosen explicitly and logs every bypass (an explicit 403 from the sidecar always blocks). The full contract is on Extending httpjet . ## The PHP jail suEXEC (` `, off by default, `uidMin`/`gidMin` floors; uid 0 always rejected), chroot (`chrootMode` 0–2) and optional namespaces (`pid` accepted but not honored). A configured jail that turns out unsafe disables PHP for that vhost with a 503 — never unjailed, never falling through to static. Workers never run as root and a setuid `lsphp` is refused. See LSAPI & PHP . ## Control surfaces | Surface | Guard | | `/__hj_cache_purge`, `/__hj_cache_ready` (main listeners) | Raw TCP peer must be loopback — no shared secret; CDN-relayed or LAN peers get 404. The purge endpoint answers POST only (405 + `Allow` otherwise) and honors every `X-LiteSpeed-Purge` header line; the ready probe answers GET/HEAD. PROXY headers cannot claim loopback; a `unix:` listener opens these endpoints to any local process (`check` warns — the socket mode is the boundary). | | `--metrics-addr` (`/metrics`, `/cache-entries`, `/__mimalloc-stats`, `/__alloc-count`, `/debug/pprof/profile`) | Loopback-only by default; `/metrics` is open on loopback, while `/cache-entries`, `/__mimalloc-stats`, `/__alloc-count` and profiling require `Authorization: Bearer ` matching `--profile-token-file` (returning 403 when unset or mismatched). Do not reverse-proxy it. | | `--admin-addr` (read-only `/v1/status`) | Loopback TCP only (argument parsing rejects anything else); Host must be a loopback authority; Origin headers and bodies rejected; no secrets in the schema. | | `--admin-write-addr` (`/v1/revision`, `/v1/config[/validate]`) | Loopback + bearer token from a 0400/0600 single-link file (constant-time compare) + exact `If-Match` revision + explicit resource roots for any filesystem reference. | | LSAPI control and generation files | Root-only control socket; generation file mmap'd read-only by web pools; non-group-writable parent directory required. | | ACME store · DNS token · integrity key · admin token | Owner and mode checks on the descriptor, no symlink following, bounded reads. | ## Audit posture Seven whole-codebase review rounds plus the sixteen-round October bug-bash campaign have landed on top of the August hardening pass: **2026-08-30** (issues #362–#368: no exploitable High, five low/informational fixes), **2026-09-06** (five confirmed medium/low findings: H2/H3 body-reservation leaks, retained-body and gzip-expansion accounting, scoped Basic-auth merging, public caching of authenticated static responses, UA-classification memo dependencies), **2026-09-16** (twelve fixes: h2 monoio deadlines, sendfile stall bounds, unreadable `.htaccess` fail-closed, PROXY loopback claims, `[P]` target screening, CGI-var deny-list, never-root and setuid refusal, isolation-mode hard errors, restart-required trust fields, seccomp ptrace gating, Origin-varied cache entries, CPU_SETSIZE clamp and single Content-Length), **2026-10-06** (issues #560–#575, released as 0.0.4: the H3 connection budget is charged at `Connected` with stateless Retry under handshake pressure, RFC 9112 request-target/Host enforcement at the H1 intake, one rule for which client headers reach PHP, `AuthUserFile` confined to the docroot with constant-cost verification and a bcrypt cost cap, POST-only purge + validated integrity keys + bounded control surfaces, quinn GHSA-53rm-773f-4q8c, new `qpack_decode`/`proxy_protocol_parse`/`lscache_control` fuzz targets, and a daily RustSec dependency-advisory CI job), **0.0.5** (15 findings #579–#593), **0.0.6** (2026-10-13 audit #594–#611: truncated LSAPI/FastCGI response aborts, HPACK over-cap converted to a stream error so dynamic tables stay in sync on pooled CDN connections, H3 streaming backpressure, OCSP responder TLS SAN verification), and **0.0.7–0.1.5** (2026-10-14..24 bug-bash rounds 1–16 and audit #1138: OLS XML wildcard/routing parity, APR1/MD5-crypt htpasswd + per-client auth rate limit, `.htaccess` `Expires*` + `Header merge`, per-URL cookie vary, SHA-256 vary hashing, Unix-socket WAF peer-UID verification, Bearer token gating on `/cache-entries` and `/__mimalloc-stats`, `lsapilib-hardening.patch` + `BEGIN_REQUEST` body scan, time-refilled H2 rapid-reset budget and send-stall resets, rustls 0.23.46 and webpki emSign distrust). Each round's regressions are in the tree; gates re-run after each: full workspace suite (3,019 tests), fast-memo parity, cache xurl/scheme/xhost, h2spec 146/146. i Reporting Security issues: use private responsible disclosure per `SECURITY.md` in the repository, not a public issue. Findings are reconciled against source and regression tests; none of this is evidence of production exploitation or a proof of absence of other vulnerabilities. ============================================================================== PAGE: Page cache (https://httpjet.net/docs/page-cache) SUMMARY: The opt-in LSCache-protocol origin page cache: keys, tiers, admission, purge, the file tier, dictionaries, stale policies, capsules, and generation-scoped namespaces. ============================================================================== # The origin page cache An opt-in full-response origin cache speaking the LiteSpeed cache protocol — `X-LiteSpeed-Cache-Control`, `X-LiteSpeed-Tag`, `X-LiteSpeed-Vary`, `X-LiteSpeed-Purge` — so a site already tuned for LSCache needs no application change. A **hit** returns the stored response without touching the backend; everything else is engineered so the cache can never serve the wrong page. resolve acl rewrite cache dispatch finalize store ## Overview Two crates split the work. `hj-pagecache` is the transport-agnostic store: a sharded byte-weighted LRU index built on `hj-cache`, W-TinyLFU admission, key + vary + tag indexes, TTL and stale windows, the LSCache header classifier, and the HJPC file tier. The binary's `lscache/` module is the request-aware glue: `cache_lookup` runs after rewrite and access control but before any backend; `cache_store` runs after each terminal's `finalize_response`. Both are inert and near-zero-cost without `--page-cache`. Off by default · app opt-in per response Keyed by the ORIGINAL URI · identity-guarded Public · private · shared-path tiers RAM index · hot tier · tmpfs file tier Tag purge · stale-while-revalidate · stale-if-error · single flight Per-vhost dictionaries · survives restarts i Not the fast-path memo Static files have a separate, default-on per-worker finished-response memo (1 s TTL) that sits even earlier on the on-core path — see Rewrite → fast-path memo . It never serves page-cache entries. ## Opt-in by the app The cache never guesses. A response is stored only when the application says so. Per-vhost ` ` (and the six LSWS gates on a context's `cachePolicy`), the `.htaccess` `CacheLookup` directive, and the `[E=no-cache]` env all gate lookups; the response header gates storage. **app → httpjet** · the cache protocol headers ``` X-LiteSpeed-Cache-Control: public,max-age=3600 # public entry, one hour X-LiteSpeed-Cache-Control: private,max-age=120 # per-session entry (needs --page-cache-private; TTL capped 300 s) X-LiteSpeed-Cache-Control: public,max-age=60,stale-while-revalidate=600,stale-if-error=3600 X-LiteSpeed-Cache-Control: no-cache # never store this response X-LiteSpeed-Tag: T1234,forum7 # purge tags X-LiteSpeed-Vary: cookie=xf_style_id # vary the public entry on a declared cookie value X-LiteSpeed-Purge: tag=T1234 # drop every entry carrying this tag (also: * for all) ``` **Standards mode.** `--page-cache-standard-vhosts a.example,b.example` puts listed vhosts into OpenLiteSpeed's `checkPublicCache` shape: a plain `Cache-Control: public, max-age=N` is an opt-in, a default public policy applies when no ` ` block exists, and `CacheLookup` defaults on. It is an allowlist, so a status/admin vhost cannot start caching because it merely marked something public. All other guards still apply. ! Backends cannot self-purge Proxied upstream responses may not assert `X-LiteSpeed-*`; those headers are stripped so a remote backend cannot inject policy or invalidate entries. Only LSAPI/PHP responses and loopback callers of the purge endpoint shape cache behavior. ## The lookup path > [Diagram: cache-lookup] Lookup, tier routing and freshness. Member-cookie requests route to the private tier (or, for allow-listed visitor-invariant endpoints, to the shared public tier); the key is derived from the original URI; a found entry serves only if its stored identity matches; stale entries within their windows serve immediately with a background refresh; concurrent misses collapse onto one render. ## Keys, tiers and the identity guard key & tier explorer — change the request and the app's header method GET HEAD POST scheme https http host path query vary cookie xf_style_id request carries the member cookie (xf_user) --page-cache-private on path matches --page-cache-shared-paths the .htaccess chain reads Origin X-LiteSpeed-Cache-Control on the response Illustrative composition, not the byte layout. The real key is a compact id plus the stored identity; the guard makes any id collision a miss. ! Invariant 1 · key by the original URI The key uses `orig_path`/`orig_query`, never the rewrite-resolved path. XenForo's front controller rewrites *every* page to `/index.php`; keying by the resolved path once collapsed the whole site onto one entry and served the wrong page to everyone. The incident is documented in the crate itself. ! Invariant 2 · the identity guard Every entry records the host + raw request path it was built for; a lookup serves only if both match. A key collision degrades to a miss (`httpjet_pagecache_key_id_collisions_total`), never to wrong content. Three more dimensions were added later and are part of the key today: **per-URL cookie vary** (`X-LiteSpeed-Vary: cookie=NAME` per URL in addition to the vhost-wide `--page-cache-vary-cookies` list, with request-header vary dimensions hashed with SHA-256), **Origin** when the vhost's `.htaccess` chain reads it (2026-09-16 — a reflected `Access-Control-Allow-Origin` no longer replays the first storer's CORS verdict; an absent Origin keeps the historic base key so persisted entries survive), and the **epoch + generation** namespace introduced by the first API publication so an old in-flight render can never overwrite a new generation's entry. Only the canonical Host spelling may read or populate a vhost's entry. | Tier | Routed when | Keyed by | Notes | | **Public** | No member cookie | vhost · scheme · host · original path?query · declared vary-cookie values (· Origin) | `Set-Cookie` and `X-LiteSpeed-*` stripped from the stored copy; a response that sets a private cookie is never stored. | | **Private** | Member marker present (`xf_user`, or a bare `xf_session` with no `xf_user` — session-only logins leave the public tier, 2026-10-04) and `--page-cache-private` | public key + session-cookie value | Never reads or writes public entries in either direction. TTL capped at 300 s. Member POSTs typically emit a tag purge so their own reply is never hidden. | | **Shared paths** | Member cookie, path matches `--page-cache-shared-paths` (`PATH?PARAM` exact or `PATH` prefix), within the sticky canary % | public key | Only for visitor-invariant, HMAC-gated image endpoints — never HTML. Malformed specs abort startup; empty = inert. | | **Bypass** | Non-GET/HEAD, `[E=no-cache]`, no accepted header, member without private tier | — | Counted as `httpjet_cache_bypass`. | ## The store path and admission > [Diagram: cache-store] Store gates. After the app opts in, the response still has to pass the status list, the private-cookie guard, the proxied-header strip, the self-redirect guard, the purge-race check and the object cap, and then earn admission. Only then is the stored copy written — into RAM, or at birth into an immutable tmpfs file. ### W-TinyLFU admission > [Diagram: admission] A fixed-memory frequency sketch rejects one-hit wonders: at the default threshold of 2 a URL is stored on its second sighting. One request is ONE sighting — the on-core fast path probes uncounted and records only a hit it serves. Size weighting adds +1 per 256 KiB. With dictionaries making storage cheap, --page-cache-admit-threshold 1 caches on first sight. Tune by httpjet_cache_hit_ratio_real and httpjet_cache_admit_ratio. The **self-redirect guard** declines an app-marked-cacheable 3xx whose `Location` is the request's own URL (hosts normalized without port, because an h2 `Host` carries `:443` and a generated absolute Location may not); it is re-checked at serve time, and the XF add-on refuses at a third layer. A cacheable origin *redirect* is also de-cached from the CDN's perspective via the deny-CDN-cache transform, so a canonical loop cannot pin itself at the edge. ## Storage tiers & the file tier > [Diagram: cache-tiers] Dual-cap tier model. The RAM index describes every entry; with a file tier, bodies live as immutable HJPC files and are charged metadata-only against --page-cache-mem while their footprint counts against --page-cache-disk-mem. A small hot tier fronts the files. Every teardown path (evict, expire, purge, replace) is synchronous under the shard lock; only the unlink(2) syscall is deferred to just after the lock is released. By default the cache is in-RAM only and flushes on restart. `--page-cache-store-path /dev/shm/jetcache` (or `default`/`jetcache`) makes it **tmpfs-primary** in the LiteSpeed Enterprise style: `PageStore::store` offloads every body to a file at birth in identity form, off the process heap; one immutable file per body version; self-describing HJPC containers authenticated with the `--page-cache-integrity-key`. The LiteSpeed `cacheStorePath` is parsed but deliberately unused so httpjet never collides with a resurrected LiteSpeed. - **Survives restarts.** A background boot scan (up to 8 threads over the fan-out dirs) rebuilds the index from the files — ~300k entries in seconds. `/__hj_cache_ready` returns 503 while scanning and 200 + `x-hj-cache-ready: 1` + `x-hj-cache-loaded: N` when done; probing a URL mid-scan re-primes it and unlinks the persisted copy, so wait for readiness. - **Reclaim invariant.** Live entries resolve to live bodies; a missing file fails closed to a miss and counts `disk_read_err`. Orphan reconciliation reclaims superseded files it can prove are safe to delete and defers the rest. - **Durable purges.** Purges unlink eagerly and retain a purge-all stamp, so old data cannot be resurrected on restart. - **Dictionaries compress at store.** The birth copy is identity; a bounded background task then dict-recompresses it in place against the vhost's own zstd dictionary (~3.5–9× smaller) at store time — never inline on a response, and never triggered by a first hit. An admission filter rejects one-hit public content before it reaches this work, and incompressible Content-Types are never encoded. Judge by `httpjet_pagecache_dict_recompress_{total,skipped_total{reason}}`; the served bytes stay standard zstd/br/gzip either way. budget calculator — how many pages fit? --page-cache-mem (MiB) --page-cache-disk-mem (MiB) --page-cache-hot-mem (MiB) typical page (KB, identity) dictionary ratio × Rules of thumb only (≈6 KB of metadata per entry, ratio 3.5–9× measured on forum HTML). tmpfs *is* RAM — size the disk cap against the box's headroom. Production runs disk-mem 6 GiB, hot-mem 384 MiB. | Flag | Default | Bounds | | `--page-cache-mem` | 128 MiB | RAM index: metadata, variants, any in-RAM body. With a file tier, persisted bodies are charged metadata-only here. | | `--page-cache-disk-mem` | 2 GiB | tmpfs footprint (actual blocks when available); a per-shard disk LRU evicts the least-recently-*served* entry. 0 ⇒ fall back to `--page-cache-mem`. | | `--page-cache-hot-mem` | 192 MiB | In-RAM hot bodies in front of the files: zero-syscall serves; cold hits pay one ~µs tmpfs read + decode. | | `--page-cache-store-path` | *none* | Enables the file tier. Requires the integrity key; deploy scripts gate it behind `ALLOW_PAGE_CACHE_FILE_TIER=1`. | | `--page-cache-dict-vhost` / `--page-cache-dict` / `--page-cache-dict-level` | — | Per-vhost zstd dictionaries (train one per site with `scripts/train-pagecache-dict.sh`) / global fallback / per-vhost internal zstd level (1–19, default 12). Rotation needs a restart; old-generation entries become safe misses (each body records its `dict_gen`). | | `--page-cache-vary-default` | *empty* | `name=value` pairs: a vary cookie carrying the default value (e.g. `xf_style_id=50`) is keyed as if absent. | | `--page-cache-opt-in-status` / `--page-cache-303-prefixes` | *empty* | Opt-in caching of 303/404/410 — only on an explicit `X-LiteSpeed-Cache-Control: public`, TTL capped 300 s, no stale windows; a 303 only under a listed prefix. Default: only 200/301 are cached. | | `--page-cache-stream-fill-vhosts` | *empty* | Vhosts whose cacheable public 200 misses without a Content-Length stream to the client frame by frame and store when the backend finishes (~ms earlier first byte on progressive template output). Errors/short bodies/over-cap serve but never store. | ! Dictionaries can retain content Train only on public, logged-out pages of one vhost; a dictionary trained on another site's boilerplate does little, and one trained on private pages can leak fragments. Served bytes are never dictionary-encoded — a missing compatible dictionary is a miss, never undecodable output. ## Variants, compression and egress The store keeps identity bytes plus **one best precompressed variant** chosen by the request's `Accept-Encoding` under the server priority zstd > br > gzip (when `--cf-send-zstd` is active, background refreshes fill only the zstd variant); a client whose accepted set differs gets a per-serve compress of the identity form. Egress is therefore always standard zstd/brotli/gzip/identity, and a hit re-negotiates encoding — stored responses are never tied to the first client. Hits from the hot tier are zero-copy `Bytes` on h2. See Compression . ## Purge Invalidation is tag-based. The app emits `X-LiteSpeed-Purge: tag=T1234` (or `*`) on a response — every such header line is honored (0.0.5) — or a local process POSTs the loopback endpoint on the main listener. The endpoint answers **POST only** (2026-10-07); a loopback GET or other method gets 405 + `Allow`, and the ready probe takes GET/HEAD. A store that started before a relevant purge and finished after it is rejected (`httpjet_pagecache_store_purge_rejections_total`), so a stale render cannot win a race against the purge. **purge** · loopback-only endpoint ``` curl -s -X POST -H 'x-litespeed-purge: tag=T1234' -o /dev/null -w '%{http_code}\n' http://127.0.0.1/__hj_cache_purge # 204 curl -s -X POST -H 'x-litespeed-purge: *' http://127.0.0.1/__hj_cache_purge # everything ``` The endpoint is gated by the **raw TCP source being loopback** — no shared secret, no per-URL purge, and a CDN-relayed or LAN request gets 404. The former `--cache-peer` fan-out and `--cache-purge-secret-file` are gone. A purge with the file tier unlinks eagerly; a restart cannot resurrect purged entries. ## Stale policies & single flight | Mechanism | Flags | Behavior | | stale-while-revalidate LSCache-authoritative: an `X-LiteSpeed` opt-in does not merge in the standard `Cache-Control`'s `stale-while-revalidate`/`stale-if-error` (2026-10-04); `s-maxage` still implies proxy-revalidate and suppresses these defaults | `--page-cache-stale-default-secs` (0), `--page-cache-stale-max-secs` (86400) | Within the window a stale entry serves immediately and a background refresh (`httpjet_pagecache_refresh_inflight`) re-renders through the same dispatch (so WAF and gates apply). | | stale-if-error | `--page-cache-stale-if-error-default-secs` (0), `--page-cache-stale-if-error-max-secs` (86400) | A backend 5xx (including an lsphp-down 503) on a cacheable public GET serves the retained entry as `X-LiteSpeed-Cache: hit,stale-if-error` with a short `public,max-age=30` egress, Age reset, CDN-CC stripped — the CDN coalesces a 30 s burst but never pins it. Production sets 30 s. Gate: `scripts/cache_sie_test.sh`. | | single flight | built in | Concurrent misses of one key collapse onto one leader via a `Notify`; followers wait rather than each rendering. Registering on the leader's Notify *before* the pre-wait lookup closed a lost-wakeup that once serialized misses. | ## XenForo public-shell capsules application-specific A second, XenForo-shaped tier behind `--xf-capsule` (requires `--page-cache`): a public HTML *shell* for member requests, with the member-specific bits filled client-side by the SessionValidator/CacheOptimizer add-on. It is fail-closed by allow-list — `--xf-capsule-vhosts` empty disables it for every vhost — with route coverage by `--xf-capsule-paths` prefixes or the `all-get-classified` conservative classifier, deterministic canaries for guests and members, and its own stale window (`--xf-capsule-stale-secs`, production 3600 so a missed purge self-heals within an hour). Since 2026-10-04 members are served from the shell-capable *public* entry — `--xf-capsule-dedicated-store` (default off) restores the second dedicated copy. The capsule purge gate (`scripts/cache_capsule_purge_test.sh`) proves a tag purge invalidates the member-served shell. Metrics: `httpjet_xf_capsule_*`. > [Diagram: xf-capsule] Guest vs logged-in member routing across the four cache surfaces. Guests use the public tier directly; logged-in members route visitor-invariant HMAC image paths (--page-cache-shared-paths) to the public tier, eligible forum routes (--xf-capsule) to the anonymized public HTML shell, and remaining opt-in pages to the per-session --page-cache-private tier. ## Generation-scoped namespaces The first successful API publication introduces a response-cache *epoch* derived from the process-bound revision; every later generation (including SIGHUP) keys public, private and capsule entries by epoch + generation while preserving request identity and purge labels. Old in-flight renders finish in their own namespace; restart returns to the on-disk configuration's namespace, and volatile API entries cannot replay there. Obsolete entries age out under the ordinary budgets. The finished-response memo is generation-keyed for the same reason. ## Microcaching today, and what is missing Short TTLs already work: a backend that sends `X-LiteSpeed-Cache-Control: public,max-age=2` gets every guard above and a two-second hit window (`scripts/microcache_policy_example.sh` proves miss→hit→expiry→fresh render, distinct-query isolation and `no-store` respect on an isolated instance). What does not exist yet is a **first-class route-level policy**: named profiles with deterministic precedence, route TTL floors/ceilings, per-profile eligibility, bypass matchers, declarative key dimensions, profile budgets, and bounded collapse/refresh controls. The proposed ` ` contract is specified in the source tree (`docs/microcache-policy.md`) and the parser intentionally rejects it until implemented. ## Observing the cache **observe** · headers, entries, metrics ``` $ curl -sI -H 'Host: example.test' http://127.0.0.1:8080/ | grep -i x-litespeed-cache x-litespeed-cache: miss # ran the backend; stored if admitted $ curl -sI -H 'Host: example.test' http://127.0.0.1:8080/ | grep -i x-litespeed-cache x-litespeed-cache: hit # served from cache (also: hit,stale-if-error) $ curl -s -H "Authorization: Bearer $(cat /etc/httpjet/profile.token)" http://127.0.0.1:9090/cache-entries # totals, per-URL-class counts, top-N by size, ram/disk line, health line $ curl -s http://127.0.0.1:9090/metrics | grep -E '^httpjet_(pagecache|cache)_' $ curl -sk https://127.0.0.1/__hj_cache_ready -o /dev/null -w '%{http_code}\n' # 200 once the boot scan finished ``` Judge health by `httpjet_cache_hit_ratio_real` (non-loopback lookups), `httpjet_pagecache_{hits,misses,stores,purges,evictions}_total`, the per-vhost `httpjet_cache_{hits,misses}_by_vhost{vhost}` split (2026-10-04), `disk_{read,write,full}_errors_total`, `missing_file_invalidations_total`, `poisoned_locks_total` (nonzero = indexes may be inconsistent), and direct content checks — never by a transient raw file count. The full metric list is on Observability . ## Deploy gates These scripts must pass before `--page-cache` ships anywhere; they run against a loopback instance and each catches a class of bug that has actually happened: - **`cache_xurl_test.sh`** Three distinct URLs keep their own content — the front-controller poisoning class. - **`cache_scheme_test.sh`** An HTTP-origin canonical 301 must not poison the HTTPS entry — the "Too Many Redirects" loop. - **`cache_xhost_test.sh`** Foreign-host isolation. - **`cache_private_test.sh`** Private-tier session isolation, plus the shared-paths case when configured. - **`cache_persist_test.sh`** With a file tier: waits on `/__hj_cache_ready` and asserts the boot scan restored entries. It restarts the service — run once (five restarts a minute trip systemd's start limit). - **`cache_sie_test.sh` · `cache_capsule_purge_test.sh`** Stale-if-error (kills its own lsphp mid-test) and the capsule purge invariant. ! A cache bug outlives a rollback A wrong response cached at the CDN edge stays there for its edge TTL. If bad content shipped: confirm the origin is correct on loopback, then purge the CDN zone — a binary rollback does not remove objects already stored upstream. ============================================================================== PAGE: Compression (https://httpjet.net/docs/compression) SUMMARY: Response compression with gzip, brotli and zstd, server-priority negotiation, streaming encoders, precompressed siblings, and bounded request decompression. ============================================================================== # Compression `hj-compress` encodes responses with **gzip**, **brotli** and **zstd**, chosen by server priority but always bounded by the client's `Accept-Encoding`. Buffered bodies encode in one shot; streamed bodies stay streaming; precompressed static siblings skip the work entirely; and a trusted proxy can be handed forced zstd. The reverse direction — **request** decompression — is bounded and mostly opt-in. resolve acl rewrite cache dispatch finalize store ## Codecs and negotiation | Codec | Backend | Default level | XML switches | | zstd | `zstd` | 3 (1–22) | `enableZstdCompress`, `enableDynZstdCompress`, `zstdCompressLevel` | | brotli | `brotli` | 5 (0–11) | `enableBrCompress`, `enableDynBrCompress`, `brCompressLevel` | | gzip | `flate2` | 6 | `enableGzipCompress`, `enableDynGzipCompress` | > [Diagram: compression-negotiation] Choosing an encoder. Bodies of 200 bytes or fewer are never compressed (OpenLiteSpeed parity), the compressible-type set matches LiteSpeed's compressibleTypes, already-encoded and 206 responses are skipped, and every encoded response carries Vary: Accept-Encoding. zstd encoding uses a per-thread `ZSTD_CCTX` pool and pooled transient output buffers (`hj_core::recycle::BODY_BUFS`) so neither contexts nor output buffers page-fault per response. Streamed bodies (`Body::Stream` — LSAPI, proxy, SSE) wrap in a lazy per-chunk `CompressStream`: each inbound chunk is encoded as it arrives, so SSE stays real-time and a slow backend never holds the first byte hostage. ## Trusted-proxy zstd egress `--cf-send-zstd` (or `CF_SEND_ZSTD=1`) forces zstd to a *trusted-proxy* peer regardless of the forwarded `Accept-Encoding`. Cloudflare asks the origin for br/gzip but decodes an origin `Content-Encoding: zstd` and re-encodes per browser at its edge, so this hands the CDN the cheapest-to-produce form (zstd ≪ brotli to compress) without changing what browsers receive. Off by default, officially unsupported CDN↔origin, enabled in production; it never applies to untrusted or direct clients. ## Precompressed siblings If a `.br` or `.gz` sibling of a static file exists in the same directory, httpjet serves it directly with the matching `Content-Encoding` and `Vary: Accept-Encoding` — `gzip_static`/`brotli_static` behavior, brotli preferred. Ranged requests skip the swap so byte ranges stay exact; same-directory placement keeps `accessDenyDir` and symlink verdicts unchanged; a missing sibling falls back to on-the-fly compression at the cost of one stat. ## Interaction with the page cache The page cache keeps identity bytes plus one best precompressed variant chosen under the same priority; a client whose acceptable set differs is re-encoded per serve. Internal dictionary compression is a storage optimization only — dictionaries never reach the wire, and a hit decodes to identity before standard client compression is selected. ## Request decompression httpjet decodes a single supported `Content-Encoding` after the transport has fully framed the request body and before rewrite, cache, WAF, proxy or application dispatch — the same decoder for HTTP/1.1, HTTP/2, h2c and HTTP/3. Gzip is on by default for compatibility; brotli and zstd are opt-in: **serve** · opt-in codings ``` httpjet --root /abs/root serve … --request-decompression-extra br,zstd # accepts br, brotli, zstd; "gzip" is rejected as redundant ``` > [Diagram: request-decompression] Bounded, buffered, fail-safe. Only exactly one coding is decoded; malformed streams and the 16 MiB ceiling forward the original representation unchanged (the destination applies its own policy); the codec workspace and both body forms are charged to the shared budget first. - A successful decode removes `Content-Encoding` and replaces `Content-Length` with the decoded length before any handler sees the request. - Decoded output may not exceed the effective `maxReqBodySize` (413). The absolute decoder ceiling is 16 MiB — at it, httpjet forwards the encoded body rather than exposing a truncated one. - Brotli large-window extensions are disabled; zstd frames are limited to a 2 24 -byte window; `deflate` is not decoded. - Brotli/zstd reserve 17 MiB for decoder tables before constructing the codec; if that plus both body forms cannot be reserved, the request gets 503. **tests** ``` cargo test --locked -p httpjet uring::request_body --no-fail-fast # all three codecs, bombs, malformed input, stacked encodings, workspace exhaustion ``` ## Decode safety Response-side decoding (page-cache variants, dictionary frames) caps `windowLogMax` and bounds decoded output size; a hostile or corrupt frame fails the decode rather than ballooning memory. Compression cost depends on body size, codec, cache state and backend latency — compare representative hits and misses before trading CPU for bytes. In production the origin's compression phase is sampled as `httpjet_phase_compress_seconds`. ============================================================================== PAGE: Configuration (https://httpjet.net/docs/configuration) SUMMARY: The LiteSpeed-compatible XML reference: server, tuning, security, listeners, external processors, upstream groups, virtual hosts, contexts, and httpjet-native extensions. ============================================================================== # Configuration reference httpjet is configured by a **LiteSpeed-compatible XML tree** selected with `--root`: `conf/httpd_config.xml`, the per-vhost files it references, and `conf/mime.properties`. An existing LiteSpeed or OpenLiteSpeed tree works as-is. On that base httpjet layers its own opt-in directives — marked httpjet — which LiteSpeed ignores. Operational overrides live on the command line , not in the XML. ## The configuration tree With no `--root`, httpjet uses `/usr/local/lsws`. The parser (`hj-config`) applies LiteSpeed value semantics faithfully: - Variables `$SERVER_ROOT`, `$HOSTNAME`, `$VH_ROOT`, `$VH_NAME` substitute in `$VAR` and `${VAR}` forms. - Sizes accept `K`/`M`/`G`/`B` suffixes; times are bare seconds. - Bounded integers use OLS semantics: absent, unparsable *or out-of-range* values fall back to the default (never clamped). The isolation modes `setUIDMode`/`chrootMode` are the exception: out-of-range is a hard error (2026-09-16). - Booleans are `1`, `2` or `true` (case-insensitive); `followSymbolLink`/`allowSymbolLink` treat `2` (owner-restricted) as fail-closed never-follow. - Unknown elements are ignored so real trees load; `check --strict` reports the ones that would silently do nothing. **layout** · a server root ``` /usr/local/lsws/ ├── conf/ │ ├── httpd_config.xml # server, tuning, security, listeners, extProcessors, vhost declarations │ ├── mime.properties # mime/type = ext1, ext2 (best-effort) │ └── vhosts/example.test.xml # : docroot, contexts, rewrite, vhssl, cache, logging └── logs/ ``` > [Diagram: config-inheritance] Configuration cascade and parse-time memo classification. Server-wide defaults in httpd_config.xml cascade into each , then into longest-prefix overrides and the per-directory .htaccess chain, where static analysis computes the MemoClass eligibility verdict reported by httpjet check. ## Server and tuning | Directive | Default | Meaning | | `serverName` | `$HOSTNAME` | Server identity (else `localhost`). | | `user` / `group` | `nobody` | Run-as credentials (the systemd unit usually sets these). | | `indexFiles` | `index.html` | Directory-index candidates (vhost may override). | | `useIpInProxyHeader` | `2` | Client-IP trust: 0 never · 1 always · 2 trusted peers only (+ `CF-Connecting-IP`) · 3 as 2 for IP purposes · 4 always. Restart-required through the write API. | | ` ` | `0` | HTTP/3 master switch; also drives `Alt-Svc`. Needs the UDP port published. | | ` maxConnections` | `65000` | Connection cap. | | `connTimeout` / `keepAliveTimeout` | `60` / `5` s | Idle / keep-alive timeouts. | | `maxKeepAliveReq` | `1000` | Requests per keep-alive connection (0 or 1 disables keep-alive, matching OpenLiteSpeed). | | `maxReqURLLen` / `maxReqHeaderSize` | `8192` / `16380` | URL and header-block caps. | | `maxReqBodySize` | `100M` | Server-wide request-body cap (413); per-context override below. | | `perIpRate` · `perIpRateWindowSecs` httpjet | `0` · `1` | Per-client-IP request throttle (0 = off) and its fixed window. 429 on exceed; fail-open under state pressure. | | `bandwidthLimit` httpjet | `0` | Per-connection egress throttle, bytes/s (0 = unlimited). H2 paces per connection; H3 unpaced. | | `maxCachedFileSize` / `totalInMemCacheSize` | `4M` / `4G` | Small-file in-memory promotion threshold and budget (`CacheCaps`). | | `maxMMapFileSize` / `totalMMapCacheSize` | `256K` / `2G` | Parsed for compatibility; the mmap file cache was removed. | | `fileETag` | `28` | ETag bitmask (INODE|MTIME|SIZE → `"size-mtime-inode;;;"`); 0 = no ETag. | | `enable{Gzip,Br,Zstd}Compress` (+`Dyn`) | on | Static/dynamic compression per codec; `zstdCompressLevel` 3, `brCompressLevel` 5. | | ` enableExpires`, `expiresByType` | off | `image/*=A604800, text/css=M86400` (A = access, M = modify). Also honored in `.htaccess` via `ExpiresActive` / `ExpiresDefault` / `ExpiresByType` (sharing the same grammar). Authenticated responses get `private, no-store` when no explicit policy exists. | | ` expireInSeconds` · `privateExpireInSeconds` · `cacheStatusCode` · `enablePostCache` | `900` · `0` · `200,301` · off | Default public/private TTLs and cacheable statuses. `cacheStorePath` is parsed but deliberately unused (see `--page-cache-store-path`). | ## Security directives | Directive | Default | Meaning | | ` ` `allow`/`deny` | — | `ALL` or CIDR lists, most-specific wins. A trailing `T` on an allow rule marks a trusted proxy. | | ` ` | — | Globs never served; a malformed glob is a hard start/reload error. | | ` followSymbolLink` | off | Folded into each vhost's effective `allowSymbolLink`; `2` fails closed. | | `geoipDBFile` · `geoAllow`/`geoDeny` · `asnAllow`/`asnDeny` httpjet | none | Offline CidrList source and label lists; deny-first, allow-only = whitelist, unknown label = build error. → | | ` enable` · `uidMin` · `gidMin` | `0` · `100` · `100` | suEXEC master gate and floors; uid/gid 0 always rejected. | | ` ` `enable`, `mount`, `pid`, `net`, `uts`, `ipc` | off | Namespace isolation for PHP workers; `pid` is accepted but not honored. | | ` ` `CPUSoftLimit`/`CPUHardLimit` | — | Parsed for compatibility; ignored on persistent `lsphp` workers (`check` warns). | | `lsrecaptcha`, `censorshipControl`, permission masks | — | not enforced — `check` warns. | ## Listeners and TLS | Directive | Default | Meaning | | `name`, `address` | `*:80` | `ip:port` / `host:port`. httpjet also `unix:/path` or `uds:/path` for an HTTP-over-Unix-socket listener (TLS over UDS is rejected). Listener names are the stable identity across reloads. | | `secure`, `keyFile`, `certFile`, `certChain` | off | TLS material; `certChain=1` appends `CACertFile` to the served chain. | | `CACertFile` · `clientVerify` · `verifyDepth` | none · `0` · `1` | Client-certificate verification (0 none · 1 optional · 2 required · 3 optional without CA) against the listener CA. | | `crlFile` httpjet | none | PEM CRL/bundle enforced by the client verifier. | | `enableStapling` | `0` | Compatibility only; OCSP needs `--features ocsp` + `--ocsp-responder`. | | `proxyProtocol` httpjet | `0` | PROXY v2/v1 required on *this* listener; fail-closed; never on a public bind. | | `vhostMapList/vhostMap` `vhost`, `domain` | — | Host→vhost; domains lowercased; exact matches take precedence over longest-match leading (`*.example.com`) or suffix (`example.*`) wildcards, then the bare `*` catch-all. | **httpd_config.xml** · a TLS listener with h3 ``` 1 https *:443 1 /etc/letsencrypt/live/example.test/privkey.pem /etc/letsencrypt/live/example.test/fullchain.pem 0 example.test example.test,www.example.test ``` ## External processors An ` ` (server-level ` ` or per-vhost) is a reverse-proxy upstream (`proxy`), an LSAPI app (`lsapi`), or httpjet a FastCGI app (`fcgi`/`fastcgi`). ` ` configures the default PHP handler with the same pool knobs. Unknown legacy types (`rails`, `appserver`, …) keep proxy behavior; `cgi` contexts are not executed. | Directive | Default | Meaning | | `name`, `type` | required | `proxy` · `lsapi` · `fcgi`/`fastcgi` httpjet . | | `address` (repeatable) | required | `ip:port`, `host:port`, `uds:///path`, `http://`, `https://`, and httpjet `h2://` (prior knowledge) / `h2s://` (TLS, ALPN fail-closed). Additional addresses form a group: ordered failover by default, load-balanced with the options below. | | `loadBalancePolicy` httpjet | `primary-first` | `weighted-round-robin` · `weighted-least-active`. Invalid on non-proxy processors. | | `addressWeights` httpjet | all 1 | One integer 1–1000 per address. | | ` ` `mode` · `path` · `host` · `interval` · `timeout` · `rise` · `fall` · `expectedStatus` httpjet | `connect` · `/` · — · 10 · 2 · 2 · 3 · 200 | Active probes; intervals 1–3600 s, timeout ≤ interval, thresholds 1–100, status 200–599. → | | `maxConns` | `100` | Pool cap (LSAPI: worker count source unless `--php-children` overrides). | | `initTimeout` / `retryTimeout` / `pcKeepAliveTimeout` | `60` / `0` / `60` s | Connect timeout / bad-mark cooldown (retry dials fresh; a backend timeout is 504) / idle recycle. | | `respBuffer` | `0` | Parsed; responses always stream (`check` warns if set). | | `clientCertFile` / `clientKeyFile` httpjet | none | Upstream mTLS client identity for `https://`/`h2s://` targets; must lie within an admin resource root when submitted via the API. | | `env` (repeatable) | — | `K=V` worker environment. Server-computed CGI names are protected. | | `path`, `backlog`, `instances`, `autoStart`, `runOnStartUp` | — / `100` / `1` / `0` / `0` | LSAPI worker binary and supervision (`instances>1` warns: one fixed upstream). | | *`phpConfig` additionally:* `phpHandler` (`command`, `suffixes`), `memSoftLimit`/`memHardLimit`, `maxProcessTime` (0 = unlimited; `--php-max-process-time-ms` overrides), `procSoftLimit`/`procHardLimit`, `extMaxIdleTime`, `backlog` default 1024, `detachedMode`. | ## Virtual hosts Declarations live in `httpd_config.xml` (`name`, `vhRoot`, `configFile`, `allowSymbolLink`, `enableScript`, `restrained`); behavior lives in the referenced ` `. `allowSymbolLink` is a tri-state override (file > declaration > server); `restrained` confines the vhost to `vhRoot` and forces symlinks off; `enableScript=0` disables script handlers for the vhost. | Vhost directive | Meaning | | `docRoot`, `indexFiles`, `enableExpires`/`expiresByType` | Root and per-vhost overrides of the server settings. | | ` ` `enable`, `autoLoadHtaccess`, `base`, `rules` | Inline rules run before the `.htaccess` chain. | | ` ` `allowOverride`, `accessFileName` | `allowOverride 0` kills the whole chain; the access file name defaults to `.htaccess`. | | ` ` `keyFile`, `certFile`, `certChain`, `CACertFile` | The SNI identity for this vhost's domains (chain material, not a client-verify root). | | ` ` `enableCache`, `enablePublicCache`, `enablePrivateCache` | Per-vhost page-cache gates (with `--page-cache-standard-vhosts` a missing block gets a default public policy). | | ` ` `suffix`, `type`, `handler` | Suffix routing to `lsapi`/`php` handlers or a `cgi` handler bound to a FastCGI processor. | | ` ` `uri`, `address` | WebSocket relay maps (plain TCP HTTP/1 upstreams). | | ` ` `accessLog`/`errorLog` with `useServer=0`, `fileName`, `rollingSize`, `keepDays`, `logHeaders` httpjet | Per-vhost log files (default roll 50 MB, keep 30 days; `logHeaders` 7 = combined). Log paths are boot-owned. | | `setUIDMode` (0/1/2), `chrootMode` (0/1/2), `chrootPath`, `suexecUser`/`suexecGroup`, ` ` | Per-vhost PHP isolation (needs the server ` ` gate). Out-of-range modes are hard errors. → | | ` ` | Vhost-scoped processors (distinct scope from server-level ones). | ## Contexts Contexts map URI prefixes to terminals (`static`, `proxy`, `redirect`, `php`/`lsapi` via `handler`) and carry per-context overrides: | Context directive | Default | Meaning | | `type`, `uri`, `location`, `handler`, `enabled` | — | Kind, prefix, on-disk location (static), processor name (proxy/lsapi). | | `extraHeaders` | — | Additional response headers for the context. | | `addDefaultCharset` / `addDefaultCharsetCustomized` | off | Charset appended to text types. | | `cachePolicy/*` | on | `checkPublicCache`, `checkPrivateCache`, `respectCacheable`, `enableCache`, `enablePrivateCache`, `enablePostCache` — the six LSWS gates, enforced on both cache tiers. | | `maxReqBodySize` httpjet | inherit | Per-context body cap (413) on every terminal. | | `responseTimeout` httpjet | inherit | Upstream response-head deadline, seconds (proxy paths). | | `bandwidthLimit` httpjet | `0` | Egress throttle overriding the server-wide rate. | | `subFilter` (repeatable) · `subFilterOnce` · `subFilterTypes` · `subFilterMaxBody` httpjet | — · `0` · `text/html` · `4M` | `SEARCH => REPLACEMENT` literal body substitution. The `=>` separator is mandatory — a rule missing it disables the whole context's sub_filter with a warning. → | **vhost XML** · a proxied API prefix with overrides ``` proxy /api/ application 2M 15 http://internal.example => https://example.test text/html,application/json ``` ## Checking a configuration **check** ``` httpjet --root /abs/root check # summary: listeners, vhost map, fast-memo verdicts, warnings httpjet --root /abs/root check --strict # + topology lint, missing docroots, vhost-map-fingerprint; non-zero on findings ``` Run the check under the intended service identity as well as as root: readable-by-root is not proof the runtime user can read certificates or write logs and sockets. The same normalization runs for the write API 's in-memory bundle (`parse_bundle`), which additionally refuses DTDs, more than 64 nesting levels, more than 128 vhosts, and 1 MiB of XML. A config file read from disk is capped at 16 MiB per file (`ConfigError::TooLarge`, 2026-10-06) — a bound on all readers, `check` included. ## Environment variables The full list, with the flags they pair with, is on the CLI reference . The ones most often set in a unit drop-in: `HTTPJET_ACCESS_LOG_FORMAT=json`, `HTTPJET_ERROR_LOG_FORMAT=json`, `HTTPJET_SYSLOG_TARGET`, `CF_SEND_ZSTD=1`, `HJ_FAST_MEMO=0`, `HTTPJET_OTEL=1`, `RUST_LOG`. ============================================================================== PAGE: CLI reference (https://httpjet.net/docs/cli) SUMMARY: Every httpjet subcommand and serve flag, grouped and filterable, plus the environment variables the binary reads. ============================================================================== # CLI reference One binary, four subcommands. `--root` is the only global option. Persistent policy lives in the XML; **everything on the command line is an operational override** — addresses, workers, PHP wiring, cache budgets, feature opt-ins. This page is generated from `httpjet serve --help` of v0.1.5 plus the feature-gated flags. ## Subcommands **httpjet** [--root DIR] ``` httpjet --root /abs/root check [--strict] # parse + summary; --strict lints topology and exits non-zero on findings (deploy gate) httpjet --root /abs/root serve [flags…] # bind the configured listeners and serve httpjet --root /abs/root lsphp [flags…] # run a standalone persistent lsphp pool (the PHP half of a zero-downtime deploy) httpjet lsphp-reload [flags…] # ask a running pool to replace its worker generation without closing the listener httpjet --version # httpjet 0.1.5 ``` | Subcommand · flag | Default | Meaning | | `--root ` (global) | `/usr/local/lsws` | Server root containing `conf/httpd_config.xml`. Use an absolute path. | | `check --strict` | off | Hard-error on shared docroots, missing docroots, vhosts without an exact `vhostMap` domain; warn on apex-without-www, TLS over UDS, PROXY protocol on public binds, geo lists without a db, inert LiteSpeed elements, generic CGI contexts, unsupported auth forms; print `vhost-map-fingerprint` and fast-memo verdicts. | | `lsphp --php-socket` | `/tmp/php8-httpjet.sock` | Socket lsphp listens on — the path `serve --lsphp-external` dials. A systemd-activated fd at the same path is adopted. | | `lsphp --php-children` | `0` | 0 = honor the XML `PHP_LSAPI_CHILDREN`/`maxConns`; >0 overrides; values below 2 normalize to 2. | | `lsphp --control-socket` | ` .control` | Root-only control socket used by `lsphp-reload`. Defaults beside the pool socket so a test pool can never answer the production control path. | | `lsphp --generation-file` | ` .generation` | The 16-byte epoch + marker file external web pools map. Defaults beside the pool socket so a test pool can never advance the production epoch. | | `lsphp-reload --control-socket` | `/run/httpjet/lsphp-control.sock` | Control socket of the running supervisor. | | `lsphp-reload --timeout-secs` | `90` | Maximum wait for candidate readiness and old-generation drain. | ## serve flags Type to filter; click a group to narrow. Defaults are the binary's; *(empty)* means an empty string disables the feature. | Flag | Default | Meaning | | Listeners & transport | | --http-addr | 127.0.0.1:8080 | Address for the plain-HTTP listener | | --https-addr | 127.0.0.1:4443 | Address for the TLS listener. Empty to disable | | --workers | — | Number of accept-loop workers (SO_REUSEPORT sockets). Defaults to CPUs | | --no-core-pinning | off | Do not pin the per-core io_uring transport threads to their CPU cores. Pinning is automatic (and only) when the worker count equals the machine's CPU count; this flag is the kill switch if pinned placement worsens tail latency | | --uring-ring | legacy | io_uring ring setup for the per-core transport runtimes. `legacy` (default) = flagless rings — the A/B measured `coop` (COOP_TASKRUN + SUBMIT_ALL) as noise and `defer` (+SINGLE_ISSUER + DEFER_TASKRUN) as a consistent ~-2% on this virtualized guest; both stay available for other hardware. Ring size always stays default — ring memory is memlock-charged and bigger rings under the prod unit's LimitMEMLOCK bankrupt late runtimes (2026-08-27 crash-loop). Unsupported flags fall back to plain rings per-thread with a warning Values: `legacy, coop, defer`. | | --no-multishot-accept | off | Disable multishot accept: submit one accept SQE per connection (the pre-multishot behavior) instead of one armed SQE per listener | | --http-uds | — | Additionally serve plain HTTP on this unix domain socket (H1 + h2c prior knowledge, one accept core). Peers have no address: ACLs and logs see fabricated loopback and the access log renders `unix:` — the socket file's mode/owner is the real access boundary. A socket-activated fd at the same path (socket unit) is adopted for zero-downtime restarts | | --ktls [ ] | auto | Kernel-TLS on the io_uring TLS path: after the rustls handshake, upgrade the socket to kTLS so H1/H2 serve plaintext over the raw fd (kernel encrypt/decrypt), removing the userspace AEAD copy on large-body egress. Runs only on the io_uring TLS path (the default transport). `auto` (default) requires a `--features ktls` build, a physical default-route NIC, and active `tls-hw-tx-offload`; otherwise it logs one reason and stays on rustls. `on` forces kTLS for isolated benchmarking and fails when the feature is absent; `off` never enables it. A bare `--ktls` remains an alias for `--ktls=on`. TLS 1.3 only (1.2 falls back to userspace); peer KeyUpdate is handled (RX rekey + reply). STAGED — validate on an alt port before production Print help Values: `auto, on, off`. | | Security & limits | | --no-mtls | off | Local testing / explicit rollback only: drop the mandatory Cloudflare client-cert requirement so TLS content can be tested locally without a Cloudflare-issued client cert. Without this flag, clientVerify=2 validates every presented certificate during TLS. A peer that presents none completes the handshake, then the request pipeline rejects it before dispatch (except the loopback exemption) | | --per-ip-rate | — | Per-client-IP request throttle: max requests per client IP per second. 0 (default) = disabled — overrides the ` ` tuning field when set. Protects the origin from single-source floods; a shared NAT egress aggregates many visitors, so size accordingly | | --request-decompression-extra | (empty) | Additional request Content-Encoding values to decode before dispatch. Gzip remains enabled for compatibility; Brotli and zstd are opt-in. Accepted comma-separated values: br,zstd | | --trusted-proxy-idle-secs | 0 | Idle window in seconds for connections whose peer presented a VERIFIED client certificate (in prod, only Cloudflare's authenticated-origin-pull cert). CF never resumes TLS to the origin, so closing a connection it would have reused costs a full TCP + TLS + mTLS handshake. 0 keeps the normal 90 s floor for everyone; uncertified peers always keep it | | --trusted-proxy-idle-max-conns | 6000 | Above this many open connections, certified connections fall back to the normal idle floor (bounds idle state during a connection surge) | | --tls-session-tickets | 2 | TLS 1.3 session tickets issued per full handshake (rustls default 2). 0 stops minting them and shrinks the session cache to TLS 1.2 session IDs: Cloudflare's origin pull never resumes (`httpjet_tls_handshakes_resumed_total` stays 0), so the tickets only cost two records per handshake plus a cached copy of its client-cert chain | | --auth-user-file-roots | (empty) | Extra directories (comma-separated) where a `.htaccess` AuthUserFile may live, besides the vhost docroot | | PHP / LSAPI | | --no-php | off | Do not spawn lsphp; PHP requests fall back to static serving | | --php-children | 0 | LSAPI worker children for the lsphp pool. 0 (default) = honor the lsws config's `PHP_LSAPI_CHILDREN` / `maxConns` (true drop-in). A value > 0 is an explicit override (e.g. to cap concurrency on a shared box); values below 2 are normalized to 2 for persistent supervision | | --php-socket | /tmp/php8-httpjet.sock | Socket path for httpjet's OWN lsphp pool — deliberately separate from the production php8.sock so the two never contend | | --lsphp-external | — | EXTERNAL lsphp: connect to an lsphp master already listening on this socket (run by a separate `httpjet lsphp` process) instead of spawning one. Lets the web tier restart for a zero-downtime binary deploy without cold- starting PHP. Overrides --php-socket. Mutually exclusive with spawning | | --lsphp-generation-file | — | Shared lsphp generation epoch used to invalidate pooled external connections after a supervisor reload or restart. When supplied with --lsphp-external, startup fails if the 16-byte mapping cannot be opened | | --php-max-process-time-ms | — | Total request processing deadline in milliseconds. When exceeded, the handler closes the connection (504); in OWNED/spawn mode the monitor also kills+restarts the worker (Tier-2). Under --lsphp-external (prod) there is no monitor in this process, so only the Tier-1 504 applies — a wedged worker is NOT auto-reclaimed here. 0 = no deadline (matches the LiteSpeed XML `maxProcessTime` 0-means-unlimited convention). Absent ⇒ the XML value if present, else no deadline. Must be conservative: too low kills slow renders | | Page cache | | --page-cache | off | Enable the LSCache-equivalent origin full-page cache (OFF by default; opt-in / R&D). Per-vhost ` ` + `.htaccess CacheLookup` still gate each request, and the app must opt responses in via `X-LiteSpeed-Cache-Control`. With this flag absent the cache is inert | | --page-cache-mem | 134217728 | RAM budget (bytes) for the in-process index: entry metadata + precompressed variants + any in-RAM body. With --page-cache-store-path, persisted (tmpfs) bodies do NOT count here — they are bounded by --page-cache-disk-mem instead — so this should be sized for metadata, not body bytes. Without a file tier it is the whole body budget, as before. Default 128 MiB | | --page-cache-disk-mem | 2147483648 | tmpfs file-tier footprint budget (bytes) for --page-cache-store-path: the disk LRU evicts the least-recently-served entry once charged file footprint under the store path exceeds this. tmpfs IS RAM, so size it against the box's headroom (with dict compression a 2 GiB cap holds the compressed equivalent of ~7–18 GiB of pages). 0 ⇒ fall back to --page-cache-mem. Ignored without a file tier. Default 2 GiB | | --page-cache-store-path | none | Directory for the PERSISTENT tmpfs file tier. The default is "none": bodies stay in the in-process RAM cache and flush on restart. Pass an explicit directory such as /dev/shm/jetcache to opt into the LiteSpeed-Enterprise-style store where bodies live as files and the cache survives restarts via a boot-time warm scan. The LiteSpeed config's cacheStorePath is intentionally NOT used | | --page-cache-integrity-key | /usr/local/httpjet/conf/.jetcache.key | Integrity key used for persistent cache containers | | --page-cache-integrity-required | off | Refuse to start the file tier when the integrity key cannot be installed, instead of running with untagged containers | | --page-cache-hot-mem | 201326592 | Byte cap of the in-RAM hot tier in front of the file store (zero-syscall zero-copy serves for the hottest bodies). Only meaningful with --page-cache-store-path. Default 192 MiB | | --page-cache-dict | (empty) | Path to a shared zstd dictionary used to store cached bodies far smaller. CMS pages share ~60%+ of their bytes as position-shifted boilerplate that only a dictionary captures; with this set, `cache_store` dict-compresses the stored body (the SERVED bytes stay standard zstd/br/gzip — browsers/CF can't decode a private dict). Absent/empty/unreadable ⇒ bodies are stored as plain identity (today's behaviour). Build one from a reviewed local corpus with scripts/train-pagecache-dict-from-files.sh | | --page-cache-dict-vhost | (empty) | Comma-separated `vhost=path` pairs: a PER-VHOST zstd dictionary, trained on that vhost's own content (train one local corpus per vhost). A shared dictionary trained on one site's boilerplate does little for a differently templated site, so each vhost compresses against its own. A vhost not listed here falls back to --page-cache-dict if set, else stores identity (today's behaviour). Example: `forum.example=conf/pagecache-forum.dict,moon.example=conf/pagecache-moon.dict` | | --page-cache-dict-level | (empty) | Comma-separated `vhost=level` pairs overriding the internal zstd level (1-19, default 12) of that vhost's dictionary. A vhost whose compressed bodies round to the same 4 KiB tmpfs pages at a cheaper level gains nothing from 12 (one production vhost costs 5× less at level 3 with no footprint change). A malformed pair aborts startup | | --page-cache-vary-default | (empty) | Comma-separated `name=value` pairs: a vary cookie carrying this value is keyed like no cookie (the app treats the default the same way). E.g. `xf_style_id=50` | | --page-cache-opt-in-status | (empty) | Comma-separated extra statuses (from 303, 404, 410) cached only when the app sends an explicit `X-LiteSpeed-Cache-Control: public`, with the TTL capped at 300 s and no stale windows. Empty = off (only 200/301 are cached) | | --page-cache-303-prefixes | (empty) | Comma-separated request-path prefixes under which an opted-in 303 may be stored (e.g. `/whats-new/`). A 303 anywhere else is never stored | | --page-cache-stream-fill-vhosts | (empty) | Comma-separated vhost names whose cacheable page-cache misses WITHOUT a Content-Length (progressive template output) are sent to the client as the backend writes them and stored once the backend finishes, instead of being buffered whole first. Only identity-encoded public 200s; a body that errors, is cut short or grows past the object cap is served but never stored. Empty = off | | --page-cache-admit-threshold | 2 | (W-TinyLFU) Base admission bar: minimum frequency a cacheable response must show before it is STORED. One request counts as ONE sighting — the on-core fast path probes uncounted and records only a hit it serves. 2 = store on the 2nd sighting (miss-miss-hit; rejects the long-tail of one-hit-wonders — the default). 1 = store on the 1st sighting (miss-hit; cache everything — now cheap with --page-cache-dict, with the store's byte LRU protecting the hot set). Size-weighting adds +1 per 256 KiB on top. Watch httpjet_cache_hit_ratio_real to tune | | --page-cache-vary-cookies | xf_style_variation,xf_style_id,xf_language_id | Comma-separated public-vary cookie names: a cacheable public response may declare `X-LiteSpeed-Vary: cookie=NAME,...` and the entry is keyed by these cookies' values (shared, non-sensitive prefs). Default matches the XenForo style/language set | | --page-cache-private-cookies | xf_session,xf_user | Comma-separated cookie names that forbid public caching if a public response tries to Set-Cookie them (session guard). Default: XenForo | | --page-cache-private | off | Enable the per-session PRIVATE cache tier: an `X-LiteSpeed-Cache-Control: private[,max-age=N]` response from a logged-in request is stored keyed by the session cookie and served only to that same session. OFF by default — without the flag a private opt-in bypasses exactly as before | | --page-cache-private-session-cookie | xf_session | Request cookie whose VALUE keys a private entry's owner (the session) | | --page-cache-private-user-cookie | xf_user | Request cookie whose PRESENCE routes a request to the private tier (the logged-in marker; XenForo's remember cookie) | | --page-cache-shared-paths | (empty) | Comma-separated matchers for visitor-invariant endpoints that MEMBER (logged-in) requests may still read/populate on the PUBLIC cache tier — sharing one entry across guests and every member session instead of duplicating per session and re-rendering each member's first view through PHP. Two matcher forms: `PATH?PARAM` (path equals PATH exactly AND the query carries `PARAM=`, e.g. `proxy.php?image`) and `PATH` (path prefix, e.g. `/wf-unfurl/image`); a missing leading `/` is normalized on. ONLY list endpoints whose bytes are identical for every visitor (HMAC-gated image proxies) — never HTML pages. Malformed specs abort startup. Empty/absent = feature inert (kill switch; members keep today's private routing). GLOBAL like the other --page-cache-private-* knobs — not per-vhost — but a matching request still passes every per-vhost public gate (`vhost` cache policy, `.htaccess` CacheLookup) and every public-store guard before it is served or stored | | --page-cache-shared-paths-canary-percent | 100 | Deterministic percentage (0-100) of member requests matching --page-cache-shared-paths that actually route to the public tier; the rest keep today's private-tier behavior. Sticky per member (bucketed by a hash of the user/session cookie value, exactly like --xf-capsule-member-canary-percent), so one member never flickers between tiers. 100 = all matching requests | | --page-cache-standard-vhosts | (empty) | Comma-separated vhost names put into "standards mode": they honor a standard `Cache-Control: public, max-age=N` as a cache opt-in (OpenLiteSpeed's `checkPublicCache` equivalent), get a default public cache policy when they declare no ` ` block, and default `CacheLookup` to on — so a non-LiteSpeed app (e.g. moon.example) caches like a standards-compliant shared cache. ALLOWLIST, not a global switch: a vhost not listed is unaffected (X-LiteSpeed opt-in only), so opting one content site in can't make an unrelated vhost (status/admin API) cache a response it merely marked public. All other guards (GET-only, Set-Cookie/private-cookie, vary, status, self-redirect) always apply | | --page-cache-stale-default-secs | 0 | Stale-while-revalidate window (seconds) applied when the app declares none. 0 = off (serve only fresh unless the app sets stale-while-revalidate / max-stale) | | --page-cache-stale-max-secs | 86400 | Hard cap (seconds) on the stale-while-revalidate window honored from the app | | --page-cache-stale-if-error-max-secs | 86400 | Hard cap (seconds) on the stale-if-error window honored from the app | | --page-cache-stale-if-error-default-secs | 0 | Stale-if-error window (seconds) applied when the app declares none: how long past freshness a public entry stays servable as a backend-5xx fallback (a brief lsphp/DB outage serves slightly-stale pages instead of error pages). 0 = off (the prior behavior: only an app-declared stale-if-error window arms the fallback) | | XenForo capsules | | --xf-capsule | off | Enable XenForo public-shell capsules. Requires --page-cache; otherwise inert | | --xf-capsule-vhosts | (empty) | Comma-separated resolved vhost names allowed to serve capsules. Empty DISABLES the capsule tier for every vhost (#239 fail-closed default) | | --xf-capsule-paths | /,/forums/,/threads/,/whats-new/,/help/ | Comma-separated request-path prefixes eligible for capsule lookup/store | | --xf-capsule-safe-get-mode | prefixes | Capsule route coverage. `prefixes` honors --xf-capsule-paths; `all-get-classified` uses a conservative unsafe-route classifier Values: `prefixes, all-get-classified`. | | --xf-capsule-stale-secs | 86400 | Stale-while-revalidate window for capsule shell entries. This is applied to public-shell capsule stores even if the app declares a smaller window | | --xf-capsule-canary-percent | 100 | Deterministic percentage of eligible requests that may be served a capsule hit | | --xf-capsule-members | off | Allow explicitly-marked member-cookie requests to receive public-shell capsules | | --xf-capsule-member-canary-percent | 0 | Deterministic percentage of eligible member-cookie requests that may receive capsules | | Performance knobs | | --no-fast-memo | off | Disable the finished-response memo on the on-core static fast path (default ON; per-thread, 1 s TTL, byte-identity-gated — see `pipeline::fast_memo`). Env `HJ_FAST_MEMO=0` is equivalent | | --cf-send-zstd | off | Compress responses to a trusted-proxy (Cloudflare) peer as zstd, ignoring the forwarded Accept-Encoding. CF asks the origin for br/gzip but decodes an origin `Content-Encoding: zstd` and re-encodes per browser at its edge — so this hands CF the cheapest-to-produce form (zstd ≪ brotli to compress) without changing what browsers receive. Officially unsupported CF↔origin; opt-in, default off. Never applies to untrusted (direct) clients. Off ⇒ byte-identical to today | | --rewrite-outcome-ttl-ms | 1000 | TTL in milliseconds for the rewrite-outcome cache (memoized rewrite decisions for `path_cacheable` chains, keyed by vhost/scheme/ method/host/path/query + any keyable header vars). 0 disables the cache entirely. The default (1000 ms) matches the historical hard-coded TTL — it bounds `-f`/`-d` filesystem staleness exactly like the StatCache and lets `.htaccess` edits take effect within ~1 s | | --rewrite-ua-classify | off | Key UA-reading rewrite chains by the bitmap of matching User-Agent RewriteConds instead of the raw UA string, when parse-time analysis proves the outcome depends on the UA only through those match results. Collapses real-world UA diversity (near-zero hit rate on raw UAs) onto a handful of outcome-cache entries. Default OFF — enabling is a deploy-time decision | | Memory | | --mimalloc-trim-secs | 120 | Interval (seconds) for the background mimalloc OS-trim that calls mi_collect(force) to hand retained/cold arena pages back to the OS (counters the swap retention seen in prod after a burst). 0 = disable | | --mimalloc-trim-threshold-mib | 768 | Only run the mimalloc trim (periodic and after a connection close) when process RSS+swap (read from /proc/self/status) is at least this many MiB; 0 = always. Lets a healthy process skip the collect entirely. Default 768: above a cache-less steady state (~360 MiB). Size it above the deployment's healthy peak (a large page cache holds 1-2 GiB) so only genuine post-burst arena retention trips the safety-net reclaim | | Observability | | --metrics-addr | 127.0.0.1:9090 | Loopback-only Prometheus metrics endpoint address. Empty to disable. Exposes the page-cache hit/miss ratio (the key operational number once --page-cache is live) plus request/connection counters | | --telemetry-file | (empty) | Append a cumulative per-request telemetry snapshot row to this file every --telemetry-flush-secs: durability across restarts + a self-contained time-series for the two-node A/B. Empty = no disk flush (the in-RAM aggregates + the :9090 endpoint still work). e.g. logs/telemetry.tsv | | --telemetry-flush-secs | 30 | Snapshot flush interval in seconds (only used when --telemetry-file is set). Fixed RAM; ~1 MB/day on disk at the default | | --php-slow-log | (empty) | Per-request PHP render log: a TSV line for every LSAPI request whose render TTFB exceeds --php-slow-threshold-ms, plus a 1-in-128 sample of ALL PHP requests as the population baseline — the per-URL / user-class attribution behind the lsapi_ttfb histogram. Rolls at 64 MiB, gzip archives, 30-day prune. Empty = disabled. e.g. logs/php-slow.tsv | | --php-slow-threshold-ms | 50 | "Slow" threshold in milliseconds for --php-slow-log | | --request-id-header | off | Echo the per-request correlation id (also in the access/error/php-slow logs) as an `X-Request-Id` response header, so a client/CDN can join a response to the server-side logs. Off by default | | --profile-token | (empty) | Bearer secret required for profiling and sensitive metrics diagnostics. Empty disables those routes; ordinary metrics scraping stays available. Visible in /proc/*/cmdline; prefer `--profile-token-file` | | Admin API | | --admin-addr | — | Opt-in read-only operational JSON endpoint (loopback TCP only) | | --admin-write-addr | — | Authenticated application-config writes on a separate loopback listener | | --admin-token-file | — | Private file containing a 64-character lowercase hexadecimal bearer token | | --admin-resource-root | — | Allowed existing filesystem root for submitted resources; repeatable | | WAF sidecar | | --waf-sidecar-addr | — | Opt-in HTTP request-inspection sidecar: `unix:/path` (its listener's uid is verified) or a loopback `ip:port` (unauthenticated). The sidecar must return 204 to allow or 403 to block; absent keeps the WAF seam inert | | --waf-sidecar-uid | — | The uid a `unix:` sidecar must run as. Default: root or this process's uid | | --waf-sidecar-path | /inspect | | | --waf-timeout-ms | 100 | | | --waf-body-max | 65536 | | | --waf-concurrency | 128 | | | --waf-fail-open | off | Continue when the configured WAF sidecar is unavailable or malformed. Default is fail-closed (503); this weaker policy requires explicit opt-in | | ACME (--features acme) | | --acme-directory | — | ACME directory URL (HTTPS). Requires --acme-domains, --acme-store and --acme-accept-terms; nothing runs without all four. | | --acme-domains | — | Comma-separated DNS identifiers (1–100, ASCII A-labels). Wildcards need DNS-01. Quote a wildcard in the shell. | | --acme-store | — | Pre-provisioned private directory, mode 0700, owned by the service uid; holds atomic account/order/certificate snapshots and the DNS cleanup journal. | | --acme-accept-terms | off | Explicit agreement to the CA terms of service. | | --acme-bootstrap | off | Start with no listener default certificate; TLS handshakes fail until issuance succeeds. No self-signed placeholder. | | --acme-reconcile-order | — | One-shot recovery of an ambiguous newOrder from the same account's CA records. | | --acme-dns-webhook | — | Provider-neutral DNS controller endpoint (selects DNS-01). Requires --acme-dns-zones. | | --acme-dns-zones | — | Zone allowlist for TXT operations. | | --acme-dns-token-file | — | Bearer token for the controller: regular single-link file, mode 0600, 16–4096 printable characters. | | --acme-test-mode | off | Loopback CA HTTP and alternate loopback validation ports for fixtures only; serving addresses must be loopback. | | --acme-test-ca-root · --acme-test-issuer-root · --acme-dns-test-root | — | Replace trust roots for the test CA / issuance / DNS controller without disabling verification (test mode only). | | OCSP (--features ocsp) | | --ocsp-responder | — | Explicit HTTP(S) responder for every enabled certificate issuer. Applies regardless of the XML enableStapling boolean; disables TLS resumption. | | --ocsp-required | off | Refuse new TLS handshakes without a fresh authenticated GOOD response (also implied by Must-Staple). | | --ocsp-test-mode | off | Permit a literal-loopback responder, with loopback serving addresses only. | | --xf-capsule-dedicated-store | off | Also store every capsule-capable page under a DEDICATED (admission-exempt) key. Off by default since 2026-10-04: members are served from the shell-capable PUBLIC entry, which keeps one copy per page and lets the admission gate govern every store | ! Removed flags `--cache-peer` / `--cache-peer-fill*` (cross-node cache peers) and `--cache-purge-secret-file` are gone; the purge endpoint is loopback-gated. `--io-uring` / `--io-uring-h3` never existed as opt-ins — the io_uring transport is unconditional. ## Environment variables | Variable | Default | Meaning | | `HTTPJET_ACCESS_LOG_FORMAT` | combined | `json` selects JSON access lines (fixed field order: `ts, client_ip, remote_user, method, uri, protocol, status, bytes, referer, user_agent, reqid`). | | `HTTPJET_ERROR_LOG_FORMAT` | text | `json` selects JSON error events (`schema_version 1`, credential redaction, bounded fields). Independent of the access format; does not change the stdout/journald tracing formatter. Restart to change; rotate first to avoid mixed files. | | `HTTPJET_ACCESS_ROLLING_BYTES` / `HTTPJET_ACCESS_KEEP_DAYS` | 50 MB / 30 | Unified access-log rolling size and retention. | | `HTTPJET_LOG_DIR` | unset | Redirect every opened log file (unified and per-vhost) into this directory so isolated test or PGO runs never touch production log paths. | | `HTTPJET_URING_NO_IOWAIT` | unset | `1` sets `IORING_FEAT_NO_IOWAIT` on supported kernels so network-wait time on io_uring rings is not accounted as system iowait. | | `HTTPJET_SYSLOG_TARGET` | unset | Mirror every rendered access line to syslog: `udp://host:514`, bare `host:port`, or a unix-dgram path such as `/run/systemd/journal/syslog`. Sent from the writer task; failures counted, never fatal. | | `HTTPJET_SYSLOG_FACILITY` · `_SEVERITY` · `_RFC` · `_HOSTNAME` · `_APP_NAME` | `daemon` · `info` · `5424` · server name · `httpjet` | Syslog framing (`3164` for the BSD format). | | `CF_SEND_ZSTD` | unset | `1` ≡ `--cf-send-zstd`. | | `HJ_FAST_MEMO` | `1` | `0` ≡ `--no-fast-memo`. | | `HTTPJET_OTEL` · `HTTPJET_OTEL_METRICS` | unset | `1` enables OTLP tracing / metrics (`--features otel`; startup fails if the feature is absent). | | `HTTPJET_OTEL_SAMPLE_RATIO` | `0.01` | Root-span ratio sampler, 0–1. | | `HTTPJET_OTEL_TRUSTED_PARENTS` | empty | Comma-separated exact gateway IPs (≤128) allowed to supply a W3C `traceparent`; everyone else starts a new root. | | `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` · `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` (+ SDK header settings) | — | Complete collector URLs, e.g. `http://127.0.0.1:4318/v1/traces`. | | `RUST_LOG` | `info` | Tracing filter for the error log / journal, e.g. `RUST_LOG=debug,hj_rewrite=trace`. | | `HJ_ROOT`, `HJ_URING_*`, `HJ_PEBBLE_*`, `HTTPJET_*_TEST_CHILD` | — | Test-harness variables only. | ## Exit behavior and signals - **SIGHUP**: Reload configuration and certificates through the publication coordinator; a bad config keeps the current generation (see Admin API & reload ). - **SIGTERM / SIGINT**: Graceful shutdown: stop accepting, GOAWAY h2, drain within the unit's `TimeoutStopSec`, close the admin coordinator, flush logs and telemetry, bounded OTel shutdown. - **Argument validation**: Public admin binds, missing companion options (`--admin-token-file`, `--admin-resource-root`, ACME requirements), malformed `--page-cache-shared-paths` or `--page-cache-vary-default` / `--page-cache-dict-level` pairs, an unopenable generation file, or a rejected WAF configuration fail startup before any socket is bound; `--page-cache-integrity-required` additionally refuses to start the file tier when the integrity key cannot be installed. - **Socket activation**: Inherited fds are adopted by address/path; their count must match `--workers` for TCP listeners. ============================================================================== PAGE: Operations (https://httpjet.net/docs/operations) SUMMARY: Systemd socket activation, the persistent PHP pool, safe changes, deploy gates, rollback, kill switches, and the hardened OCI example. ============================================================================== # Operations & deployment httpjet is designed for a conventional Linux service: a dedicated account, the LiteSpeed-compatible tree, **systemd socket activation** so a binary deploy never refuses a connection, and a **separate persistent PHP pool** so a web-tier restart never cold-starts opcache. The checked-in units are generic references; a hardened OCI example covers container adoption. ## Installation No signed packages are published yet; build from source and install the binary at a stable path *outside* Cargo's `target/` so a later `cargo clean` cannot delete what a running unit depends on. source install ``` cargo build --locked --release -p httpjet sudo useradd --system --home-dir /var/lib/httpjet --no-create-home --shell /usr/sbin/nologin httpjet sudo install -d /usr/local/httpjet/bin /usr/local/httpjet/conf /usr/local/httpjet/logs sudo install -m0755 target/release/httpjet /usr/local/httpjet/bin/httpjet # an existing LiteSpeed tree works as-is — validate it first, as root AND as the service user sudo /usr/local/httpjet/bin/httpjet --root /usr/local/lsws check --strict sudo -u httpjet /usr/local/httpjet/bin/httpjet --root /usr/local/lsws check --strict # minimal loopback example unit (no TLS, no PHP, no cache) sudo install -m0644 packaging/systemd/httpjet-example.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable --now httpjet-example ``` ! Review before start The example units bind loopback and use example document roots. Set listeners, domains, docroots, TLS material, writable paths and the service identity deliberately before enabling a public service. A permission-preparing `ExecStartPre` that runs as root and then drops to the unprivileged account is the reference shape. ## The systemd model > [Diagram: systemd-units] Five units. The socket units own the listening fds; the boot gate keeps the public listeners closed until the PHP pool can answer; the web tier adopts the fds and dials the pool. Restarting the web tier or reloading the pool never closes a listener. | Unit | Role | | `httpjet.socket` | N× `:80` TCP, N× `:443` TCP, N× `:443` UDP with `ReusePort=yes`, `Backlog=8192`. The multiplicity reproduces the per-worker SO_REUSEPORT fan-out, so N must equal `serve --workers`. Static (not enabled under `sockets.target`); pulled in by the service after the gate to avoid a cold-boot ordering cycle. | | `httpjet-boot-ready.service` | Boot-only oneshot gate, availability-first: the `Type=notify` LSAPI pool is the one hard dependency; databases/caches are ordered but not required — it waits up to 300 s, then proceeds with a loud warning so a cold boot with a slow database still serves static + warm cache + stale-if-error. Later backend maintenance cannot tear down the sockets. | | `httpjet.service` | The web tier: `httpjet --root … serve --lsphp-external … --lsphp-generation-file …` under a dedicated user, `Restart=always`, `OOMScoreAdjust=-500`, `LimitNOFILE=1048576`, `TimeoutStopSec=20`, `Conflicts=lshttpd.service`. Effective flags come from drop-ins in `httpjet.service.d/` — check `systemctl show -p ExecStart --value httpjet.service` before reasoning about them. The repository ships a validated **hardening drop-in** (`systemd/httpjet.service.d/zz-hardening.conf`: `ProtectSystem=strict` with logs-only `ReadWritePaths`, `PrivateTmp/PrivateDevices`, `ProtectProc=invisible`, empty capability set, `MemoryDenyWriteExecute`, `RestrictAddressFamilies`, and a **log-only seccomp stage** that filters the dangerous syscall groups while logging — enforce `@system-service` only after a clean soak), measured `systemd-analyze security` 9.0 → 1.9; it is the reference copy, not applied by the checked-in unit. | | `httpjet-lsphp.socket` | Owns the LSAPI Unix listener (mode 0600, backlog 8192, `RemoveOnStop`). Never restart it during a reload or supervisor deploy. | | `httpjet-lsphp.service` | `httpjet lsphp …`, `Type=notify` (active only after a marker-attributed LSAPI response), `ExecReload=httpjet lsphp-reload …`, its own identity, `RuntimeDirectory=httpjet` for the control socket and generation file. | | `httpjet-edge-firewall.service` | Optional oneshot edge firewall (`scripts/httpjet-edge-firewall.sh`) for origins without a cloud firewall: restricts new `:80`/`:443` connections to Cloudflare IPv4 + private ranges (keep its list in step with the XML `T` entries). | i Scope of the zero-loss guarantee Socket activation eliminates RSTs of *new and queued* connections across a deploy. Requests the old process has already accepted — h1/h2 mid-serve and especially h3/QUIC streams — still drain under `TimeoutStopSec`. "Never drops a connection" means never RSTs a new or queued one. ## The persistent PHP pool Static and proxy deployments need no PHP. For LSAPI, run the pool as its own unit and point the web tier at it; the units above wire the socket, the root-only control socket and the 16-byte generation file. Routine opcache recycle is `systemctl reload httpjet-lsphp.service` — synchronous and candidate-first. Reserve a hard restart of the pool *service* for supervisor binary/unit changes; verify the listener inode is unchanged before and after. Reference: LSAPI & PHP . replace the supervisor without closing the listener ``` before=$(stat -Lc '%d:%i' /run/httpjet/php.sock) systemctl daemon-reload systemctl restart httpjet-lsphp.service # the SERVICE only — never the .socket test "$(stat -Lc '%d:%i' /run/httpjet/php.sock)" = "$before" test "$(stat -Lc %s /run/httpjet/lsphp.generation)" = 16 ss -xlnH | awk '$5 == "/run/httpjet/php.sock" {print $4}' # Send-Q should equal the XML backlog ``` ## Safe change workflow - **Edit in a reviewable change** Configuration tree or application content. - **`check --strict`** The deploy gate. SIGHUP with a broken config keeps the old generation, but the check should catch it first. - **Test in isolation** Alternate ports, a separate PHP socket, a separate cache store path, low worker/child counts. Kill test instances by exact PID; a leaked SO_REUSEPORT instance silently shares the port with independent cache and htaccess state. - **Apply** `systemctl reload httpjet` (SIGHUP) for vhost/rewrite/access/expires/tuning/certificate changes; `systemctl restart httpjet` for the binary or boot-owned resources; the write API for validated in-memory replacement. - **Verify on loopback, then watch** `curl -sk -H 'Host: …' https://127.0.0.1/…`, the journal, `httpjet_requests_total` and 5xx counters. Purge any outer cache if an incorrect cacheable response reached it. ## Binary deploy and rollback > [Diagram: deploy-flow] The deploy gate chain. The repository's scripts/deploy-lscache.sh automates this for one deployment (strict check, co-listener guard, external-lsphp recurrence guard, health probe with automatic revert to the previous binary, capsule purge gate); the shape is what matters for any deployment. - **Kill switch: cache**: Drop `--page-cache` from the unit (a drop-in override) and restart; the cache code is inert without the flag. - **Kill switch: capsules**: Drop the `--xf-capsule*` flags and restart. - **Kill switch: fast memo**: `--no-fast-memo` or `HJ_FAST_MEMO=0`. - **Kill switch: WAF**: Remove `--waf-sidecar-addr` (the seam is completely inert without it). - **Rollback**: Keep the previous binary beside the current one (`bin/httpjet.prev`); restore and restart. A rollback does not remove bad objects already stored in the origin cache (purge) or at the CDN edge (purge the zone). - **Start limits**: `StartLimitBurst=5` per 60 s: more than five restarts a minute trips systemd and takes the service down until `systemctl reset-failed`. The persistence gate restarts the service — run it once. The production build uses profile-guided optimization (`scripts/build-pgo.sh`: instrument → drive a representative loopback workload → rebuild). PGO scratch data is kept off tmpfs. Never benchmark a non-PGO binary against a PGO one. ## What SIGHUP reloads Vhost maps, rewrites, document roots, contexts, access/expires/tuning settings, cache policy gates, and TLS certificates on an unchanged listener topology. Signal streams (`SIGHUP`, `SIGUSR1`, `SIGUSR2`) are registered immediately at runtime startup so a certbot or reload signal arriving during boot queues cleanly instead of terminating the process. A change that needs a restart (listener addresses, QUIC enablement, PHP topology, cache-store identity, privileges, log paths) is logged as "requires a restart — keeping current config" and nothing changes. A reload that would 404 a mapped vhost because its per-vhost file failed to load is refused. The dictionary registry is retained across SIGHUP (rotation needs a restart). Full semantics: Admin API & reload . ## Observability checklist the loopback-only surfaces ``` curl -s http://127.0.0.1:9090/metrics | grep '^httpjet_' # Prometheus text curl -s -H "Authorization: Bearer $(cat /etc/httpjet/profile.token)" http://127.0.0.1:9090/cache-entries # cache totals, budgets, health line curl -s http://127.0.0.1:9091/v1/status | jq . # with --admin-addr: generation, fingerprint, upstream health curl -sk https://127.0.0.1/__hj_cache_ready -o /dev/null -w '%{http_code}\n' journalctl -u httpjet -n 100 ``` Metric families, log formats, the request-id header, telemetry TSV, PHP slow log, OpenTelemetry and profiling are all on Observability . Pair the counters with a process monitor (the tree ships a `monit` check whose start timeout stays above the boot gate's worst case). ## Hardened OCI example `packaging/oci/` is an adoption example, not a prescription. The image runs as numeric UID/GID `10001:10001`, binds unprivileged ports, needs no capabilities; the compose file adds a read-only root filesystem, drops every capability, sets `no-new-privileges`, and mounts a small `noexec,nosuid,nodev` `/tmp`. Because Moby's default seccomp allowlist omits io_uring, `seccomp-httpjet.json` is the pinned Moby default plus only `io_uring_setup`, `io_uring_enter` and `io_uring_register` (ptrace/`process_vm_*` gated on `CAP_SYS_PTRACE`). | Path | Access | Purpose | | `/run/httpjet-certs` | read-only | TLS certificate + key (no key ships in the image; generate one for `example.test`). | | `/var/lib/httpjet` | read-write | Page-cache files and the integrity key. | | `/etc/httpjet/litespeed/logs` | read-write | Access/error logs. | | `/etc/httpjet/litespeed/{conf,example}` | image read-only | LiteSpeed XML and static content. | | `/tmp` | bounded tmpfs | Temporary runtime files. | build, run, smoke ``` docker compose -f packaging/oci/compose.yaml up --build # publishes 8080/tcp, 8443/tcp AND 8443/udp docker run … --security-opt seccomp=/abs/path/seccomp-httpjet.json … # a hand-written run must pass the profile explicitly bash scripts/oci-smoke.sh # read-only root, UID 10001, caps dropped, real HTTP/3 through the UDP port ``` A seccomp denial is a startup/test failure — do not work around it with `--privileged`, `--cap-add` or `seccomp=unconfined`; narrow the runtime policy or pick a supported host. Rootless engines may add memlock, UDP or io_uring restrictions of their own. The image health check calls `/__hj_cache_ready`, so readiness means the cache boot scan finished, not that a user URL was warmed. ## Release and upgrade discipline A public release must provide portable x86_64 and aarch64 artifacts, checksums, signatures or attestations, an SBOM, the GPL-3.0 license text and a documented Linux/kernel/libc floor; architecture-tuned PGO helpers must not be uploaded as generic binaries (production builds and gathers PGO natively on each architecture). Upgrades preserve the tree, dictionaries and state; stage and validate the new binary before restart; keep a known-good binary. Dictionary rotation: write the new file atomically, validate, restart, expect old-generation cache entries to become safe misses. Dependency upgrades run through `scripts/update-deps.sh` and the full suite (the 0.1.x sweeps refreshed tokio 1.54, rustls 0.23.46, rustls-webpki 0.103.16 + webpki-roots 1.0.10, fancy-regex 0.20, parking_lot 0.13 and zstd 0.14.3; a daily dependency-advisory job fails the build on any RustSec finding). ============================================================================== PAGE: Observability (https://httpjet.net/docs/observability) SUMMARY: Prometheus metrics, the cache-entries report, access and error logs in text or JSON, syslog, request ids, telemetry, the PHP slow log, OpenTelemetry, and profiling. ============================================================================== # Observability Everything httpjet tells you about itself: **Prometheus metrics** on a loopback listener, the **cache-entries** report, access and error **logs** in text or JSON with an optional syslog tap, per-request **correlation ids**, the telemetry TSV and PHP slow log, opt-in **OpenTelemetry** traces and metrics, and the profiling endpoints. All control surfaces are loopback-only by design. ## Prometheus metrics `--metrics-addr` (default `127.0.0.1:9090`; empty disables) serves `/metrics` in the Prometheus text format. Over 130 series today; the families that matter, grouped: | Metric | Type | Meaning | | Requests & connections | | httpjet_requests_total · httpjet_active_requests · httpjet_active_connections | counter / gauge | Served requests, requests in the response-head phase, open connections. | | httpjet_requests_by_vhost · httpjet_responses_by_vhost · httpjet_request_duration_by_vhost_seconds | labeled | Per-vhost volume, status class and handle() wall time. | | httpjet_status_{2xx,3xx,4xx,5xx} · httpjet_status_{400,403,404,410,414,429,500,502,503,504} | counter | Status classes and the interesting exact codes. | | httpjet_proto_{h1,h2,h3} · httpjet_served_{static,php,proxy} | counter | Wire protocol and terminal mix. | | httpjet_request_duration_seconds · httpjet_phase_{rewrite,cache_lookup,compress,upstream}_seconds | histogram | Total handle() time and sampled per-stage costs (upstream = proxy checkout → response head). | | httpjet_tls_handshakes_{full,resumed}_total · httpjet_conn_{idle,peer}_closes_total | counter | Full vs resumed TLS handshakes; connection closes by idle timeout vs peer FIN/RST. | | httpjet_h2_rapid_reset_drops_total · httpjet_h3_send_stall_resets_total | counter | H2 connections dropped on rapid-reset budget exhaustion; H3 response streams reset on send stall. | | httpjet_throttle_{allowed,rejected}_total | counter | Per-IP throttle decisions (0 when disabled). | | httpjet_body_buffer_{bytes,capacity_bytes,rejected_total} | gauge / counter | The process-wide request-body budget. | | Fast path & rewrite | | httpjet_fast_memo_{hits,stores,ineligible}_total | counter | Finished-response memo activity; *ineligible* = the chain refused the store (see `httpjet check`). | | httpjet_fast_cookie_{none,member_session,benign_only}_total · httpjet_fast_bridge_total{reason} | counter | Cookie shape of fast-path GET/HEADs and attribution of why a GET/HEAD fell off the on-core fast path into the tokio bridge. | | httpjet_rewrite_outcome_{hits,misses,uncacheable}_total · httpjet_rewrite_ua_classify_entries | counter / gauge | Rewrite-outcome cache and UA-classification table size. | | Page cache | | httpjet_pagecache_{hits,misses,stores,purges,purges_received,store_purge_rejections,expired_misses}_total · httpjet_pagecache_hit_ratio · httpjet_cache_hit_ratio_real | counter / gauge | Core activity; `_real` excludes loopback probes; `httpjet_cache_{hits,misses}_by_vhost{vhost}` split it per vhost and `httpjet_pagecache_evictions_total` counts size/TTL teardowns. Telemetry twins `httpjet_cache_{hits,misses,bypass,stale_serves,sie_serves,admitted,rejected}` and `httpjet_disp_*` (why a response was or wasn't stored) plus `httpjet_cache_admit_ratio`. | | httpjet_pagecache_entries · _memory_bytes · _hot_bytes · _decoded_entries · _meta_{raw,compressed}_bytes · _tag_{keys,memberships,index_bytes} · _tag_purge_{tombstones,floor_milliseconds} | gauge | Exact RAM accounting of the index and tag indexes. | | httpjet_pagecache_disk_{bytes,max_bytes} · _disk_evictions_total · _disk_evict_age_{lt_1h,1h_6h,6h_24h,ge_24h}_total · _swept_expired_total | gauge / counter | File-tier footprint vs budget, LRU evictions by entry age, proactive expiry sweeps. | | httpjet_pagecache_disk_{read,write,full}_errors_total · _missing_file_invalidations_total · _orphan_{reclaimed,deferred}_total · _meta_decode_errors_total · _key_id_collisions_total · _poisoned_locks_total | counter | Health: every one of these should be flat. Poisoned locks nonzero = indexes may be inconsistent. `httpjet_pagecache_integrity_active` = 1 once the file-tier integrity key is installed. | | httpjet_pagecache_dict_recompress_total · _dict_recompress_bytes_total · httpjet_pagecache_refresh_inflight | counter / gauge | Store-time dictionary recompressions (bounded background pool); stale-while-revalidate refreshes in flight. | | httpjet_pagecache_shared_path_{public_routes,canary_skipped}_total · _shared_paths_{matchers,canary_percent} | counter / gauge | Shared-paths routing. | | httpjet_xf_capsule_{hits,misses,stale_hits,dedicated_stores,canary_filtered}_total · _hits_by_class_total · _shell_age_secs | counter / histogram | XenForo capsule tier. | | httpjet_process_rss_bytes · httpjet_process_rss_anon_bytes · httpjet_pagecache_unaccounted_rss_anon_bytes | gauge | Process RSS and the part the cache accounting cannot explain. | | LSAPI / PHP | | httpjet_lsapi_{acquire,dispatch,ttfb}_seconds | histogram | Pool contention · backend call · request-committed-to-first-byte. | | httpjet_lsapi_generation_advances_total · _stale_{idle,checked_out}_drops_total · _stale_worker_retire_{signals,failures}_total · _worker_attribution_failures_total · _eagain_{retries,terminal_exhaustions}_total | counter | External-pool generation lifecycle and backlog pressure. | | Proxy | | httpjet_proxy_peer_{healthy,probes_total,health_transitions_total,active_requests,selections_total,prehead_failures_total} · httpjet_proxy_failover_total | labeled | Per-peer (scope, group, ordinal) health and selection; failovers served by a non-primary peer. | | Logging, OTel, debug | | httpjet_log_{access,vhost}_dropped_lines_total | counter | Unified and per-vhost log lines shed by the bounded writer queues (65,536 lines / 64 MiB per logger). | | httpjet_otel_spans_{queue,dropped_total,exported_total,export_failed_total} · httpjet_otel_http_{exports,export_failures}_total | gauge / counter | Span queue depth and loss; collector HTTP outcomes (observable even when the collector is down). | | httpjet_debug_cache_entries_{renders,throttled}_total | counter | Renders of the `/cache-entries` report and locally throttled requests for it. | ## The cache-entries report `GET /cache-entries` on the metrics listener (not the main port — there it hits the vhost and 301s) renders a throttled text report: total entries and bytes, per-URL-class counts, top-N by size, a `ram: entries=… disk: bytes=…/max evictions=… swept_expired=…` line (live index count vs charged tmpfs footprint), and a health line (hits/misses/stores/purges/disk_{read,write}_err/poisoned_locks). `/__mimalloc-stats` beside it prints allocator statistics; `/debug/pprof/profile` (build feature `profiling`) returns a CPU profile — the release binary keeps its symbol table and frame pointers for exactly this. Whereas `/metrics` is open on loopback, `/cache-entries`, `/__mimalloc-stats`, `/__alloc-count` and `/debug/pprof/profile` all require `Authorization: Bearer ` matching `--profile-token-file` (returning 403 when unset or mismatched). ## Logs Access log Error log Syslog tap Per-vhost logs The unified access log (path from the XML; combined format with a trailing `reqid=` token) writes from a dedicated non-blocking task with a bounded queue (65,536 lines and 64 MiB per logger; `httpjet_log_access_dropped_lines_total` counts sheds), opens and rotates without following symlinks (`O_NOFOLLOW`), rolls asynchronously at 50 MB and prunes after 30 days (`HTTPJET_ACCESS_ROLLING_BYTES`, `HTTPJET_ACCESS_KEEP_DAYS`). `HTTPJET_ACCESS_LOG_FORMAT=json` emits one object per line with a fixed field order: access · json ``` {"ts":"17/Sep/2026:11:02:09 +0000","client_ip":"203.0.113.9","remote_user":null,"method":"GET","uri":"/threads/hello.123/","protocol":"HTTP/2.0","status":200,"bytes":41211,"referer":null,"user_agent":"Mozilla/5.0 …","reqid":"3f9c2a7b81d04e55"} ``` Unix-socket peers render the literal host `unix:`. Cache disposition is in the response header (`x-litespeed-cache`), not the log line; join on `reqid`. The rolling `httpjet_error.log` (and captured backend stderr) is text by default. `HTTPJET_ERROR_LOG_FORMAT=json` switches to JSON events with `schema_version 1`: `timestamp_unix_ms`, `level`, nullable `target`, `message`, and a `fields` object that keeps tracing WARN/ERROR fields named rather than flattened. JSON-only redaction replaces values of authorization/cookie/password/token/secret/key/credential-named fields and suppresses whole values that look like credential syntax or URL userinfo — a conservative heuristic, not a guarantee. Bounds: 32 fields, 1024-char values, 8192-char messages; inputs over 32 KiB become `[OVERSIZED]`. The stdout/journald tracing formatter (`RUST_LOG`) is unaffected. `HTTPJET_SYSLOG_TARGET` mirrors every rendered access line as an RFC 5424 (or 3164) datagram from the writer task — request path untouched, file logging unaffected, send failures counted. The MSG part is the rendered line itself, so existing parsers keep working. Targets: `udp://host:514`, bare `host:port`, or a unix-dgram path like `/run/systemd/journal/syslog`. Facility/severity/hostname/app-name knobs are on the CLI page . A vhost's ` ` block with `useServer=0` and a `fileName` gives it its own access and error files (rolling size and keep-days per file; `logHeaders` 7 = combined format with request headers). Log paths are boot-owned: they cannot be changed through the write API. ## Request ids Every request gets a 64-bit correlation id (a per-process seed mixed into a lock-free counter — no syscall, no RNG on the hot path; a background stale-while-revalidate refresh inherits its parent's id) that appears in the access, error and PHP-slow logs. `--request-id-header` also echoes it as `X-Request-Id` so a client or CDN can join a response to the server-side records. Off by default. ## Telemetry TSV and the PHP slow log - **`--telemetry-file logs/telemetry.tsv`**: Appends a cumulative snapshot row of the in-RAM aggregates every `--telemetry-flush-secs` (30): durability across restarts and a self-contained time series (~1 MB/day). The `:9090` endpoint works without it. - **`--php-slow-log logs/php-slow.tsv`**: A TSV line for every LSAPI request whose render TTFB exceeds `--php-slow-threshold-ms` (50), plus a 1-in-128 sample of *all* PHP requests as the population baseline — the per-URL / user-class attribution behind `httpjet_lsapi_ttfb_seconds`. Rolls at 64 MiB, gzip archives, 30-day prune. These two files are what produced the production attribution on the Performance page: mean request 9.99 ms, of which 8.96 ms is inside PHP. ## OpenTelemetry --features otel Build with `--features otel`, then set `HTTPJET_OTEL=1`; without it no provider or exporter is created, and requesting it in a binary without the feature fails startup. The official Rust SDK exports OTLP HTTP/protobuf to `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` (complete URL). `HTTPJET_OTEL_METRICS=1` adds a 30-second periodic metrics reader to `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`. > [Diagram: otel-spans] What gets exported. One root span per request at the dispatch boundary, backend client spans, internal stage spans on sampled requests, and a loss-accounted batch exporter. Spans carry status and bounded internal attributes only — no headers, bodies, URLs or error text. - **Sampling and trust.** `HTTPJET_OTEL_SAMPLE_RATIO` (default 0.01) ratio-samples roots; children follow. Inbound `traceparent`/`tracestate`/`baggage` are discarded by default — clients cannot dictate parentage or force sampling. `HTTPJET_OTEL_TRUSTED_PARENTS` lists exact gateway IPs (direct peers on a known non-PROXY listener) allowed to supply a single version-00 traceparent; a listed gateway controls parentage *and sampling*, so it must strip public trace headers first. Do not list a CDN merely because it forwards client IPs. - **Propagation.** Named configured proxy targets and WebSocket maps receive the generated traceparent; LSAPI receives it as `HTTP_TRACEPARENT`; ad-hoc targets get nothing. - **Completion.** H1 spans end when the transport's write returns (including sendfile bodies); H2 spans ride the `ResponseCompletion` extension through flow control and the final flush; H3 spans complete when QUIC accepts the FIN. `httpjet.body.outcome` is `complete`/`error`/`cancelled` (`upgraded` for WebSocket). This is local write completion, not peer receipt. - **Bounds.** Dedicated export thread, 2048 queued spans + one 256-span batch, 1 s interval, 2 s request timeout, 64 KiB response cap, no redirects or environment proxies, 3 s provider shutdown. Saturation drops telemetry rather than blocking serving; drops are counted on both the OTLP and Prometheus sides. - **Metrics instruments.** `httpjet.pipeline.requests.completed` and `httpjet.pipeline.response_head.duration` (seconds, bounded status labels), plus `httpjet.otel.http.exports`. enable ``` HTTPJET_OTEL=1 HTTPJET_OTEL_METRICS=1 HTTPJET_OTEL_SAMPLE_RATIO=0.05 \ OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:4318/v1/traces \ OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:4318/v1/metrics \ httpjet --root /abs/root serve … ``` ## The read-only status API `--admin-addr 127.0.0.1:9091` exposes `GET /v1/status`: generation, configuration fingerprint, listener names, vhost names, upstream peer health/active/selection counters and page-cache aggregates — never paths, credentials or cached URLs. Full contract on Admin API & reload . ## Debugging a request tracing filters ``` RUST_LOG=debug,hj_rewrite=trace httpjet --root /abs/root serve --http-addr 127.0.0.1:8080 … # where to look: vhost resolution (hj-core router) → rewrite outcome (hj-rewrite eval) → # cache lookup (httpjet lscache) → terminal (pipeline dispatch) → response (handler) ``` Production units run `RUST_LOG=httpjet=info`; anything you must see in the journal should log at WARN or above. ============================================================================== PAGE: Admin API & reload (https://httpjet.net/docs/admin-api) SUMMARY: SIGHUP reload semantics, the read-only status API, the transactional configuration API, revisions, and the restart-required boundary. ============================================================================== # Admin API & reload Three ways to change or inspect a running httpjet: **SIGHUP**, the opt-in **read-only status API**, and the separately authenticated **transactional configuration API**. All writers — including SIGHUP — go through one publication coordinator, so a change is either fully published as a new generation or not applied at all, and anything the process cannot safely replace at runtime is reported as `restart_required` rather than half-done. ## SIGHUP `systemctl reload httpjet` (or `kill -HUP`) re-parses the tree, builds application state *before* touching certificate resolvers, and publishes through `config_transaction::Coordinator`. Preserved from the command line: `--no-mtls` and `--per-ip-rate` overrides. Outcomes, in the journal: - **published**: New generation serving; in-flight requests finish on the old one; certificate handles reloaded for the unchanged listener topology. - **requires a restart — keeping current config**: The diff touched a boot-owned resource (hot reload covers vhost/rewrite/access/expires/tuning, cache policy gates and TLS material). - **reload would 404 mapped vhost(s)**: A per-vhost file failed to load; the old generation is kept rather than serving 404s. - **reload rejected**: Parse or build error — nothing changes. - **certificate reload failed**: Application state published; certificates whose reload failed are retained. ## Read-only status API Opt in with `serve --admin-addr 127.0.0.1:9091`. Non-loopback addresses are rejected at argument parsing; a bind failure aborts startup. It is a separate listener from metrics, changes nothing, and exposes no profiling, allocation or purge operations. **GET** /v1/status · schema_version 1 ``` $ curl -s http://127.0.0.1:9091/v1/status | jq . { "schema_version": 1, "generation": 3, # process-local, starts at 1; rejected candidates stay invisible "config_fingerprint": "sha256:…", # of the normalized model's Debug form — comparable within one binary version only "listeners": [{"name":"https","tls":true,"unix":false}], # configured names + flags, NOT bound addresses "vhosts": ["example.test"], "upstreams": [{"scope":"server","group":"application","peers":[{"ordinal":0,"healthy":true,"active":2,"selections":1290}]}], "page_cache": {"entries": 41210, "memory_bytes": 60817408, "hits": 1904411, "misses": 88123}, # or null when off "truncated": false } ``` No configuration values, paths, credentials, request headers, session keys or cached URLs are returned. Names are limited to 256 characters and each collection to 256 rows; the response is capped at 1 MiB (503 beyond). Only GET at the exact path (405/404 otherwise); Host must be a loopback socket authority or `localhost`; Origin headers and request bodies are rejected; `Cache-Control: no-store`; every connection closes; 4 KiB/32 header fields; four concurrent tasks with five-second deadlines. Local users are trusted — this is not an authentication boundary between users on the same host, and it must never be reverse-proxied. ## Transactional configuration API Writable access is a separate, explicit opt-in that never becomes available merely because the read-only listener is on. Three options are required together: **serve** · enabling writes ``` httpjet --root /abs/root serve … \ --admin-write-addr 127.0.0.1:9092 \ --admin-token-file /operator/private/token \ # 64 lowercase hex chars (32 random bytes), mode 0400/0600, one hard link, root or service uid --admin-resource-root /operator/content \ # repeatable; up to 32 existing directories; "/" is refused --admin-resource-root /etc/letsencrypt/live ``` > [Diagram: reload-transaction] Validate, then publish with an exact precondition. Every write carries If-Match with the revision it read; a stale revision or an out-of-band publication (SIGHUP included) loses with a conflict rather than silently rebuilding against newer state. Publication is volatile: the on-disk XML is never rewritten and a restart returns to it. ### Endpoints and contract | Request | Behavior | | `GET /v1/revision` | The current revision: a random 128-bit process incarnation plus a monotonic generation, so a restart can never revive an earlier process's revision. | | `POST /v1/config/validate` | Builds the candidate, returns a redacted verdict, publishes and persists nothing. | | `PUT /v1/config` | Builds, rechecks the exact revision under the publication boundary, publishes, transfers health-check ownership, notifies bridge admission. Returns the next revision and `persistence: volatile`. An interrupted response after publication does not roll back — re-read the revision to reconcile. | Framing: `Authorization: Bearer ` (missing/duplicate/malformed → 401 without reading the body; constant-time digest compare; nothing about the token in Debug, logs or responses), exact quoted `If-Match` (missing → 428; stale → conflict), `Content-Type: application/json`, one positive `Content-Length` ≤ 1 MiB, headers ≤ 8 KiB/32 fields, five-second combined deadline, Host equal to the bound loopback address or `localhost: `; transfer/content encodings, Origin, Expect, Upgrade and Trailer are rejected; one request per connection, never pipelined. At most four connections and one candidate build; writer-lock acquisition is bounded at 5 s and candidate waiting at 10 s — a timed-out build keeps its admission permit until it exits and can never publish after its caller stopped waiting. **body** · the in-memory bundle (exactly three fields) ``` { "server_xml": " … ", "vhosts": [ {"name": "example.test", "xml": " … "} ], "mime": "text/html = html, htm\n" } ``` `hj-config::parse_bundle` normalizes the bundle without ever reading the submitted `configFile` paths: 1 MiB aggregate, ≤128 vhosts, one root element per document, no DTD, ≤64 nesting levels, missing/extra/duplicate declarations fatal, diagnostics suppressed, errors carry no submitted data. The server root always comes from startup. ### What a transaction may change #### Reloadable (application) - Listener→vhost mappings on the unchanged listener set - Rewrites, document roots, contexts, static locations - Reloadable request settings, cache policy gates, expires - Proxy processors and groups (health schedulers restart cleanly) - GeoIP database and upstream client-cert files — inside a resource root #### Constrained trust (TCP transaction) - Listener identity/routing, PROXY-protocol policy, TLS certificate/key/CA/CRL material — through prepared same-socket TCP successors - An existing `--http-uds` endpoint is handed off - An already-enabled QUIC endpoint adopts the new TLS/mTLS policy for future handshakes without rebinding UDP - UDS ext-processor addresses authorized against resource roots #### restart_required - Listener addresses, count, order, secure/plain roles; QUIC enablement; UDS topology - PHP topology and pools; non-proxy processors - Cache-store identity and boot-time cache caps/policy - User/group, namespace and isolation settings, per-vhost log resources - `useIpInProxyHeader` and the whole ` ` block **Resource policy.** Every filesystem reference in a candidate (vhost roots, docroots, static contexts, GeoIP file, upstream client certs, TCP certificate/key/CA/CRL paths) must resolve inside one of the operator's `--admin-resource-root` directories: parent traversal, relative paths, missing resources and symlinks resolving outside the roots are rejected at submission time. This is authorization, not a sandbox — roots and their symlinks stay operator-controlled. Failures return bounded categories, never configuration values. **Cache namespaces.** The first successful publication introduces a response-cache epoch; every later generation keys page-cache and memo entries by epoch + generation, so a late render from an old generation cannot overwrite a new entry and a previous document root's memoized response cannot replay. See Page cache → generations . **Shutdown.** Closing the coordinator precedes cancelling serving; a prepared candidate receives `Closed` and cannot reactivate probes; the write listener drops its connection tasks. Blocking candidate work cannot be forcibly stopped by tokio, so use responsive local resource files — the ten-second deadline bounds the caller's wait, not an uninterruptible filesystem call. ## The reload boundary in depth Live state is a *resource generation*: application state plus references to listener generations, each with immutable bind, PROXY-protocol and TLS-verification policy, a resource identity, and separately owned accept/drain control. Connections pin their trust generation at acceptance (`CoreHandler` reads a trust-epoch-aware `ServingView`, pinned again at TCP/UDS accept and per QUIC connection). Compatible application reloads stay visible to existing connections; an incompatible trust change retires the old listener group — H2 gets GOAWAY, H1 closes after its current response, H3 stops admitting streams while preserving connection IDs through a bounded drain — rather than letting an old unauthenticated connection reach newly protected routes. | Resource | Owner | Replacement rule | | TCP addresses, worker count, inherited fds | listener plan + `uring` worker groups | Prepared successors on the same sockets behind an activation barrier; kernel socket identities verified; raced accepts transferred to the successor. | | UDS path and permissions | listener adoption | Stage a distinct owned path; never unlink an inherited or unrelated socket; retain the old inode until retirement. | | TLS acceptor, mTLS requirement, PROXY binding | per-core TLS bridge | Pinned to one listener generation per connection. | | Certificates / SNI map | `CertReloadHandle` | Validated snapshots published as one trust-generation handle; multi-resource swaps are not advertised as atomic. | | QUIC endpoint / UDP fds | process-persistent `QuicResources` | UDP ownership stable; a shared policy handle replaces quinn's server config for future handshakes; a stale policy rejects new handshakes once its epoch is no longer current. | | kTLS template | TLS worker | New templates apply to new handshakes; existing key schedules are connection-owned. | | ACME overlays, OCSP staple slots | feature runtimes | Certificate targets owned by resource generation; refresh leases retained through retirement. | | PHP supervisor / external worker registry | startup | Restart-required. | | Log writers, cache budgets, body budget, metrics, bridge admission | process-owned Arcs | Retained across reloads; outstanding reservations preserved. | | User/group, privileges, namespaces, systemd topology | startup | Never acquired at runtime; rejected before any resource acquisition. | Acquisition order for a TCP trust transaction: authenticate and check the revision → derive the resource diff with unchanged resources retained by identity → acquire candidate-only resources with every descriptor, thread and pending side effect owned on creation → start candidate workers behind an activation barrier (readiness proves adoption, not permission to serve) → validate the complete bundle (failure drops and joins everything) → recheck the revision under the coordinator's publication/close boundary and publish atomically — no fallible resource creation after that point → activate and stop old acceptors → retire old generations outside the publication lock, bounded. ## Reproducible checks **tests** · isolated loopback listeners and temporary trees ``` cargo test --locked -p httpjet admin python3 -B scripts/admin-runtime-test.py --binary target/debug/httpjet # served-file replacement, validate-only non-mutation, stale writes, SIGHUP restoration, restart incarnation python3 -B scripts/admin-runtime-test.py --binary target/debug/httpjet --memo-parity # 48-cell memo parity across a publication python3 -B scripts/admin-runtime-test.py --binary target/debug/httpjet --persistent-cache # authenticated on-disk containers, hits across publication, boot-scan restoration ``` ============================================================================== PAGE: Extending httpjet (https://httpjet.net/docs/extensions) SUMMARY: Compile-time extension hooks, the bounded WAF sidecar contract, and why there is no dynamic plugin ABI. ============================================================================== # Extending httpjet Two extension points, each with a deliberately narrow contract: **compile-time extensions** — trusted Rust crates linked into a custom build that register read-only pre-handlers and response transforms — and the **WAF sidecar**, a bounded loopback HTTP inspection call for an independently deployed engine. There is no dynamic plugin ABI and no in-process sandbox, on purpose. ## Compile-time extensions `hj-extension::ExtensionRegistry` is assembled at the binary's registration point, `crates/httpjet/src/extensions.rs`. The distributed binary registers nothing, so the facility is inert unless an operator maintains a custom build. It is a *source-level* compatibility contract over the frozen `hj-core` types, not a stable native ABI. **extensions.rs** · registration ``` pub(crate) fn compiled_registry() -> hj_extension::ExtensionRegistry { let mut registry = hj_extension::ExtensionRegistry::new(); registry.register_pre_handler("company.request-policy", company_policy::hook()).expect("unique"); registry.register_response_transform("company.headers", company_headers::transform()).expect("unique"); registry } ``` Registration order is execution order. Names are globally unique, 1–64 bytes of lowercase ASCII letters, digits, `.`, `-` and `_`. Registered extensions are logged once at boot; changing them means rebuilding and restarting, and SIGHUP carries the same linked registry into the next generation. Extension settings belong to the extension crate — the registry does not reinterpret LiteSpeed XML and exposes no runtime loader. > [Diagram: extension-hooks] Where hooks run. Pre-handlers run after trust and access resolution but before everything that could serve a byte; response transforms run after dispatch (and after page-cache storage) but before httpjet's own transforms, so an extension changes egress, not the canonical cached object, and is re-applied on hits. ### Pre-handler contract A `PreHandler` receives an immutable `ReqCtx` and a `RequestView` exposing only method, URI and headers — no body, no mutable parts. It returns `Continue` (next hook, then built-in dispatch), `Respond(response)` (skip remaining hooks and dispatch; normal transforms, metrics, request-id handling and access logging still run), or `Err(HandlerError)` (fail closed to that status and log the named failure). Pre-handlers never see reserved operational routes intercepted before vhost dispatch (ACME challenges, loopback cache endpoints) or requests rejected before the hook boundary. **Any registered pre-handler disables the on-core memo/cache/static fast path**, because that synchronous path cannot run async hooks; response-only extensions keep it. ### Response-transform contract Registered `ResponseTransform`s run in order on every full-pipeline response and every on-core cache/static response, before the host's static promotion, expires, sub_filter, compression, redirect-cache guard and Alt-Svc transforms — so an extension sees the origin representation before compression and cannot bypass the final transport-header policy by ordering. There is no error return: keep work bounded and express fallible initialization at registration. ! In-process means full authority Extension code runs with httpjet's memory and filesystem authority. A panic, deadlock, unbounded allocation or blocking call affects the server. Only trusted, reviewed crates belong in the registry. ## WAF sidecar httpjet embeds no rules engine and the ignored LiteSpeed `lsrecaptcha`/`censorshipControl` elements stay inactive. Instead, `--waf-sidecar-addr` turns on a bounded inspection call to a local HTTP service over a Unix socket (`unix:/path`, peer UID verified via `SO_PEERCRED` against `--waf-sidecar-uid`) or loopback TCP (`ip:port`) — the integration point for an existing maintained engine such as a Coraza/OWASP CRS adapter. httpjet does not claim CRS compatibility; the adapter owns the translation from the stable inspection document to its engine's model. **serve** · sidecar flags ``` --waf-sidecar-addr unix:/run/waf/inspect.sock # or loopback 127.0.0.1:19090; absent = seam inert --waf-sidecar-uid 0 # uid a unix: sidecar must run as (default: root or serving uid) --waf-sidecar-path /inspect # absolute, no control characters --waf-timeout-ms 100 # total deadline: admission + allowance + connect + write + parse (≤ 30 s) --waf-body-max 65536 # bodies at or below this are sent as body_hex (≤ 1 MiB) --waf-concurrency 128 # ≤ 4096 --waf-fail-open # weaker policy: continue on sidecar failure (every bypass is logged; 403 still blocks) ``` > [Diagram: waf-sidecar] One inspection per request. Inspection runs after URI decoding/normalization and body-limit enforcement but before rewrite, cache lookup and any backend — background cache refreshes go through the same dispatch and are inspected too. Only two replies are valid; everything else is a sidecar failure. - **Document.** `version: 1`, method, normalized path, query, resolved vhost, resolved client IP, protocol, effective TLS state, all request headers. Header values and the body are lowercase hexadecimal so raw bytes are lossless and cannot inject log or protocol controls; adapters must reject unsupported versions. - **Bodies.** ≤ `--waf-body-max` → `body_hex`, restored byte-identically for downstream processing; larger → `body_omitted: true` and the sidecar must decide that shape explicitly. The global request limit stays authoritative. - **Memory.** Beyond the concurrency cap, all generations share a 64 MiB serialization budget; each inspection reserves a byte-weighted allowance before creating hex/JSON copies, so stalled inspections cannot multiply temporary heap up to the concurrency limit. Waiting for the allowance is inside the deadline and follows the fail policy. - **Transport.** One close-delimited HTTP/1.1 POST per inspection, never reused (no response-boundary ambiguity); response head ≤ 4 KiB; duplicate/nonzero Content-Length, Transfer-Encoding, malformed status, EOF and any status other than 204/403 are failures. **tests** ``` cargo test -p httpjet --bin httpjet pipeline::e2e::waf_runs_before_static_cache_and_can_block_a_previously_allowed_path cargo test -p httpjet --bin httpjet pipeline::e2e::waf_sidecar_failure_policy_is_closed_unless_explicitly_opened cargo test -p hj-extension ``` ## Why there is no dynamic or sandboxed plugin ABI httpjet does not `dlopen` third-party trait objects: Rust promises no stable trait-object ABI across independently built shared libraries, and a native module would share allocator, signal, thread and process authority with the server. There is no current requirement to execute untrusted third-party code, so a WASM runtime and its dependency and attack surface are not included. Untrusted or independently deployed inspection belongs behind the bounded sidecar contract above. If an untrusted-plugin workload appears, it must first define capabilities, memory and fuel limits, deadline and cancellation behavior, body access, state and network permissions, version negotiation, and fail-open/closed semantics — only then would an isolation mechanism be selected. ============================================================================== PAGE: Performance (https://httpjet.net/docs/performance) SUMMARY: Measured throughput by regime: loopback three-way benchmarks, the origin cache result, production attribution, sendfile and kTLS receipts, and PGO discipline. ============================================================================== # Performance httpjet is fast — and these docs are strict about *where* the time goes. Every figure names its regime, because a **loopback benchmark** (synthetic, on-box, PGO build) is not the same thing as **production telemetry** (real CDN-fronted traffic). The short version: on a Cloudflare-fronted PHP site, almost all wall time is PHP, and httpjet's own CPU is a rounding error. ~908 k rps h2 static, c=50, dedicated-host loopback 44.8 k rps PHP/LSAPI over HTTP/1.1, PGO loopback 101 × cache hit vs the same page through PHP 0.2 % httpjet's share of production request latency 1.55 IPC h2 hit path at saturation (hardware counters) 0.0002 flt v0.1.5 h2 static minor page faults/req ## Read the regime, not just the number loopback benchmark · synthetic, on-box, PGO production telemetry · real CF-fronted traffic hardware counters · perf at saturation A loopback benchmark removes the network, real TLS handshakes and the CDN — it isolates the server's code path, which is exactly what you want to compare two servers and exactly what you must not generalize to end-user latency. Production telemetry measures what users feel, dominated by the backend and the edge. The same server shows ~908k req/s on loopback and ~0.2% of user-visible latency in production; those answer different questions. ## Performance across builds — interactive presentation Every benchmark campaign from the initial Phase 1–7 baseline (`0.0.1`, June 2026) through the October 2026 zero-fault pooling release (`v0.1.5`) is recorded below with its exact regime, commit, and CSV receipt in github.com/faratech/httpjet . Use the **All-Builds Curve vs LiteSpeed & nginx** below to see the full trajectory on one canvas, or step through the **13 build slides**, **Workload Trajectories**, and **Build-vs-Build Diff**. ## Loopback throughput The primary dataset is the August 2026 **dedicated-host three-way** (httpjet vs OpenLiteSpeed vs nginx on a quiet box, all release/PGO-equivalent builds). The June PGO campaign against a commercial-LiteSpeed baseline remains the PHP-path reference. > [Benchmark Chart: h2-static — HTTP/2 static throughput by concurrency (dedicated-host loopback · 2026-08-27 · empty bodies · req/s)] One h2 connection is not a ceiling (~58k req/s at c=1), and throughput keeps rising with concurrency instead of collapsing; the earlier "h2 collapses past c=50" shape belonged to the June build and its measurement conditions, not the transport. HTTP/1.1 empty bodies at c=1000: ~202k vs OpenLiteSpeed's ~183k. > [Benchmark Chart: heavy-rewrite — Heavy .htaccess static (806-line chain, /robots.txt) (PGO three-way · 2026-08-28 · c=100 · req/s)] > [Benchmark Chart: memo-ab — Fast-path memo, same binary A/B (forum chain · /robots.txt · c=100 · req/s)] > [Benchmark Chart: php-h1 — PHP over LSAPI, HTTP/1.1 (loopback · PGO build · 2026-06 · req/s)] ## The origin cache result The single largest performance fact about the real workload is not a micro-optimization — it is the origin page cache . For the cacheable fraction of traffic, a hit replaces an entire PHP render with a memory (or tmpfs) copy. > [Benchmark Chart: cache-vs-php — One forum page: cache hit vs full PHP render (HTTPS loopback · PGO · 152 KB page · req/s)] A 2026-08-30 refresh under shared-box conditions (live production on the same host — directional only) put cache-hit serves at ~5.4k req/s h2 and ~6.7k h1 on real `/help/*` pages. The production hit ratio reads low for a structural reason: ~51% of origin GET-200s are one-shot bot crawls whose repeat interval exceeds any TTL, so no origin tier can absorb them — the lever there is edge TTL. ## Where the time actually goes > [Benchmark Chart: attribution — Where a production request spends its time (live sample · 2026-08-30 · mean request 9.99 ms)] ⎁ Measured Across millions of production requests, the `lsphp` render is **86.5–98.8% of request wall time**. httpjet's own CPU is about **0.2%** — roughly **138 µs per request**, ~38 µs per cache hit, and that includes TLS, HTTP/2, rewrite, cache lookup and access logging. The 2026-08-30 sample: mean request 9.99 ms, 8.96 ms of it LSAPI TTFB, rewrite 5.4 µs, cache lookup 3.2 µs; pool acquire p99.9 production telemetry · --telemetry-file + --php-slow-log If httpjet is 0.2% of latency, halving its CPU shaves ~0.1% off a page — imperceptible. Turning a PHP render into a cache hit removes the other ~99%. That asymmetry is why the project stopped shaving the serving path and invested in the cache, PHP/opcache tuning and raising the cacheable fraction. ## At the instruction floor ⎁ Measured At saturation the h2 cache-hit path shows **IPC 1.55**, a **0.31% branch-miss rate** and **1.6 L1d misses per 100 instructions**: execution is efficient and the bottleneck is instructions retired per request, not locality. Futex traffic (~85% of syscall time) turned out to be idle parking and cross-thread waker churn — the box was ~50% idle at the apparent ceiling — not lock contention. perf stat / strace · off-CPU analysis Two structural moves followed: the global **mimalloc** allocator (fewer arena futexes across the per-core monoio workers plus the tokio side-runtime) and the **on-core fast path** with the finished-response memo, which removes pipeline work rather than speeding it up. The October 2026 campaign (`0.0.9`–`0.1.5`) extended this to the remaining hot paths: keeping static-missing 404s and non-cacheable PHP GETs off the fast-path double lookup (−14.5% CPU on cookied guest pages), memoizing per-file `.htaccess` access rule matches (`access_decision_for_request` 13% → 2.7% of process CPU), extending the certified-peer idle window via `--trusted-proxy-idle-secs 1800` (cutting full TLS handshakes per request by 28% on x86 and 78% on ARM), tuning per-vhost dictionary zstd levels via `--page-cache-dict-level` (cutting ARM dictionary encode from ~50% to 11% of tokio CPU), and pooling transient response/TLS buffers and h2/bridge task slots (`hj_core::recycle::BODY_BUFS`) so sparse size classes do not page-fault under `MIMALLOC_PURGE_DELAY=0`. ## File egress: sendfile and kTLS > [Benchmark Chart: sendfile — File egress receipts (1 MiB bodies, c=50) (loopback · 2026-09-09 · req/s)] Plain-TCP H1 file bodies leave through bounded nonblocking `sendfile(2)`; TLS 1.3 H1 can too under kernel TLS, but only when `--ktls=auto` finds a physical NIC with active `tls-hw-tx-offload`. The production host's virtual NIC reports `off [fixed]`, and forcing software kTLS there measured slightly *worse* — which is exactly why `auto` refuses it. Details: Protocol stacks . ## The codegen tier — PGO + mimalloc Production binaries combine profile-guided optimization, the mimalloc allocator and `-C target-cpu=native`. The checked-in `scripts/build-pgo.sh` is the classic three-step — instrument, drive a representative loopback workload (h2 cache hits dominant, plus h2/h1 static and PHP), merge and rebuild — with frame pointers kept so the pprof endpoint can symbolize the live binary. Its workload and CPU tuning belong to one deployment; build a neutral release artifact separately. ! A non-PGO benchmark is not a benchmark Debug builds are ~5× slower and meaningless. Even a plain release build differs from PGO by enough (~10% from binary layout alone) to fake a regression: build both sides the same way and interleave runs. The bench harness (`scripts/bench-suite.sh`, `scripts/bench-report.py`) records CSVs under `scripts/bench-results/`. ## Knobs that actually move production | Lever | Effect | | Page-cache opt-in coverage and TTLs (app side) | The dominant lever: every hit removes a PHP render. | | `--page-cache-admit-threshold 1` with dictionaries | Cache on first sight when storage is cheap; watch `httpjet_cache_hit_ratio_real`. | | `--cf-send-zstd` | Hands the CDN the cheapest encoding to produce; no browser-visible change. | | Fast-memo eligibility of the docroot chain | An ineligible chain (see `httpjet check`) pays the full pipeline on every static request; fix the directive rather than disabling the memo. | | `--rewrite-ua-classify` | Collapses UA diversity on UA-reading chains onto a handful of outcome-cache entries. | | TLS resumption & certified idle window | Full handshakes dominate behind a CDN that does not re-offer tickets; ECDSA chains, the single AES-128-GCM suite, `--tls-session-tickets 0` and `--trusted-proxy-idle-secs` keep handshake CPU low. | | PHP: opcache, worker count, slow endpoints | Use `--php-slow-log` attribution; the pool is rarely the bottleneck (acquire p99.9 ## All builds & benchmark receipts ledger Filter by build tag, commit, workload, regime, or CSV receipt file across all 13 benchmarked eras: | Build / Commit | Regime | Measured Results & Source Receipt | | `0.0.1` · Phase 1–7 2026-06-01 · `dcf4ca1e` | loopback A/B vs LSWS 6.3.5 | Light `.htaccess` H1: **135,448 RPS** vs LSWS 182,743 (`0.74×`); Heavy 477-line `.htaccess` H1: **95,840 RPS** vs LSWS 160,920 (`0.60×`); H2 1 KB `c=50`: **~128k RPS** vs LSWS 220.5k. Receipt: `RESULTS.md` Phase 7, `litespeed-*-run1.csv` | | `Tier A` · StatCache + DFA 2026-06-04 · `2e1db15` | loopback A/B vs LSWS 6.3.5 | 1s TTL on `.htaccess` `statx`, `FileCache` & `StatCache` (`-f`/`-d`) + `RegexSet` DFA prefilter: heavy `/robots.txt` ratio lifted **`0.46× → 0.63×`** (`0.70×` peak); 66 KB `og-image.png` reached **`0.90–0.99×`** parity. Receipt: `RESULTS.md` Tier A | | `Batches 1–5` · Micro-Opts 2026-06-06 · `2d80a25` | same-session A/B shared host | Lazy header map, `Cow` CGI env keys, literal-prefix rule filter, `zstd > br > gzip` negotiation, OR-condition regex folding (28 evals → 1): PHP H1 `c=40` **31,013 → 31,300 RPS**; heavy H1 **78.6k vs 72.3k** (within noise, isolating codegen/allocator as the true bottleneck). Receipt: `RESULTS.md` Batches 1–5 | | `Batch 6` · Codegen PGO 2026-06-07 · `5091dfc` | isolated PID A/B PGO + mimalloc | `-C target-cpu=native` + global `mimalloc` + 3-step PGO: heavy `.htaccess` H1 **75.4k → 84.3k RPS (`+11.8%`)** matching LSWS (84.3k); light static H1 **88.3k → 93.0k RPS (`+5.2%`)**. Receipt: `RESULTS.md` Batch 6, `scripts/build-pgo.sh` | | `LSCache + H2 Fix` 2026-06-07 · `dc38d640` | loopback PGO HTTPS suite | Single-flight follower Notify fix lifted heavy `c=50` H1 **298 → 72,371 RPS** and H2 **2,939 → 165,939 RPS**. H2 1-conn 64-stream fan-out: **52.5k → 116k RPS (`2.2×`)**. 1s `open_beneath` memo: `openat` **20,000 → 0** (`+12%` static). H2 1 KB `c=50`: **250.8k** vs LSWS 220.5k; PHP H1 `c=40`: **44.8k** vs LSWS 38.5k. Origin page-cache HIT (152 KB): **19,912 RPS (`p50 2ms`)** vs PHP **196 RPS (`p50 93ms`, `101×`)**; tmpfs `jetcache` hit: **18,211 RPS**. Receipt: `httpjet-sweep-fixed.csv`, `httpjet-cache-run1.csv`, `httpjet-cache-jetfile.csv` | | `io_uring` · monoio 2026-06-21 · `30b7fda` | transport A/B uring vs epoll | Unconditional thread-per-core `monoio` `io_uring` transport for H1/H2 TCP+TLS and UDP H3 + vendored `monoio-rustls` vectored `writev`: H1 **`+19%`**, H2 **`+22%`**, H3 **`+14–20%`** over tokio/epoll + quinn. Receipt: `CLAUDE.md` / `RESULTS.md` 2026-06-21 | | `v0.0.1` · 3-Way + Memo 2026-08-28 · `456711f5` | dedicated 3-way + prod telemetry | H2 static empty: `c=50` **907.7k** (HJ) vs 792.8k (OLS); `c=200` **937.0k** vs 687.2k; 1 KB `c=200` **803.9k** vs 790.5k. Fast-memo A/B (`c=100`): H1 **107k → 153k (`+43%`)**, H2 **137k → 248k (`+81%`)**. 806-line `.htaccess` PGO 3-way: H2 **298.0k** (HJ) vs 283.3k (OLS) vs 125.4k (nginx). Prod telemetry: mean **9.99 ms** (8.96 ms PHP = `89.7%`, HJ CPU **138 µs** = `0.2%`, **1.55 IPC**). Receipt: `versus5-dedicated-final-20260827.csv`, `compare-rewrite-memo-vary-pgo.csv` | | `v0.0.2` · sendfile & kTLS 2026-09-23 · `f987d7cd` | loopback egress 1 MiB × 5,000 c=50 | Plaintext H1 `sendfile(2)`: **4,852.5 RPS** (`10.08 ms` mean); TLS 1.3 H1 rustls (`--ktls=off`): **2,999.7 RPS**; forced software kTLS on virtual GVE NIC: **2,944.9 RPS** (validating why `--ktls=auto` requires physical `tls-hw-tx-offload`). `--page-cache-dict-level windowsforum.com=6` halved encode CPU for `+4%` footprint. Receipt: `httpjet-sendfile-sendfile-release.csv`, `httpjet-ktls-sendfile-*-20260909.csv` | | `v0.0.3` · openat2 + H3 PGO 2026-09-27 · `6e36d652` | 6-pair ABBA H2 + H3 PGO | Rooted `openat2(RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS)` file-tier opens + H3 QPACK PGO training: warm H2 dict `/help/smilies/` **`+4.46%`** (combined `+1.45%`), no-dict **`+11.66%` / `+2.81%`**, H3 20k-req ABBA **`+0.64%`** (**28,653 RPS** peak, `p50 0.427 ms`). Receipt: `httpjet-0.0.3-vs-previous-*-20260927.csv` | | `v0.0.9` · CPU Campaign 2026-10-18 · `d4a4e333` | PMU counters x86 + ARM64 | Idle ticker parking: idle CPU **`−54..63%`** (TLS wakes `32/s → 0`). Cookied front-controller early bridge: cookied guest **`−14.5%` CPU** (TLS **`−30%`**), PHP **`−12%`** (TLS **`−44%`**). Per-file `.htaccess` access-rule match memo (`#963`): `access_decision_for_request` **`13.0% → 2.7%`** CPU. ARM64 PMU instructions/req: PHP **`−18%`**, guest **`−11%`**, hit **`−8.5%`**. Live x86 prod CPU/req: **970 µs → 921 µs**. Receipt: `RESULTS.md` 2026-10-18 (`#958`–`#964`) | | `v0.0.10` · TLS Idle + Dict L3 2026-10-18 · `f73fcedf` | live prod metrics x86 + voidrune | `--trusted-proxy-idle-secs 1800` cut full TLS handshakes/req **`0.052 → 0.0365 (−28%)`** on x86 and **`0.77 → 0.169 (−78%)`** on ARM64; `--tls-session-tickets 0` cut handshake CPU **`−5%`**. `--page-cache-dict-level moontimenow.com=3,thisdayinhistory.ai=3` sped up ARM64 dict encode **`5.3×`** (`3.53 → 0.66 ms/page`), dropping dict CPU from **`~50% → 11%`** of tokio CPU. Receipt: `CLAUDE.md` `#486`, `#489`, `#967` | | `v0.1.2–0.1.3` · Task Pools 2026-10-24 · `2f259201` | perf stat PGO PURGE_DELAY=0 | `hj_core::recycle::Recycled ` for H2 stream & bridge futures, LSAPI read/BEGIN pools, and per-thread zstd/gzip scratch: minor page faults/req dropped **`0.80 → 0.0005`** on H2 1 KiB (`−22..32%` CPU), **`1.61 → 0.05`** on heavy `.htaccess` (`−19..25%` CPU), **`1.60 → 0.026`** on 64 KiB, and **`0.41 → 0.001`** on cache hits. `v0.1.3` (`0c10fdc4`) skipped quality-9 brotli on internal refresh under `--cf-send-zstd` (**`−8k` faults & `−10 ms`/page**). Receipt: PR `#1157`, commit `0c10fdc4` | | `v0.1.5` · BODY_BUFS 2026-10-24 · `16ed92f9` | perf stat PGO x86 + ARM64 | `BODY_BUFS` `TieredPool >` frozen via `Bytes::from_owner`, zero-copy LSAPI `PacketReader` slices, `spawn_blocking` variant/dict fills, per-thread `PageDict` `DCtx`, `MAP_POPULATE` tmpfs mmaps, and per-core TLS record size-class retention: streamed 117 KB `/login/` H2 faults/req dropped **`1.80 → 0.106 (−94%)`**, W1 cache hit **`0.0325 → 0.0005`**, W3s1 1 KiB **`0.185 → 0.0002`**, and 500-URL miss→store RSS dropped **`165–195 MB → 125–132 MB`**. Receipt: PR `#1163` (`16ed92f9`) | ============================================================================== PAGE: Conformance & testing (https://httpjet.net/docs/conformance) SUMMARY: h2spec 146/146, RFC coverage, LiteSpeed parity, the test layers, CI interoperability jobs, fuzzing, and the deploy gates. ============================================================================== # Conformance & testing Two axes define httpjet's posture: **LiteSpeed compatibility** on the same configuration tree (LSWS 6.3.5 target — XML model, LSAPI, ETags, LSCache protocol, `.htaccess`), and **standards conformance** on the wire. Where they agree there is one right answer; where LiteSpeed made a deliberate choice a strict reading would call optional, httpjet matches LiteSpeed. Everything is held in place by ~3,000 tests, h2spec, fuzzing, CI interoperability jobs and the production deploy gates. ## Conformance matrix | Area | Standard / reference | Status | | HTTP/1.1 | RFC 9110 · RFC 9111 · RFC 9112 | Strict framing, conditional requests, single + multipart ranges (RFC 9110 §14), `Vary`, 103 Early Hints (RFC 8297); RFC 9112 request-target forms enforced at intake (origin-form; `*` OPTIONS-only; absolute-form reduced with its authority replacing Host), 405 for CONNECT, 400 for authority-form/raw `#`/bare-LF lines, Host required on HTTP/1.1 and validated by the h2 authority rule | | HTTP/2 | RFC 9113 (native `hj-h2`) | **146 / 146** h2spec 2.6.0 over TLS; h2c informative 145/146 (documented dual-protocol exception) | | HPACK | RFC 7541 | Zero-allocation codec; two fuzz targets | | HTTP/3 / QUIC | RFC 9114 over RFC 9000 (quinn-proto), QPACK RFC 9204 | aioquic transfer and OTel trace fixtures in CI; migration disabled | | TLS | rustls + aws-lc-rs · RFC 8446 mandatory suite | SNI mirrors OpenLiteSpeed; mTLS with CRLs; single-use tickets, 0-RTT off | | ACME | RFC 8555 §8.1 / §8.3 (Pebble v2.10.1 fixtures) | HTTP-01 and DNS-01 behind `--features acme` | | OCSP | RFC 6960 with the lightweight-profile conventions (SHA-256 issuer identity, bounded fetch — not full client conformance) | Behind `--features ocsp` | | PROXY protocol | v2 binary + v1 text | Per-listener, fail-closed, pre-TLS/pre-sniff; loopback claims refused | | LSAPI | Native wire codec (vendored php-src `lsapidef.h`) | Parity with `lsphp`; fuzzed response-header parser | | FastCGI | FastCGI 1.0 (65,535-byte records, application record types) | Synthetic responder interoperability | | ETag | LiteSpeed `fileETag=28` | `" - - ;;;"`, byte-for-byte | | `.htaccess` | Apache mod_rewrite / mod_access / mod_auth_basic / mod_headers / mod_expires subset | Directive and flag parity for the common core (including `Expires*`, `Header merge` and APR1/MD5-crypt htpasswd); `Header` boundary characterized by fixtures | | OpenTelemetry | OTLP HTTP/protobuf, W3C traceparent (version 00) | Behind `--features otel`; decoded-protobuf tests | | LiteSpeed parity | Retired differential matrix vs LSWS 6.3.5 | 11 / 11 on the retained behaviors; now exercised continuously by production | ## LiteSpeed parity and the clean-room boundary Differential testing against a live LiteSpeed reached 11/11 on ETag format, `.html`→PHP dispatch, LSAPI encoding and rewrite-flag semantics before the apparatus was retired: httpjet *is* now the same-config drop-in serving the live tree. The authoritative reference for LiteSpeed's exact behavior was the GPL OpenLiteSpeed source, read to extract *behavior* and then reimplemented from scratch — no OpenLiteSpeed code was copied. httpjet itself is licensed **GPL-3.0-only**, with no paid edition or license key, and is independent of and not endorsed by LiteSpeed Technologies. **fileETag=28** · byte-for-byte ``` ETag: "4096-1718800000-1572934;;;" # size-mtime-inode, three trailing semicolons — strict validators compare bytes ``` Suffix routing and the `.html`→PHP decision follow OpenLiteSpeed's order, including the source-disclosure guard: a script whose handler has no pool is a 503 and never falls through to static. ## Test layers | Layer | What it covers | Run | | unit | Pure logic per crate: config parsing, rewrite evaluation, HPACK, ETags, cache keys, ACME snapshots, OCSP profiles, FastCGI records. | `cargo test -p ` | | integration | Crate boundaries: real `.htaccess` files through the engine (`redirect_repro`, `header_compat`), upstream groups on ephemeral ports, page-cache expiry/race tests, full-pipeline e2e and security regressions in the binary, the H1 corpus replay (2,333 seed/truncation/mutation cases). | `cargo test --workspace` | | live (`#[ignore]`) | Spawn `lsphp`, read real certificates, bind ports, QUIC round-trips, Pebble issuance. Run individually with their prerequisites; never as one batch. | `cargo test -- --ignored ` | | scripts | Real-server fixtures on isolated loopback listeners: h2spec, ACME/OCSP/admin runtime tests, the cache gates, memo parity, OCI smoke. | see each page | The last full runs: the whole workspace suite green at the 0.1.5 release (3,019 passed / 0 failed); 2,028 at 0.0.5. A failure is a regression, not "pre-existing." Timing-flaky meta-assertions were deleted in June 2026 so the concurrency tests are deterministic. > [Diagram: verification-pyramid] Four-layer verification stack. Every change passes ~3,019 unit and integration tests, bounded corpus and ASan fuzz targets, protocol interoperability suites (h2spec 146/146, aioquic, Pebble ACME, OTel), and live loopback deploy gates before cutover. ## CI and interoperability jobs - **Ubuntu 24.04 x86_64**: formatting, default and profiling builds, strict example-config validation, dependency-license and DCO checks, workspace tests, H1 corpus replay. - **Ubuntu 22.04 x86_64 and Ubuntu 24.04 ARM64**: compile all features, replay H1, run the real io_uring UDS H1/H2 test — selected compatibility probes on GitHub-hosted runners, not a full kernel matrix. - **Protocol interoperability**: an OTel-enabled binary, h2spec 2.6.0 over TLS against a fresh example configuration (archive SHA-256 pinned, exactly 146/146 required), H1 WebSocket / H2 dispatch tracing, and the aioquic 1.3.0 QUIC test in an isolated venv. - **Dependency advisories**: a daily job fails on any RustSec finding and on a newer compatible transport-critical crate (quinn, rustls, aws-lc-rs, tokio, httparse, io-uring, zstd, brotli, h2, hyper, http, mio, socket2, openssl, bcrypt, libc). - **Nightly/manual fuzz**: all eight parser targets with bounded time, input size, memory and artifact retention. **local reproduction** ``` cargo test --locked -p httpjet --test h1_corpus -- --nocapture cargo build --locked -p httpjet --features otel --bin httpjet python3 -B scripts/ci-h2spec.py --binary target/debug/httpjet --h2spec h2spec cargo test --locked -p httpjet --features otel traced_fast_and_bridged_requests cargo test --locked -p httpjet --features otel traced_quic_roundtrip -- --ignored ``` Hosted execution evidence is tracked separately from local runs; a runner label being available is not proof a job executed. ## Fuzzing Eight `cargo-fuzz` targets share one invariant (never panic, never allocate unboundedly, agree with the reference decoder where one exists): `h1_chunked_decode`, `h1_request_framing`, `hpack_decode`, `hpack_roundtrip`, `lsapi_resp_header`, and — added in the 2026-10-06 review — `qpack_decode`, `proxy_protocol_parse` and `lscache_control`. Short ASan smoke runs of the original five covered 164,558 chunked, 739,373 framing, 452,899 HPACK-decode, 58,082 HPACK-roundtrip and 438,582 LSAPI-header executions without a finding. **bounded fuzz** ``` FUZZ_TOOLCHAIN=nightly FUZZ_SECONDS=5 bash scripts/fuzz-bounded.sh h1_request_framing python3 scripts/fuzz-seeds.py # regenerate the shared seed corpus ``` ## Production gates Beyond the suites, a deploy runs the cache-correctness gates (cross-URL, cross-scheme, cross-host, private tier, persistence), the fast-memo parity gate, the capsule purge gate and the h2spec gate. They are described with their failure classes on Page cache → deploy gates and Operations → deploy . ============================================================================== PAGE: Migrating from LiteSpeed (https://httpjet.net/docs/migration) SUMMARY: A validation-first migration path from LiteSpeed or OpenLiteSpeed: inventory, check --strict, isolated candidate, behavioral proof, caching, cutover and rollback. ============================================================================== # Migrating from LiteSpeed httpjet consumes LiteSpeed-compatible XML and LSAPI, but a migration is a **compatibility project**, not a binary swap that can skip validation. Directive coverage, filesystem identity, listener ownership, PHP supervision, TLS trust and cache behavior must each be proved for the target deployment. This is the path that proves them in order. > [Diagram: migration-path] Eight steps, three gates. Nothing binds the incumbent ports until behavior — not just startup — has been proved on an isolated candidate; caching is introduced only after uncached parity. ## 1 · Inventory before changing traffic - Main and per-vhost XML, included files, variables, listener/vhost maps. - Document roots, symlink policy, `.htaccess` files, error documents, MIME maps, htpasswd files. - TLS identities, chains, client-CA/CRL policy, renewal ownership (certbot? built-in ACME?), SNI maps. - LSAPI processors, PHP environment, child limits, socket owner/mode, application health checks. - Proxy/WebSocket targets, timeout and retry assumptions, trusted-proxy networks. - Cache opt-in and purge headers, public/private variation, stale behavior, every external cache/CDN layer. - Service user/group, writable paths, open-file/memory limits, socket activation, firewall rules, monitoring probes. Keep secrets and host-specific paths out of migration reports and example repositories. ## 2 · Validate a copied configuration **gate 1** ``` target/release/httpjet --root /abs/copied-root check --strict sudo -u target/release/httpjet --root /abs/copied-root check --strict ``` Treat errors as blockers. Review every warning — ignored security elements (`lsrecaptcha`, `censorshipControl`), generic CGI contexts, wildcard maps, TLS over a Unix socket, unsupported auth forms, missing GeoIP data, unloaded vhosts, and settings that require a restart rather than a reload. *Parsed does not mean equivalent.* Compare the resulting listener/vhost map and effective limits with the source server, and run the check as the runtime identity too: readable-by-root is not proof the service user can read certificates or write logs, sockets and cache. ! Things LiteSpeed does that httpjet will not `RequestHeader` and `Header edit` are ignored; traditional DES and `$5$`/`$6$` `crypt()` htpasswd hashes verify false (bcrypt, APR1 `$apr1$`, MD5-crypt `$1$` and `{SHA}` are supported); `Require` forms other than `all granted`/`valid-user`/`user` deny; anti-abuse elements are inert; `cacheStorePath` is unused; `respBuffer` always streams; `instances>1` is a single upstream. Each is on the capability matrix . You can also paste any `.htaccess` file into the interactive `.htaccess` & `fast_memo` readiness inspector to preview directive support and `MemoClass` eligibility in the browser. ## 3 · Build the smallest required binary **build** ``` CARGO_INCREMENTAL=0 cargo build --locked --release -p httpjet ``` Add `acme`, `ocsp`, `otel`, `profiling` or `ktls` only for a reviewed need, and keep every default-off feature and runtime switch off during initial parity testing. A `ktls` build defaults to `--ktls=auto`, which stays on rustls unless a physical NIC advertises active `tls-hw-tx-offload`. ## 4 · Start an isolated candidate **serve** · alternate everything ``` target/release/httpjet --root /abs/copied-root serve \ --http-addr 127.0.0.1:18080 --https-addr 127.0.0.1:18443 --workers 1 \ --php-socket /tmp/httpjet-migration-test.sock --php-children 4 \ --metrics-addr 127.0.0.1:19090 ``` Never bind the incumbent ports for the first run. Use loopback/alternate ports, a separate LSAPI socket (or an explicitly selected external test pool), distinct logs and a separate cache path. If PHP is out of scope use `--no-php` — and do not read that static fallback as PHP parity. Never point two supervisors at one LSAPI socket or process family. ## 5 · Prove behavior, not only startup ### Routing & transport - exact / default / foreign Host routing, HTTP→HTTPS behavior - SNI, chain validation, ALPN, client certificates, CRL rejection - h1 keep-alive and chunking, h2 multiplexing, h3 over UDP, large files, ranges, conditional requests, compression, HEAD ### Application - rewrite/redirect loops, front controllers, PATH_INFO, per-directory access, Basic auth, custom headers, error documents - PHP/FastCGI environment, uploads, streaming, timeouts, worker recycling, source-disclosure failure cases - proxy/WebSocket/SSE, upstream TLS identity, load distribution, all-backends-down, active-health recovery - logs, request ids, metrics, readiness, graceful shutdown, restart, reload Compare status, headers and body bytes with the incumbent where compatibility is required. Normalize inherently variable fields (Date, generated request ids) explicitly — do not discard all headers to make a comparison pass. ## 6 · Introduce caching separately Leave `--page-cache` off until uncached parity is established. Then, with an isolated cache and conservative caps, prove that distinct paths, hosts, schemes, queries, cookies, sessions and encodings cannot replay one another; public responses never store private cookies or authenticated data; private entries are session-bound; purge tags invalidate every tier; stale-if-error and stale-while-revalidate obey their windows; restart scanning, integrity failures, missing files and capacity eviction are fail-safe; and the upstream CDN does not retain redirects or errors longer than intended. The repository's cache gates are the executable form of this list. A cache that looks fast but is keyed wrongly is a data-isolation defect, not a performance win. ## 7 · Stage traffic and retain rollback Package the exact tested binary at a stable path. Prefer a service manager that can hold listener ownership (socket activation), apply non-root permissions, set limits and restart on failure — validate the systemd or OCI example against your environment rather than copying it. Canary through an existing load balancer or CDN, or a dedicated address, before a full listener handoff. Monitor status classes, backend latency and errors, cache dispositions, active connections, memory, file descriptors, PHP/upstream health and h1/h2/h3 traffic independently. Write and *drill* the rollback before cutover: the prior binary, configuration, unit, certificate ownership, application socket and cache/CDN purge procedure. A binary rollback does not remove bad objects already stored in origin or edge caches. ## 8 · Reload only what is reloadable Validate before SIGHUP or an API submission. Application-compatible state publishes while existing requests finish on the old generation; listener/QUIC/Unix-socket topology, PHP topology, cache-store identity, process privileges and other boot-owned resources need a restart. An API `restart_required` result is a safety boundary, not a suggestion to force the transaction. See Admin API & reload . ## Minimum acceptance record - Source commit and build features. - Sanitized configuration fingerprint (`vhost-map-fingerprint`) and validation output. - Service identity, kernel/runtime, resource limits. - Exact test commands, protocol clients, pass/fail counts. - Canary window and the observed error/cache/backend deltas. - The rollback command and a successful rollback drill. Do not claim completion from source inspection, a successful compile or one HTTP 200. The migration is complete when the intended workload, failure modes, operations and rollback have been exercised in the target environment.