Production boundary and router pair¶
Production boundary¶
A production deployment divides the work between Narwhal and the site's components:
| Component | Responsibility |
|---|---|
| Site automation | Provisions GPU hosts and engine processes |
| Public ingress | TLS, client authentication, rate limits, model routing, streaming proxy settings |
| Load balancer | Polls /ready and routes to the router that returns HTTP 200 |
| Narwhal | Admission, queueing, prefill and decode placement on the running fleet, retry, health ejection, role control, drain, readmission |
| Engine supervisor | Engine and attestation sidecar starts and stops, resource limits, restart policy, log retention |
| Shared storage | One lease domain shared by both router hosts |
| Monitoring | Metric scraping, journal retention, and paging per site policy |
Keeping one deployment set¶
Install on both router hosts, with the same release identifier:
- the Narwhal release
- the fleet configuration
- the profile store
- the first-token calibration artifact, when
engine.first_token_calibration_pathis set, readable from each router's working directory - the deployment evidence set
When preflight's calibration gate rejects the first-token calibration artifact:
- Recalibrate.
- Distribute the new artifact and fleet configuration.
- Restart the routers.
Inspect state handoff contracts before a router change:
When the handoff contract versions match on both routers, follow the rolling upgrade. When they differ, follow Upgrading across a handoff-version change.
Configuring the client path¶
Public ingress carries the completion routes clients use. The private network carries /narwhal/*, /metrics, /health, /ready, the engine APIs, and the attestation endpoints.
Ingress must:
- remove client-supplied internal credentials and request IDs
- insert trusted replacements
- authenticate clients
- apply identity policy and rate limits
- route each model to its router pair
- forward streaming chunks as they arrive
- enforce connect and idle timeouts set from the service budget
Each engine leg receives:
- a request ID per attempt and phase
- the engine credential identified by
engine.engine_api_key_env
Configure the load balancer from the shipped HAProxy configuration.
Starting a router pair¶
Run the final preflight against the deployment set.
The shared lease path must support POSIX flock, coherent reads, and atomic rename. The host clock offset must stay below --lease-safety-margin. --lease-ttl must exceed the sum of --lease-renew-interval and --lease-safety-margin.
Start the first router, with its private listen address as --host:
narwhal-serve \
--fleet config/fleet.production.json \
--host <private-listen-address> --port 8000 \
--router-id router-a \
--lease-path /shared/narwhal/router.lease
Start the standby on the second host:
narwhal-serve \
--fleet config/fleet.production.json \
--host <private-listen-address> --port 8000 \
--router-id router-b \
--lease-path /shared/narwhal/router.lease \
--standby-of http://router-a:8000
Admission sequence:
- Wait for the active router's
/readyto return HTTP 200. - Send the deployment workload through the intended ingress path.
- Open client admission.
Lease behavior¶
A router that holds a valid lease and admits traffic returns HTTP 200 from /ready. During a network partition, the active lease holder fences itself before its local lease deadline, and the standby claims the lease at expiry. When shared storage becomes unavailable, both routers withdraw readiness.
A standby or fenced router retains its saved primary state handoff when it shuts down. The active lease holder persists its latest counters before releasing control at shutdown.