Core concepts¶
Narwhal assigns prefill and decode roles across dual-capability engines that serve one model.
A role move changes placement for new requests. Model weights, KV paths, and resident requests stay on their engines.
Concept pages¶
-
Request flow and fleet topology
Engine contract, request execution, and fleet topology.
-
Role control and capacity floors
Role-controller evaluation, role floors, fallback, and degraded capacity.
-
Failure, readmission, and state
Monitoring and readiness, engine failure handling, peer memory release, readmission and drains, serving saturation and retries, and durable control-plane state.
Terms¶
| Term | Meaning | Reference |
|---|---|---|
| Time to first token (TTFT) | Router-measured time from request arrival to prefill completion, targeted by slo.ttft_s. |
Request timing documentation |
| Time per output token (TPOT) | Router-measured average interval between output tokens after prefill, targeted by slo.tpot_s. |
Load definitions documentation |
| Service-level objective (SLO) | The TTFT and TPOT targets in the slo object. |
SLO attainment documentation |
| Dual-capability engine | An engine that runs both prefill and decode. | Adaptive hot-swap documentation |
| Adjacent split | A prefill/decode split one engine move away from the current split. | Role control documentation |
| KV handoff | The backend's key-value (KV) descriptor that a prefill engine returns and a decode engine consumes. | Disaggregated backend execution documentation |
| State handoff | The active router's versioned high-availability (HA) state document at /narwhal/handoff, read by standby routers and --resume. |
State handoff documentation |
| Attestation sidecar | The narwhal-attest process that returns HTTP 503 after the engine process changes. |
narwhal-attest documentation |
| Sample sidecar | The .samples.json file beside the profile store, holding raw observations and process evidence for each fit. |
Retaining profile samples and fits documentation |
| Process generation | The engine identity that profiles and first-token calibration artifacts must match. For a fleet with engine_contract, it is the verified attestation's launch_digest or attestation_digest. For other fleets, it is the engine's version and process start time. |
Validating the engine cost model documentation |
| Whole-wave | Scope of the whole_wave restart policy, covering every configured engine as one wave. |
Restarting an engine wave documentation |
| Lease domain | Shared storage with POSIX flock, coherent reads, and atomic rename, giving both routers of a pair one lease. |
Starting a router pair documentation |
| Deployment set | The Narwhal release, fleet configuration, profile store, deployment evidence, and configured first-token calibration artifact sharing one release identifier. | Keeping one deployment set documentation |
| Engines eligible for placement | Engines left after ejection, drain, and quarantine exclusions, reported as available_instances in /health and used as the live count for role floors. |
Pool and SLO fields documentation |
Related reference material¶
-
Prefill and decode legs, KV handoff, and backend failures.
-
Fleet fields, environment inputs, and their defaults.
-
Profiling, SLO calibration, and target-workload measurement.
-
Completion, inspection, and lifecycle endpoints.
-
Ingress, routers, engines, and software upgrades.
-
Source packages and the areas each one implements.