Skip to content

Core concepts

Narwhal assigns prefill and decode roles across dual-capability engines that serve one model.

A role move changes placement for new requests. Model weights, KV paths, and resident requests stay on their engines.

Concept pages

Terms

Term Meaning Reference
Time to first token (TTFT) Router-measured time from request arrival to prefill completion, targeted by slo.ttft_s. Request timing documentation
Time per output token (TPOT) Router-measured average interval between output tokens after prefill, targeted by slo.tpot_s. Load definitions documentation
Service-level objective (SLO) The TTFT and TPOT targets in the slo object. SLO attainment documentation
Dual-capability engine An engine that runs both prefill and decode. Adaptive hot-swap documentation
Adjacent split A prefill/decode split one engine move away from the current split. Role control documentation
KV handoff The backend's key-value (KV) descriptor that a prefill engine returns and a decode engine consumes. Disaggregated backend execution documentation
State handoff The active router's versioned high-availability (HA) state document at /narwhal/handoff, read by standby routers and --resume. State handoff documentation
Attestation sidecar The narwhal-attest process that returns HTTP 503 after the engine process changes. narwhal-attest documentation
Sample sidecar The .samples.json file beside the profile store, holding raw observations and process evidence for each fit. Retaining profile samples and fits documentation
Process generation The engine identity that profiles and first-token calibration artifacts must match. For a fleet with engine_contract, it is the verified attestation's launch_digest or attestation_digest. For other fleets, it is the engine's version and process start time. Validating the engine cost model documentation
Whole-wave Scope of the whole_wave restart policy, covering every configured engine as one wave. Restarting an engine wave documentation
Lease domain Shared storage with POSIX flock, coherent reads, and atomic rename, giving both routers of a pair one lease. Starting a router pair documentation
Deployment set The Narwhal release, fleet configuration, profile store, deployment evidence, and configured first-token calibration artifact sharing one release identifier. Keeping one deployment set documentation
Engines eligible for placement Engines left after ejection, drain, and quarantine exclusions, reported as available_instances in /health and used as the live count for role floors. Pool and SLO fields documentation