Request flow and fleet topology¶
Engine contract¶
Every engine in a fleet meets the same contract:
- Speaks the configured inference-engine dialect.
- Produces and consumes compatible key-value (KV) cache.
- Sends KV to every peer eligible to receive it.
- Has measured prefill and decode performance profiles.
- Passes preflight validation before taking traffic.
- Passes readmission checks after a hold, drain, failure, or maintenance event.
The fleet's engine_contract lists the compatibility fields that every engine must match.
KV transfer for vLLM engines¶
A vLLM engine with the effective kv_both role transfers KV across the configured ring or mesh once attestation capture has read its inputs from the live process and narwhal-check has validated the attested process. The attestation sidecar binds the process to its image, NIXL connector and runtime features.
KV transfer for SGLang engines¶
An SGLang engine transfers KV through the Mooncake connector, as its runtime fields set. It starts as prefill and changes roles at runtime. Both legs of a request carry the producer's bootstrap host and port from its attestation, and the decode leg waits at most 30 seconds for its first token.
Request execution¶
A completion request passes through these stages:
- The request takes a seat under the router in-flight limit.
- The router prices each eligible prefill engine from its profile over the prompt and its resident prefill requests, with prefix-cache pricing for cached prefixes.
- With the default
serving.admissionofpredictive, the router rejects a request that fails the projected time to first token (TTFT) check on the cheapest available prefill path or the decode admission check. - On vLLM, the chosen prefill engine holds the prompt KV as the producer and returns a typed KV handoff. On SGLang, the router also places the decode leg and sends both legs together.
- An eligible engine consumes the handoff and runs decode.
- Tokens stream to the client.
- The request journal records admission, placement, retries, transfers, timing, and the final outcome.
When every seat is occupied, serving.queue_capacity sets the outcome. With a positive capacity and space in the queue, the request waits in a bounded FIFO queue within serving.queue_timeout_s and its original deadline. A full queue, or the default capacity of 0, gives a retryable refusal.
A retry reruns prefill and decode with a fresh KV handoff.
Fleet topology¶
A fleet's topology sets how its engines divide prefill and decode work.
Aggregated serving¶
Every engine does both prefill and decode, with local KV. Long prefills share a scheduler with decode batches.
Static disaggregation¶
A fixed prefill pool and a fixed decode pool serve requests. An operator changes pool membership by hand. To configure it, set each engine's role and set pin to true on every engine.
Adaptive cold-swap¶
Engines change pools by draining and relaunching. Capacity arrives after restart, weight load, and validation.
A cold-swap move runs these steps:
- Drain the engine.
- Relaunch it in the new role.
- Reload its weights.
- Register its transfer peers.
- Pass health validation.
Use cold-swap for traffic shifts longer than this sequence.
Adaptive hot-swap¶
Dual-capability engines form logical prefill and decode pools. Hot-swap changes an eligible engine's scheduler role in place, while its weights stay loaded.
Role-change guards limit role changes:
- a cooldown
- a minimum dwell time
- confirmation rules
- minimum role floors
- a check on resident work
- exclusion of unhealthy engines
- exclusion of engines in a lifecycle event