Choosing admission, queue and retry settings¶
Serving and role control defines field defaults, bounds, and router behaviour. Admission and refusal semantics and engine failure handling define client responses.
Restart the router after changing a fleet setting.
Admission mode¶
serving.admission selects predictive or open admission. The request journal records the price used for each prefill placement.
- Keep predictive admission when clients should receive a refusal instead of a first token after the TTFT target.
- Use open admission for an in-flight sweep or another measurement without predictive refusals.
- Compare
price_swithttft_son completed journal rows before relying on predictive admission. Remeasure the engine profiles when the median ratio is materially above or below 1. - Raise
serving.admission_marginonly when you accept first tokens beyondslo.ttft_s.
Queue¶
serving.queue_capacity and serving.queue_timeout_s control waits for admission and engine seats. Queue waits defines their bounds and outcomes.
Turn the queue on for short bursts when clients should wait for a seat. Set serving.queue_timeout_s to the longest acceptable wait before prefill. Review journal refusals for long, mostly cached prompts before enabling it under predictive admission.
Engine seats and KV handoff¶
The router sets engine seats from attestation and profiles. To change decode seats, relaunch with a different runtime.max_sequences and attest again, or remeasure the profile. Prefill seats follow the profile and slo.ttft_s.
The KV handoff bound comes from the lease in engine attestation. runtime.kv_lease_s sets the lease at engine launch. A longer lease permits a longer decode-seat wait and retains abandoned handoff blocks longer. Relaunch and attest engines after changing it. KV handoff expiry defines the client outcome.
Retries and quarantine¶
serving.max_attempts and retry credits bound retries before visible output. Retries defines eligible failures, placement, responses, and stream behaviour.
- Set
serving.max_attemptsto1when clients retry failed requests themselves. - Set it to
3only when each role keeps at least three live engines, including unpinned engines that aggregate fallback can use. - Raise
serving.retry_budgetwhennarwhal_retry_denied_totalrises during an engine failure and the remaining engines have spare capacity. - Lower
serving.retry_replenishto limit prefill load from retries during a sustained failure.
recovery.failure_quarantine_s controls the timed hold after a failed request leg. Failure quarantine defines coverage and release. Set a positive value when one engine fails successive requests faster than the breaker removes it and the remaining engines can carry the load.
In-flight limit and decode gaps¶
serving.max_connections protects router resources; narwhal-serve --max-concurrent can lower its in-flight limit. Use an open-admission sweep to find the first saturation rejection, then set the limit at or below that count. In-flight limit rejections gives the production diagnosis.
engine.decode_read_timeout_s bounds decode silence. Decode timeouts defines the client and journal outcomes. Raise the limit when healthy streams show longer inter-chunk gaps; lower it to end a hung stream sooner while keeping it above the largest healthy gap.
Check a changed setting¶
Run the candidate against the current value with the same request mix and offered load:
- On a test fleet, apply the value with a configuration overlay, or edit the fleet file and restart the router.
- Run load at and above fleet capacity with a load job.
- Classify outcomes with Fleet overload with healthy engines.
- Compare completed requests and requests meeting the SLO from
narwhal_served_totalandnarwhal_slo_met_totalwith the current-value run.
The dashboard shows admission in-flight, queue depth, waits, and retry credits.