Troubleshooting a fleet¶
Capturing router and engine state¶
Collect one bundle per incident router, with:
- a fresh output path
- that router's fleet configuration
- that router's run directory
Collect a bundle:
mkdir -p runs/diagnostics
narwhal diagnostics collect \
--router http://router:8000 \
--fleet config/fleet.json \
--run runs/dev/run-example \
--out runs/diagnostics/router-incident-001
On exit status 3, inspect the source rows in the partial bundle's manifest.
Add --artifact PATH to include ingress and supervisor status, engine boot logs, profiles, or deployment load results from outside the selected run. Add --include-request-content to include journal and completion content.
Before stopping an engine, capture evidence per the planned restart or unplanned-failure procedure.
Router, admission, and lifecycle signals¶
Start from the router's health and readiness:
- If a request to
/healthfails, query the peer's/readyto identify the lease holder. Inspect the router process, host, and network path. - If
/healthreportsstandby, send traffic and lifecycle actions to the active lease holder. - If
/healthreportsfenced, identify the current lease holder and remove the fenced router from the load balancer. - If
/healthreportsmaintenance, follow/narwhal/lifecyclethrough the engine wave until readiness returns. - If both routers return HTTP 503 from
/ready, compare their refusal reasons. Inspect backend health, lifecycle holds, engine monitoring, the lease holder, and state handoff freshness.
For rising client errors:
- If HTTP 429 responses increase, classify them by outcome reason before changing capacity.
- If HTTP 502 responses increase, inspect engine failures, ejection, quarantine, and in-flight work.
- If HTTP 504 responses increase, separate queue and request expiry from engine timeouts with the response error and terminal journal row.
- If a stream ends with an error frame after the HTTP 200 response starts, inspect the failed attempts, final outcome, and participating engines.
If the lifecycle state is blocked, repair the failed drain identity capture or readmission check, and retry that operation.
Procedures by path:
-
Fleet overload with healthy engines
HTTP 429, 503, or 504 increases, classified by outcome reason.
-
Engine and whole-wave recovery
An engine failed unexpectedly, or the fleet needs a whole-wave restart.
-
The primary router failed, or a standby reports a stale or incompatible state handoff.
Fleet overload with healthy engines¶
Use this procedure when HTTP 429, 503, or 504 responses rise while engines pass health checks. If the router has ejected or held engines, follow Engine and whole-wave recovery first.
Classify outcomes by reason¶
Each rejected, refused, expired, or failed request increments one outcome counter on the router's /metrics, labelled with the outcome reason that ended it. Query each counter over the incident window, grouped by its label:
sum by (reason) (increase(narwhal_rejected_total[5m]))
sum by (cause) (increase(narwhal_refused_total[5m]))
sum by (reason) (increase(narwhal_expired_total[5m]))
sum by (reason) (increase(narwhal_failed_total[5m]))
To classify individual requests, count the router journal's terminal rows by outcome. Set JOURNAL to the router's journal path, journal.jsonl beside profiles.path unless narwhal-serve --journal sets another path:
jq -r 'select(.terminal != null and .terminal != "completed")
| [.terminal, .reason, .status, .error_type, .error_code] | @tsv' "$JOURNAL" \
| sort | uniq -c | sort -rn
Each output line counts one combination of terminal state, reason, HTTP status, error type, and error code. A null field prints as an empty column.
Match the largest counts to a class:
| Counter and label | Journal terminal |
Client response | Class |
|---|---|---|---|
narwhal_rejected_total{reason="inflight_limit"} |
rejected |
HTTP 429, message router in-flight limit reached |
In-flight limit |
narwhal_rejected_total{reason="saturated"} |
rejected |
HTTP 429, message starting router saturated |
Router saturation |
narwhal_rejected_total{reason="not_ready"} |
rejected |
HTTP 503 standby or backend_unavailable |
Router readiness, not overload. Read readiness_reason and follow Router, admission, and lifecycle signals. |
narwhal_refused_total{cause="queue"} or {cause="prompt"} |
refused |
HTTP 429 | TTFT refusals |
narwhal_refused_total{cause="aggregate_unpriced"} |
refused |
HTTP 429 | TTFT refusals |
narwhal_refused_total{cause="slot_wait"}, {cause="kv_capacity"}, or {cause="tpot"} |
refused |
HTTP 429 | Decode capacity refusals |
narwhal_expired_total{reason="queue_timeout"} or {reason="deadline"} |
expired |
HTTP 504 | Wait and deadline expiries |
narwhal_expired_total{reason="handoff"} |
expired |
HTTP 504 handoff_expired |
KV handoff expiries |
narwhal_failed_total{reason="engine_overloaded"}, {reason="engine_timeout"}, or {reason="local_pool"} |
failed |
HTTP 502 or 504 | Engine-side overload |
To match a response's status, error type, and code to its cause, use Admission and refusal semantics and Engine failure handling.
Diagnose the class¶
In-flight limit rejections¶
Check: narwhal_http_retained reaches narwhal_http_retained_limit, which is the in-flight limit plus serving.queue_capacity. narwhal_admission_inflight stays at narwhal_admission_inflight_limit.
Cause: requests arrive faster than admitted requests release their seats. The router rejects each arrival before it reads the body, so the predictive checks never run for these requests.
Action: reduce ingress traffic, or add engines that passed deployment validation. Before raising serving.max_connections, measure the in-flight load the fleet sustains while healthy.
Router saturation rejections¶
Check: compare narwhal_router_loop_lag_seconds and narwhal_request_sizing_delay_seconds with narwhal_saturation_threshold_seconds. The signal at or above the threshold caused the rejection.
Cause: a late router event loop, or slow token counting and prefix hashing. With engine.tokenize on, sizing includes the exact count from an engine.
Action: for loop lag, inspect CPU load on the router host and rate(narwhal_event_loop_busy_seconds_total[5m]). For sizing delay, inspect the latency of the engines' exact-count requests. Reduce ingress traffic to this router until the signal falls below the threshold.
TTFT refusals¶
Check: read admission_price on the refused journal rows. Its parts are backlog_s, own_prefill_s, elapsed_s, and handoff_s, and price_s is their sum.
cause |
Cause | Action |
|---|---|---|
queue |
Prefill work resident on the cheapest prefill engine, backlog_s, the time the request already waited, elapsed_s, and the wait for the first decode token after prefill, handoff_s, push price_s past the TTFT budget. |
Reduce the offered rate. Add prefill capacity when backlog_s dominates, or decode capacity when handoff_s dominates. Clients can retry after Retry-After, which excludes the time the request already waited. |
prompt |
The prompt's own prefill, own_prefill_s, exceeds the TTFT budget. |
Shorten the prompt, or raise slo.ttft_s. Draining the prefill backlog does not clear this cause. |
aggregate_unpriced |
Zero prefill engines are live, and every candidate engine carries decode work. | Check pools.prefill and ejected in /narwhal/state, then restore a live prefill engine. controller.min_prefill sets the prefill role floor. |
Decode capacity refusals¶
Check: the journal rows carry refused_cause: decode, and reason names the failed decode admission check: slot_wait, kv_capacity, or tpot. Compare resident decode counts in /narwhal/state with narwhal_engine_seats{phase="decode"}.
Cause: projected decode load exceeds the live decode engines' seats, KV capacity, or slo.tpot_s.
Action: reduce the offered rate, or add decode capacity.
Wait and deadline expiries¶
Check: read queue_waits on the expired journal rows, and narwhal_queue_wait_seconds by stage.
reason |
Stage with the longest wait | Cause | Action |
|---|---|---|---|
queue_timeout |
admission |
Every admission seat stayed held for serving.queue_timeout_s. |
Treat as in-flight limit pressure. |
queue_timeout |
prefill |
Every prefill engine's seats stayed full for the rest of serving.queue_timeout_s. |
Reduce the offered rate, or add prefill capacity. |
deadline |
any | The request reached serving.request_timeout_s. |
Compare queue_waits, ttft_s, and upstream_seconds to find the stage that used the time. |
Raising serving.queue_timeout_s lengthens waits without adding seats.
KV handoff expiries¶
Check: narwhal_waiting_decode stays above 0, and the expired rows' queue_waits.decode approaches the producer's narwhal_handoff_bound_seconds.
Cause: on vLLM, every decode engine's seats stayed full until prefilled requests reached their KV handoff bound. Past the bound, the router ends the attempt instead of dispatching its decode leg. On SGLang, the router places the decode leg before it starts the prefill leg, so this expiry occurs only on vLLM.
Action: reduce the offered rate, or add decode capacity. With serving.max_attempts above 1, as in the default of 2, an expiry with attempts left retries with a fresh prefill, which adds prefill load.
Engine-side overload¶
Check: read narwhal_attempt_failures_total by phase and reason, and each journal row's attempt_failures. Each entry's retry_reason records why the router retried the attempt or ended the request. shared_budget with a rising narwhal_retry_denied_total means the retry credits ran out.
reason |
Meaning |
|---|---|
engine_overloaded |
The engine returned HTTP 408 or 429. |
engine_timeout |
The engine returned HTTP 504, or a prefill, first-token, or between-token timeout expired. |
local_pool |
A wait for a router data connection to the engine reached engine.pool_timeout_s. |
Cause: engines receive more work than they finish before their timeouts. This happens in open admission mode, which skips the predictive checks, or when prefill and decode seats fill faster than engines finish requests.
Action: reduce the offered rate, or add capacity. In open mode, compare a run in predictive mode.
Confirm the binding limit and verify recovery¶
- Keep the request mix,
serving.max_connections, queue settings, and timeouts fixed. - Test two offered rates under those conditions.
- Compare completed throughput and the share of requests meeting the service-level objective, from
narwhal_served_totalandnarwhal_slo_met_total. - Reduce ingress traffic, or add a fleet that passed deployment validation, before raising a limit.
Before you change an admission, queue, retry or in-flight setting, read Choosing admission, queue and retry settings for the client outcomes of each value.
The class is resolved when the counter series that identified it stops increasing at the offered rate, and the journal shows no new terminal rows with that reason.
The load-test router outcome ratio divides router completions by admitted requests. Deployment attainment divides the client completions that meet the service-level objective by all scheduled offers, including cancellations, predictive refusals, and unsent scheduling misses.
Validating recovery¶
Run the release validation drills.