Live router and scheduler state¶
GET /narwhal/state¶
Returns the live router and scheduler state as a narwhal.state schema version 1 document.
For the Prometheus series, use Metrics and role controller state. For their Grafana panels, use Reading the dashboard.
Top-level fields¶
The state document carries these top-level fields:
| Field | Meaning |
|---|---|
schema |
narwhal.state |
schema_version |
State schema version, 1 |
journal_run |
Run ID of the current router process in the request journal |
served |
Completed requests |
slo_met |
Completed requests that met both slo.ttft_s and slo.tpot_s |
failed |
Requests that ended in an error |
offered |
Completion requests received |
unsized_offered |
Requests that ended before workload sizing |
expired |
Requests whose deadline expired |
cancelled |
Requests whose client disconnected |
invalid_requests |
Requests refused with HTTP 400, 404, or 413 before admission |
controller |
Active role controller, reactive |
token_accounting |
token_ids when the engines supply token IDs, reported when their streams report each reply's output token count, otherwise unavailable |
control |
Role-controller mode and decision counts |
monitoring |
Timing and failure state of the engine monitoring loop |
ha |
High-availability state: readiness, standby status, lease, and failover block |
lifecycle |
Drain state, resident work, and lifecycle events |
admission |
Router and phase occupancy, queue state, limits, and admission mode |
outcome_reasons |
failed, refused, rejected, and expired counts by reason |
seats |
Each engine's prefill and decode seats and the inputs that set them |
handoff |
Each engine's attested KV lease and the handoff bound computed from it |
serving |
Retained HTTP work, attempts, and retry state |
http_pools |
Data and control connection pools, and the pool-wait timeout |
pools |
Engines grouped by prefill or decode role |
load |
Load of each pool as a ratio to the service-level objective (SLO) target, 1.0 at target |
thresholds |
Thresholds the reactive controller applies |
slo |
Time to first token (TTFT) and time per output token (TPOT) targets |
first_token_timeout_s |
Deadline to the first decode token |
first_token_calibration |
First-token calibration verified at router startup, with a label for each engine |
resident |
In-flight prefill and decode work on each engine |
pinned |
Engines excluded from role changes |
min_prefill |
Configured minimum number of live prefill engines |
min_decode |
Configured minimum number of live decode engines |
below_floor |
Current and cumulative breaches of the prefill floor |
ejected |
Engines the breaker ejected |
peer_release |
Peer release rounds for each engine out of placement |
draining |
Engines excluded by a lifecycle action |
probation |
Engines on probation, which carry a placement penalty |
health |
Drift-window accounting for each engine |
quarantined |
Engines under a timed quarantine or an inference hold |
holds |
The engines in quarantined, split into timed quarantines and inference holds |
breaker |
Failure streaks, probe state, and ejection, hold, probe and readmission counts |
residency |
Each engine's prefix-residency synchronization with its attestation sidecar |
decode_floor |
Decode floor state and restoration count |
attainment |
SLO outcome buckets for diagnostics |
demand_history |
Retained demand, shape counts, and overflow state |
demand_evidence |
Consolidation evidence used by decode-to-prefill gates |
unserved |
Phase placements where every eligible candidate exceeded the configured SLO |
panic_bypasses |
Prefill-to-decode moves that panic logic allowed during cooldown |
flips_refused |
The 20 most recent refused role changes |
flips |
The most recent role changes, up to flip_history |
After a resume or takeover, the new router process restores offered, unsized_offered, served, slo_met, failed, expired, cancelled, invalid_requests, unserved, admission.rejected, admission.refused, and outcome_reasons from the state handoff. Its other counters start at zero.
-
The drain state, resident work, and lifecycle events in
lifecycle. -
SLO attainment and demand accounting
The
attainment,demand_history, anddemand_evidenceobjects. -
The release rounds that
peer_releasereports. -
Per-request evidence for one client request.
monitoring¶
The monitoring object reports the engine monitoring loop:
| Field | Meaning |
|---|---|
degraded |
true while the router holds new admissions after controller.monitor_failure_limit consecutive failed passes |
reason |
First failure of the streak that set degraded, as <class>:<stage>, otherwise null |
core_consecutive |
Consecutive monitoring passes with a failed stage |
core_failures |
Monitoring passes with a failed stage since process start |
event_loop_lag_s |
Delay beyond the latest scheduled monitoring deadline, in seconds |
event_loop_lag_high_water_s |
Largest monitoring-deadline delay since process start, in seconds |
event_loop_busy_s |
Event-loop thread CPU seconds since the monitoring loop started |
stages |
Each monitoring stage's failures, consecutive, last_class and last_at (router monotonic-clock seconds of the latest failure) |
health¶
Each engine's health entry tracks its drift windows:
| Field | Meaning |
|---|---|
scored |
Closed drift windows that produced a score |
undersampled |
Closed drift windows whose evidence was too sparse to score |
last_scored_s_ago |
Seconds since the last scored window, null until a window scores |
prefill_paused |
true while prefill on this engine pauses evidence collection |
prefill_pauses |
Times prefill_paused changed to true |
A confirmed ejection resets the engine's health counts to zero.
breaker¶
breaker.failures holds each engine's consecutive failure streaks, keyed by failure class: connection, timeout, overload, inference_status, kv_handoff, stream, and liveness for missed health sweeps.
breaker.verifying lists each engine with a health or inference probe in flight, as its iid and the probe kind.
The remaining breaker fields count transitions since process start. Each is a list of rows with an iid, the fields below, and a count:
| Field | Row fields | Counts |
|---|---|---|
ejections |
cause |
Ejections, by ejection cause |
hold_starts |
kind |
Holds started, by hold kind |
hold_ends |
kind, cause |
Holds ended, by hold kind and hold-end cause |
probes |
kind, outcome |
Verification probes, by outcome |
readmissions |
evidence |
Returns of an ejected engine to placement, by recovery evidence |
holds¶
holds splits the engines in quarantined by hold kind. An engine appears under one kind at a time.
holds.timed lists engines under a timed quarantine from recovery.failure_quarantine_s:
| Field | Meaning |
|---|---|
iid |
Engine |
held_s |
Seconds since the hold started |
remaining_s |
Seconds until the quarantine deadline |
holds.inference lists engines held until an inference probe passes:
| Field | Meaning |
|---|---|
iid |
Engine |
held_s |
Seconds since the hold started |
recorded_producers |
Producers whose KV transfers to this engine failed, empty when none was recorded |
The probe tries producers in the order given in Inference probe producers.
peer_release¶
With recovery.engine_restart_policy: individual, peer_release has one entry for each ejected engine and for each engine in lifecycle state drained, deadline_exceeded, validating, or blocked:
| Field | Meaning |
|---|---|
rounds |
Release rounds sent since the ejection or the last state change |
next_round_s |
Seconds until the next round, null after the last round |
On SGLang, the router sends no release rounds, so each entry reports rounds: 0 and next_round_s: null.
residency¶
Each engine's residency entry reports the router's view of the engine's resident prefix blocks, synchronized from its attestation sidecar:
| Field | Meaning |
|---|---|
known |
true when the router holds a complete view of the engine's resident blocks |
reason |
Reason for the known value |
epoch |
Sidecar instance that the view follows |
sequence |
Last engine event batch the router applied |
block_size |
Tokens per cache block |
resident_blocks |
Resident blocks in each KV cache group |
resyncs |
Snapshots the router has taken since it started |
The router reports known: false in these cases:
| Case | reason |
|---|---|
| The router has yet to refresh the engine's residency | not yet synchronised |
The engine has an empty attestation_url |
engine has no attestation sidecar |
| The sidecar answers the residency snapshot route with HTTP 404 | engine publishes no cache events |
| The sidecar answers a residency refresh with an HTTP error status | residency refresh failed: HTTP <status> |
| A residency refresh fails with another error | residency refresh failed: <ExceptionClass> |
The sidecar snapshot reports known: false |
The sidecar's reason |
For an engine with known: false, the router prices each request's prefill on the cold curve of its full input.
The router refreshes each engine's residency view every controller.monitor_interval_s seconds.
The router takes a new snapshot when:
- the router starts
- the sidecar reports that the requested changes are gone
- the sidecar epoch changes
- the change sequence skips a number
- the previous refresh failed
first_token_calibration¶
first_token_calibration describes the calibration that the router verified at startup:
| Field | Meaning |
|---|---|
status |
measured, reused, or uncalibrated |
captured_at_unix |
Unix time of the calibration capture, null when uncalibrated |
candidate_deadline_s |
The calibration's candidate deadline in seconds, null when uncalibrated |
engines |
measured or reused for each engine, {} when uncalibrated |
status takes these values:
status |
Condition |
|---|---|
measured |
Every engine runs the process that the calibration timed |
reused |
One or more engines run a relaunched process with the same process generation |
uncalibrated |
engine.first_token_calibration_path is empty |
When the router readmits an engine after a relaunch, engines reports reused for that engine.
Admission and serving state¶
admission fields:
| Field | Meaning |
|---|---|
inflight |
Requests holding a router admission seat |
queued |
Current admission queue depth |
queue_capacity |
Configured maximum queue depth |
queue_high_water |
Peak queue depth |
waiting_prefill |
Requests waiting for prefill dispatch |
waiting_decode |
Requests waiting for decode dispatch |
limit |
In-flight limit: --max-concurrent when set, otherwise serving.max_connections |
loop_lag_s |
Latest router event-loop wake-up lateness |
sizing_delay_s |
Median request sizing delay over the last 2 seconds, 0 below 8 sized requests |
saturation_threshold_s |
Value of loop_lag_s or sizing_delay_s at which the router answers a saturation 429, a quarter of slo.ttft_s |
rejected |
HTTP 429 in-flight-limit and saturation rejections, and HTTP 503 router-readiness rejections |
refused |
Requests refused by global predictive admission |
engine_auth |
boundary or engine-credential |
mode |
serving.admission: predictive or open |
margin |
serving.admission_margin |
seats fields:
| Field | Meaning |
|---|---|
mean_input_len |
Mean sized input length that sets prefill seats, null before a request is sized in the window |
engines.<iid>.prefill |
Prefill seats, 0 for no limit |
engines.<iid>.decode |
Decode seats, 0 for no limit |
engines.<iid>.sequence_limit |
vLLM --max-num-seqs or SGLang --max-running-requests from the engine's verified attestation, null when the launch arguments omit it |
handoff fields, one entry per engine:
| Field | Meaning |
|---|---|
<iid>.kv_lease_s |
kv_lease_duration from the engine's verified attestation, null when the attestation records none |
<iid>.renewal_s |
The connector's lease-renewal interval, kv_lease_s // 6 |
<iid>.bound_s |
KV handoff bound for requests this engine prefills |
serving fields:
| Field | Meaning |
|---|---|
http_retained |
Completion requests counted against the in-flight limit, from arrival until the response ends |
http_retained_limit |
In-flight limit plus serving.queue_capacity |
http_retained_high_water |
Peak number of retained requests |
prefill_attempts |
Cumulative prefill dispatches |
decode_attempts |
Cumulative decode dispatches |
retry_attempts |
Prefill attempts after the first |
retry_credits |
Available shared retry credit |
retry_credits_spent |
Shared retry credit consumed |
retry_denied |
Retries that the retry budget refused |
served_after_retry |
Completed requests whose final attempt followed a failed one |
attempt_failures |
Failed attempts as phase, reason, and count rows |
decode_tokens_observed |
Decode tokens the router observed |
output_tokens |
Measured output tokens of completed requests |
upstream_seconds |
Total time in engine HTTP calls by phase, across all attempts |
Scheduler and controller state¶
Pool and SLO fields¶
These objects carry the pool, load, SLO, and floor state:
| Object | Fields |
|---|---|
pools |
prefill and decode arrays of engine IDs |
http_pools |
data_connections, control_connections, pool_timeout_s |
load |
prefill and decode load as floats relative to the SLO |
thresholds |
expand, shrink, cooldown_s, sustained_intervals, dwell_s, panic_ratio |
slo |
ttft_s, tpot_s |
resident.<iid> |
prefill and decode in-flight counts |
below_floor |
active, live_prefill, since, breaches, cumulative_s |
decode_floor |
min_decode, live_decode, below_floor, restoration_moves |
The below_floor and decode_floor fields hold these values:
| Field | Meaning |
|---|---|
below_floor.active |
true during a prefill-floor breach |
below_floor.live_prefill |
Prefill engines eligible for placement |
below_floor.since |
Start time of the open breach on the router's monotonic clock |
below_floor.breaches |
Breaches since the live prefill count first reached min_prefill |
below_floor.cumulative_s |
Total seconds in breach, across closed and open breaches |
decode_floor.live_decode |
Decode engines eligible for placement |
Controller decisions¶
control fields:
| Field | Meaning |
|---|---|
advisory |
true when the role controller records proposed role changes and keeps the live split fixed |
last_decision |
Most recent proposed or applied split, null until the first decision |
decisions |
Cumulative decision counts keyed <by>:<result> |
flips |
Cumulative role-change counts keyed <by>:<to> |
flip_reversals |
Role changes that reverse the same engine's previous role change |
flips_refused |
Cumulative count of refused role changes |
flip_inflight |
Cumulative resident requests carried through role changes, by phase |
control.last_decision has:
| Field | Meaning |
|---|---|
at |
Decision time on the router's monotonic clock |
prefill |
Proposed prefill engine count |
decode |
Proposed decode engine count |
by |
Caller that recorded the decision |
reason |
Why the role controller applied, held, or blocked the move |
result |
applied, held, blocked, or advisory |
applied |
true when result is applied |
When demand history is incomplete or fleet health changes before scoring, control.last_decision holds the inputs available at that stage. Depending on the stage, it can carry these fields:
| Field | Meaning |
|---|---|
current_prefill, current_decode |
Current prefill and decode split |
prefill_work, decode_work |
Estimated prefill and decode demand over the demand spans, in engines |
arrivals |
Arrivals in the prefill demand span |
output_observations |
Completed-output observations in demand history |
demand_complete |
Whether demand over controller.reactive.window_s is complete |
arrivals_beyond_profile |
Offers in controller.reactive.window_s with a prompt longer than some engine profile's prefill sweep |
unsized_offers |
Unsized offers in controller.reactive.window_s |
projected_ttft_ratio |
Demand model's projected TTFT for the candidate split, as a ratio to the TTFT target |
projected_tpot_ratio |
Demand model's projected TPOT for the candidate split, as a ratio to the TPOT target |
projected_decode_wait_ratio |
Candidate split's decode queueing ratio |
objective |
Candidate split's objective |
objective_delta |
Current objective minus candidate objective, positive for an improvement |
decode_request_limit |
Applied decode request limit |
decode_profile_covered |
Whether decode profiles cover the candidate split |
decode_correction |
Fleet median of each engine's bounded ratio of live to profiled decode latency |
decision_basis |
demand_projection, prefill_pressure_recovery, decode_pressure_recovery, or projected_ttft_recovery |
observed_prefill_ratio, observed_decode_ratio |
Observed phase pressure |
recovery_prefill_ratio, queued_prefill_s |
The prefill recovery ratio and its queued prefill seconds |
recovery_decode_ratio |
Value of the decode recovery ratio |
eligibility_rule |
Rule applied to a scored proposal |
demand_horizon_s |
Demand span that priced the proposal, in seconds |
steady_horizon_s |
Confirmation span, in seconds |
steady_prefill_work, steady_decode_work |
Prefill and decode demand over steady_horizon_s in engines, null with incomplete demand or on a projected-TTFT recovery evaluation |
steady_demand_s |
Seconds since confirmation-span demand began to match window demand, null while they differ, with incomplete demand, or on a projected-TTFT recovery evaluation |
departure_age_s |
Seconds since the open departure began |
departure_reverses |
Whether the open departure reverses the role controller's recent moves |
confirmations, required_confirmations |
Consecutive confirmations of an eligible proposal and the required count |
decode_capacity_safe |
Whether the candidate's decode work fits its decode capacity |
role_floors_safe |
Whether the candidate respects min_prefill and min_decode |
source_pressure_safe |
Whether source pool load is low enough to safely remove an engine, or meets an exception rule |
These decisions add fields to control.last_decision:
| Decision | Added fields |
|---|---|
| Scored | Decode capacity fields decode_tokens_per_engine, decode_slo_capacity_tokens, decode_kv_capacity_tokens, decode_requests_per_engine, pending_decode_requests, and pending_decode_tokens |
| Scored | Demand span fields demand_horizon_s, steady_horizon_s, steady_prefill_work, steady_decode_work, and steady_demand_s |
| Scored during an open departure | departure_age_s and departure_reverses |
| Decode-to-prefill | risk_kind, risk_age_s, and the demand_evidence fields with an evidence_ prefix |
eligibility_rule takes the first value that applies, in table order:
eligibility_rule |
Proposal |
|---|---|
projected_ttft_recovery |
Urgent decode-to-prefill evaluation triggered by an arriving request |
mixed_pressure |
Source load above shrink, and the receiving role at or above expand |
settled_departure |
A departure's first move, priced on confirmation-span demand |
steady_demand |
Steady-demand move with projected source load above shrink and at or below expand |
source_shrink |
Ordinary consolidation, and a leading departure's later confirmation-span moves |
Demand spans¶
demand_horizon_s, prefill_work, arrivals and decode_work cover these spans:
| Decision | demand_horizon_s |
prefill_work, arrivals |
decode_work |
|---|---|---|---|
| Hold before scoring | controller.reactive.window_s |
controller.reactive.window_s |
|
| A departure's confirmation-span move | Confirmation span | Confirmation span | Larger of residency over the latest controller.reactive.step_s and expected decode over the confirmation span |
| Other decode-to-prefill proposal | controller.reactive.window_s |
controller.reactive.window_s |
Larger of the controller.reactive.evidence_span_s and controller.reactive.window_s estimates |
| Every other scored decision | controller.reactive.window_s |
controller.reactive.window_s |
controller.reactive.window_s |
Projected-TTFT recovery fields¶
A projected-TTFT recovery evaluation adds these fields to control.last_decision:
| Field | Meaning |
|---|---|
trigger_rid |
Router request ID of the queued prefill request that triggered the evaluation |
projected_ttft_s |
Projected TTFT for that request |
ttft_slo_s |
Configured TTFT target |
trigger_projected_ttft_ratio |
projected_ttft_s divided by ttft_slo_s |
resident_prefill_s |
Profiled prefill seconds resident on live prefill engines |
queued_prefill_s |
Profiled prefill seconds waiting for prefill dispatch |
waiting_prefill |
Requests waiting for prefill dispatch |
initial_projected_ttft_s |
Projected TTFT when the first urgent signal arrived |
urgent_signals |
Urgent signals coalesced into this evaluation |
event_to_evaluation_s |
Seconds from the urgent signal to this evaluation |
A scored candidate that moves one engine from decode to prefill adds:
| Field | Meaning |
|---|---|
candidate_projected_ttft_s |
Trigger request's projected TTFT under the candidate split with the least favourable donor engine |
candidate_projected_ttft_ratio |
candidate_projected_ttft_s divided by the TTFT target |
projected_ttft_improvement_s |
projected_ttft_s minus candidate_projected_ttft_s |
Scored projected-TTFT recovery decisions set decision_basis=projected_ttft_recovery.
Blocked and held decisions keep the proposed split and objective change.
reason names the constraint that stopped the move:
- consolidation evidence
- profile coverage
- KV capacity
- role floors
- a role with zero live engines while fleet health is changing
- pins
- cooldown
- dwell
- the resident guard
- a departure's hold on the reverse move
A departure's hold sets one of these reason values:
reason |
Held move |
|---|---|
departure toward decode holds the reverse move |
Decode-to-prefill move after a departure toward decode |
departure toward prefill holds the reverse move |
Prefill-to-decode move after a departure toward prefill |
Role-change history¶
A role-change record has this form:
{
"at": 1712.4,
"iid": "e2",
"to": "decode",
"by": "reactive",
"prefill_inflight": 0,
"decode_inflight": 0,
"drained_s": 0.0
}
| Field | Meaning |
|---|---|
at |
Role-change time on the router's monotonic clock |
iid |
Engine ID |
to |
New role: prefill or decode |
by |
Caller that changed the role |
prefill_inflight |
Resident prefill work when the role label changes |
decode_inflight |
Resident decode work when the role label changes |
drained_s |
Seconds from the role change until its resident work finishes, null until then |
by values:
by |
Role change |
|---|---|
reactive |
Role-controller move |
decode_floor |
Decode-floor restoration toward min_decode |
floor_recovery |
Prefill-floor recovery, moving healthy decode engines to prefill down to min_decode |
Each flips_refused[] record contains at, to, and why.