Router state and placement monitoring¶
Interpreting router state¶
/health¶
/health returns HTTP 200 in every router state with this body:
| Field | Value |
|---|---|
status |
Router state: ok, degraded, maintenance, standby, or fenced |
instances |
Configured fleet size |
available_instances |
Engines eligible for placement: the configured fleet minus ejected, draining, and quarantined engines |
Backend exhaustion¶
When every engine is excluded from placement, the active router's /health reports status: degraded and /ready returns HTTP 503 with reason: no available engines. New completion requests receive HTTP 503 with the retryable error code backend_unavailable.
Whole-wave lifecycle hold¶
Until whole-wave readmission succeeds, the active router's /health reports status: maintenance and /ready returns HTTP 503 with the lifecycle reason as reason. New completion requests receive HTTP 503 with error code standby and the lifecycle reason as the error message.
Temporary holds¶
Under last-engine protection, the router starts no timed quarantine or inference hold on an engine that alone serves one of its roles. When another engine's ejection or drain leaves a held engine alone in its role, the router ends that hold.
Router control and replacement¶
The HTTP status of /ready shows traffic eligibility, and load balancers route client traffic by it. control_ready: true in /ready means the router holds a valid lease and its engine monitoring is healthy. A standby copies the state handoff from a primary that reports it, even while that primary's /ready returns HTTP 503.
A replacement router:
- runs the same release
- applies the same lifecycle rules
- keeps saved ejections until recovery succeeds
Monitoring placement and control¶
To set up Prometheus and Grafana, follow Setting up observability.
Use the router metrics to answer these questions:
| Signal | Operational question |
|---|---|
| Readiness and lease epoch | Which router admits traffic? |
| Pool size and normalized load | Has either phase reached its measured limit? |
| Queue depth, pressure, and sheds | Is overload waiting, protected, or refused? |
| Ejection, probation, and role floors | How much healthy capacity remains? |
| TTFT, TPOT, queue wait, and seat time | Where is client latency accumulating? |
| Retries, failures, refusals, and rejections | Which protection path is active? |
| Role changes, reversals, and blocked decisions | Is the role controller holding a stable role assignment? |
tools/observability/prometheus-alerts.yml defines these alert rules:
| Alert | Condition | Duration | Severity |
|---|---|---|---|
NarwhalRouterDown |
Router scrape fails | 2m | page |
NarwhalEngineDown |
Engine scrape fails | 30s | page |
NarwhalEngineEjected |
At least one ejected engine | 1m | page |
NarwhalErrorBurst |
Failed requests above 0.5/s over 5m | 5m | warn |
NarwhalRejectedRising |
Rejected requests above 1% of offered requests over 5m | 5m | warn |
NarwhalExpiredRising |
Expired requests above 1% of offered requests over 5m | 5m | warn |
NarwhalRetryDeniedRising |
Retries denied by the shared retry quota above 1% of offered requests over 5m | 5m | warn |
NarwhalUnservedRising |
Phase placements where every eligible candidate exceeds the configured SLO, above 0.2/s over 10m | 10m | warn |
NarwhalPoolStarved |
A pool with zero engines | 2m | warn |
NarwhalPrefillBelowFloor |
Live prefill capacity below min_prefill |
1m | warn |
NarwhalDecodeBelowFloor |
Live decode capacity below min_decode |
1m | warn |
NarwhalErrorBurst, NarwhalRejectedRising and NarwhalExpiredRising sum their counter over its reason label, so each threshold applies to the whole outcome. NarwhalRejectedRising, NarwhalExpiredRising and NarwhalRetryDeniedRising divide by narwhal_offered_total, which includes early refusals and rejections. Their 1% threshold is the share at which the dashboard's Dropped headline turns yellow.
Request journals record per-request placement and timing. Metrics hold process-level summaries since the last counter reset.