Metrics and role controller state¶
Prometheus metrics¶
For the JSON state document, use GET /narwhal/state. For the Grafana panels, follow Reading the dashboard.
A new router process starts each series at these values:
| Series | Starting value |
|---|---|
narwhal_offered_total, narwhal_unsized_offered_total, narwhal_served_total, narwhal_slo_met_total, narwhal_expired_total, narwhal_invalid_requests_total, narwhal_failed_total, narwhal_unserved_total, narwhal_refused_total, narwhal_rejected_total, narwhal_cancelled_total |
Handoff values on resume and standby takeover, otherwise 0. |
narwhal_retry_credits |
serving.retry_budget. |
Reason series of narwhal_failed_total, narwhal_refused_total, narwhal_rejected_total, and narwhal_expired_total |
Handoff values on resume and standby takeover, otherwise 0. A handoff without reason counts restores its totals under unclassified. |
| Histograms and every other counter | 0. |
Split journal rows by run when comparing restored outcome counts with offered counts.
Metric families¶
The router exports these series on /metrics, grouped by area:
| Area | Series |
|---|---|
| Contract identity | narwhal_contract_info |
| Admission mode | narwhal_admission_info, narwhal_admission_margin |
| Router lease | narwhal_router_ready, narwhal_router_lease_epoch |
| Engine monitoring | narwhal_monitoring_degraded, narwhal_monitoring_core_consecutive_failures, narwhal_monitoring_core_failures_total, narwhal_monitoring_stage_failures_total, narwhal_monitoring_stage_consecutive_failures, narwhal_event_loop_lag_seconds, narwhal_event_loop_lag_high_water_seconds, narwhal_event_loop_busy_seconds_total |
| Request outcomes | narwhal_offered_total, narwhal_unsized_offered_total, narwhal_expired_total, narwhal_served_total, narwhal_slo_met_total, narwhal_failed_total, narwhal_unserved_total, narwhal_refused_total, narwhal_rejected_total, narwhal_cancelled_total, narwhal_invalid_requests_total |
| Attempts and quota | narwhal_prefill_attempts_total, narwhal_decode_attempts_total, narwhal_retry_attempts_total, narwhal_attempt_failures_total, narwhal_served_after_retry_total, narwhal_retry_credits, narwhal_retry_credits_spent_total, narwhal_retry_denied_total, narwhal_decode_tokens_observed_total, narwhal_output_tokens_total, narwhal_upstream_seconds_total |
| Queueing | narwhal_queued, narwhal_queue_capacity, narwhal_queue_high_water, narwhal_waiting_prefill, narwhal_waiting_decode, narwhal_queue_wait_seconds |
| In-flight limit and saturation | narwhal_admission_inflight, narwhal_admission_inflight_limit, narwhal_http_retained, narwhal_http_retained_limit, narwhal_http_retained_high_water, narwhal_router_loop_lag_seconds, narwhal_request_sizing_delay_seconds, narwhal_saturation_threshold_seconds |
| Pools | narwhal_pool_instances, narwhal_pool_load, narwhal_instance_role, narwhal_resident_requests, narwhal_engine_requests_total, narwhal_engine_seats, narwhal_kv_lease_seconds, narwhal_handoff_bound_seconds |
| Health | narwhal_ejected_instances, narwhal_ejected, narwhal_engine_quarantined, narwhal_engine_held, narwhal_engine_ejections_total, narwhal_engine_hold_starts_total, narwhal_engine_hold_ends_total, narwhal_engine_probes_total, narwhal_engine_readmissions_total, narwhal_probation_instances, narwhal_health_windows_scored_total, narwhal_health_windows_undersampled_total, narwhal_health_prefill_paused, narwhal_health_prefill_pauses_total, narwhal_engine_breaker_streak, narwhal_engine_breaker_verifying |
| Floors | narwhal_prefill_below_floor, narwhal_decode_floor, narwhal_decode_below_floor, narwhal_prefill_below_floor_events_total, narwhal_prefill_below_floor_seconds_total, narwhal_decode_floor_restorations_total |
| Role controller | narwhal_flips_total, narwhal_flip_reversals_total, narwhal_flips_refused_total, narwhal_flip_inflight_total, narwhal_controller_advisory, narwhal_controller_decisions_total, narwhal_controller_proposed_engines, narwhal_controller_phase_work_engines, narwhal_controller_projected_slo_ratio, narwhal_controller_objective, narwhal_controller_decode_tokens_per_engine, narwhal_controller_decode_requests_per_engine, narwhal_controller_decode_model, narwhal_controller_last_decision |
| Demand history | narwhal_demand_history_cells, narwhal_demand_history_cell_limit, narwhal_demand_history_observations, narwhal_demand_history_overflow_observations |
| Attainment evidence | narwhal_attainment_evidence_covered_seconds, narwhal_attainment_evidence_outcomes, narwhal_attainment_evidence_buckets, narwhal_attainment_evidence_pruned_total |
| Consolidation | narwhal_demand_evidence_span_seconds, narwhal_demand_evidence_arrivals, narwhal_demand_evidence_closed, narwhal_demand_evidence_risk_age_seconds, narwhal_demand_evidence_short_decode_engines, narwhal_demand_evidence_envelope_decode_engines, narwhal_demand_evidence_trend_ratio, narwhal_demand_evidence_refused, narwhal_demand_evidence_risk_events_total |
| Latency | narwhal_slo_seconds, narwhal_ttft_seconds, narwhal_tpot_seconds, narwhal_seat_seconds |
| Lifecycle | narwhal_engine_draining, narwhal_engine_ready_to_stop, narwhal_engine_lifecycle_state |
narwhal_engine_requests_total counts the requests the router has sent to each engine since it started, including its verification, readmission and peer release probes.
Tool metrics¶
With --metrics-url or NARWHAL_TOOL_METRICS_URL, narwhal-check serves these gauges on /metrics during preflight and first-token calibration, and narwhal-profile serves them during a live sweep. Each series carries an operation label of preflight, calibration or profiling.
| Series | Value |
|---|---|
narwhal_tool_operation |
1 while the operation runs |
narwhal_tool_engine_in_use |
1 for each engine the operation sends requests to, labelled with the engine's iid |
narwhal_tool_steps_completed |
Steps completed |
narwhal_tool_steps_total |
Steps planned |
Reading outcome reasons¶
These series split request outcomes and failed attempts by their journal reason:
| Metric | Type | Labels | Value |
|---|---|---|---|
narwhal_failed_total |
counter | reason |
Requests that ended in an error. |
narwhal_refused_total |
counter | cause |
Predictive refusals: queue, prompt, aggregate_unpriced, slot_wait, kv_capacity, or tpot. |
narwhal_rejected_total |
counter | reason |
Capacity and router-readiness rejections. |
narwhal_expired_total |
counter | reason |
Expiries: deadline, queue_timeout, or handoff for a KV handoff expiry. |
narwhal_attempt_failures_total |
counter | phase, reason |
Entries added to journal attempt_failures, present after the first failure. |
narwhal_served_after_retry_total |
counter | Completed requests whose final attempt followed a failed one. | |
narwhal_admission_info |
gauge | mode |
1 on the series for serving.admission. |
narwhal_admission_margin |
gauge | serving.admission_margin. |
The four outcome counters export every documented reason other than unclassified from process start, at 0 until it occurs. An unclassified series appears once the router counts one. Sum a counter over its reason or cause label for the total that /narwhal/state reports.
Reading engine lifecycle gauges¶
The lifecycle gauges report fields of each engine's record in GET /narwhal/lifecycle.
| Metric | Type | Labels | Value |
|---|---|---|---|
narwhal_engine_draining |
gauge | iid |
1 when draining is true, otherwise 0. |
narwhal_engine_ready_to_stop |
gauge | iid |
1 when ready_to_stop is true, otherwise 0. |
narwhal_engine_lifecycle_state |
gauge | iid, state |
1 on the series whose state label matches the engine's state. |
Inspecting scheduling and role control¶
These metrics track role changes and pool load:
| Metric | Meaning |
|---|---|
narwhal_engine_seats |
Each engine's seats, labelled by iid and phase (prefill or decode), 0 for no limit. |
narwhal_flip_reversals_total |
Moves whose target role differs from the same engine's previous recorded move, counted from that engine's second move. |
narwhal_flips_refused_total |
Role changes blocked by timing, availability, role pins, role floors, the resident guard, shared-device profile coverage, or advisory mode. |
narwhal_pool_load |
Pool load normalized per phase in Role control, with 1.0 at the phase target. |
Reading latency histograms¶
narwhal_slo_seconds exports the configured ttft and tpot budgets through the metric label. Its slo label, and the slo label on each latency histogram, carry the target that scaled the histogram's bucket edges.
| Histogram | Measures | Bucket scale |
|---|---|---|
narwhal_ttft_seconds |
Arrival to prefill completion. | slo.ttft_s |
narwhal_tpot_seconds |
Mean seconds per output token after prefill completion. | slo.tpot_s |
narwhal_queue_wait_seconds |
Wait per original request at each stage it reached: admission, prefill seat, or decode seat. |
serving.queue_timeout_s, or slo.ttft_s when it is 0 |
narwhal_seat_seconds |
Time a request holds an admission seat. | slo.ttft_s |
Bucket boundaries, as multiples of the bucket scale:
Histogram aggregation:
- Compute quantiles from bucket rates grouped by
instance,sloandle, and bystagefornarwhal_queue_wait_seconds. - Divide a
narwhal_ttft_secondsornarwhal_tpot_secondsquantile by thenarwhal_slo_secondsseries with the matchingmetric,instanceandslo.
Inspecting retained attainment evidence¶
narwhal_attainment_evidence_pruned_total appears with a kind label of buckets or outcomes after the first buckets age out of the attainment retention window.
Inspecting demand history and decode floor¶
Demand histories clear on router restart.
| Metric | Type | Labels | Meaning |
|---|---|---|---|
narwhal_decode_floor |
gauge | Configured min_decode. |
|
narwhal_demand_history_cells |
gauge | window: unsized, arrivals, expected_decode, observed_decode, residency |
Retained demand cohorts. |
narwhal_demand_history_cell_limit |
gauge | same window values |
Maximum retained cohorts. |
narwhal_demand_history_observations |
gauge | same window values |
Original observations represented by the window. |
narwhal_demand_history_overflow_observations |
gauge | same window values |
Observations merged past the cohort limit. |
Inspecting consolidation gates¶
These metrics expose the consolidation evidence that gates decode-to-prefill moves:
| Metric | Type | Labels | Meaning |
|---|---|---|---|
narwhal_demand_evidence_span_seconds |
gauge | Duration of the retained arrival window in seconds. | |
narwhal_demand_evidence_arrivals |
gauge | Arrival samples in the retained arrival window. | |
narwhal_demand_evidence_closed |
gauge | 1 after the evidence window closes. |
|
narwhal_demand_evidence_risk_age_seconds |
gauge | Age of the newest risk event in seconds. | |
narwhal_demand_evidence_short_decode_engines |
gauge | Short-horizon decode demand in engine equivalents. | |
narwhal_demand_evidence_envelope_decode_engines |
gauge | Conservative decode-demand envelope in engine equivalents. | |
narwhal_demand_evidence_trend_ratio |
gauge | Short-horizon demand over long-horizon demand, present once a long-horizon estimate exists. | |
narwhal_demand_evidence_refused |
gauge | gate: risk, evidence, trend |
1 when the gate blocks consolidation. |
narwhal_demand_evidence_risk_events_total |
counter | kind: first_token_timeout, p_to_d_recovery |
Risk-event count, present after the first risk event. |