Skip to content

Metrics and role controller state

Prometheus metrics

For the JSON state document, use GET /narwhal/state. For the Grafana panels, follow Reading the dashboard.

A new router process starts each series at these values:

Series Starting value
narwhal_offered_total, narwhal_unsized_offered_total, narwhal_served_total, narwhal_slo_met_total, narwhal_expired_total, narwhal_invalid_requests_total, narwhal_failed_total, narwhal_unserved_total, narwhal_refused_total, narwhal_rejected_total, narwhal_cancelled_total Handoff values on resume and standby takeover, otherwise 0.
narwhal_retry_credits serving.retry_budget.
Reason series of narwhal_failed_total, narwhal_refused_total, narwhal_rejected_total, and narwhal_expired_total Handoff values on resume and standby takeover, otherwise 0. A handoff without reason counts restores its totals under unclassified.
Histograms and every other counter 0.

Split journal rows by run when comparing restored outcome counts with offered counts.

Metric families

The router exports these series on /metrics, grouped by area:

Area Series
Contract identity narwhal_contract_info
Admission mode narwhal_admission_info, narwhal_admission_margin
Router lease narwhal_router_ready, narwhal_router_lease_epoch
Engine monitoring narwhal_monitoring_degraded, narwhal_monitoring_core_consecutive_failures, narwhal_monitoring_core_failures_total, narwhal_monitoring_stage_failures_total, narwhal_monitoring_stage_consecutive_failures, narwhal_event_loop_lag_seconds, narwhal_event_loop_lag_high_water_seconds, narwhal_event_loop_busy_seconds_total
Request outcomes narwhal_offered_total, narwhal_unsized_offered_total, narwhal_expired_total, narwhal_served_total, narwhal_slo_met_total, narwhal_failed_total, narwhal_unserved_total, narwhal_refused_total, narwhal_rejected_total, narwhal_cancelled_total, narwhal_invalid_requests_total
Attempts and quota narwhal_prefill_attempts_total, narwhal_decode_attempts_total, narwhal_retry_attempts_total, narwhal_attempt_failures_total, narwhal_served_after_retry_total, narwhal_retry_credits, narwhal_retry_credits_spent_total, narwhal_retry_denied_total, narwhal_decode_tokens_observed_total, narwhal_output_tokens_total, narwhal_upstream_seconds_total
Queueing narwhal_queued, narwhal_queue_capacity, narwhal_queue_high_water, narwhal_waiting_prefill, narwhal_waiting_decode, narwhal_queue_wait_seconds
In-flight limit and saturation narwhal_admission_inflight, narwhal_admission_inflight_limit, narwhal_http_retained, narwhal_http_retained_limit, narwhal_http_retained_high_water, narwhal_router_loop_lag_seconds, narwhal_request_sizing_delay_seconds, narwhal_saturation_threshold_seconds
Pools narwhal_pool_instances, narwhal_pool_load, narwhal_instance_role, narwhal_resident_requests, narwhal_engine_requests_total, narwhal_engine_seats, narwhal_kv_lease_seconds, narwhal_handoff_bound_seconds
Health narwhal_ejected_instances, narwhal_ejected, narwhal_engine_quarantined, narwhal_engine_held, narwhal_engine_ejections_total, narwhal_engine_hold_starts_total, narwhal_engine_hold_ends_total, narwhal_engine_probes_total, narwhal_engine_readmissions_total, narwhal_probation_instances, narwhal_health_windows_scored_total, narwhal_health_windows_undersampled_total, narwhal_health_prefill_paused, narwhal_health_prefill_pauses_total, narwhal_engine_breaker_streak, narwhal_engine_breaker_verifying
Floors narwhal_prefill_below_floor, narwhal_decode_floor, narwhal_decode_below_floor, narwhal_prefill_below_floor_events_total, narwhal_prefill_below_floor_seconds_total, narwhal_decode_floor_restorations_total
Role controller narwhal_flips_total, narwhal_flip_reversals_total, narwhal_flips_refused_total, narwhal_flip_inflight_total, narwhal_controller_advisory, narwhal_controller_decisions_total, narwhal_controller_proposed_engines, narwhal_controller_phase_work_engines, narwhal_controller_projected_slo_ratio, narwhal_controller_objective, narwhal_controller_decode_tokens_per_engine, narwhal_controller_decode_requests_per_engine, narwhal_controller_decode_model, narwhal_controller_last_decision
Demand history narwhal_demand_history_cells, narwhal_demand_history_cell_limit, narwhal_demand_history_observations, narwhal_demand_history_overflow_observations
Attainment evidence narwhal_attainment_evidence_covered_seconds, narwhal_attainment_evidence_outcomes, narwhal_attainment_evidence_buckets, narwhal_attainment_evidence_pruned_total
Consolidation narwhal_demand_evidence_span_seconds, narwhal_demand_evidence_arrivals, narwhal_demand_evidence_closed, narwhal_demand_evidence_risk_age_seconds, narwhal_demand_evidence_short_decode_engines, narwhal_demand_evidence_envelope_decode_engines, narwhal_demand_evidence_trend_ratio, narwhal_demand_evidence_refused, narwhal_demand_evidence_risk_events_total
Latency narwhal_slo_seconds, narwhal_ttft_seconds, narwhal_tpot_seconds, narwhal_seat_seconds
Lifecycle narwhal_engine_draining, narwhal_engine_ready_to_stop, narwhal_engine_lifecycle_state

narwhal_engine_requests_total counts the requests the router has sent to each engine since it started, including its verification, readmission and peer release probes.

Tool metrics

With --metrics-url or NARWHAL_TOOL_METRICS_URL, narwhal-check serves these gauges on /metrics during preflight and first-token calibration, and narwhal-profile serves them during a live sweep. Each series carries an operation label of preflight, calibration or profiling.

Series Value
narwhal_tool_operation 1 while the operation runs
narwhal_tool_engine_in_use 1 for each engine the operation sends requests to, labelled with the engine's iid
narwhal_tool_steps_completed Steps completed
narwhal_tool_steps_total Steps planned

Reading outcome reasons

These series split request outcomes and failed attempts by their journal reason:

Metric Type Labels Value
narwhal_failed_total counter reason Requests that ended in an error.
narwhal_refused_total counter cause Predictive refusals: queue, prompt, aggregate_unpriced, slot_wait, kv_capacity, or tpot.
narwhal_rejected_total counter reason Capacity and router-readiness rejections.
narwhal_expired_total counter reason Expiries: deadline, queue_timeout, or handoff for a KV handoff expiry.
narwhal_attempt_failures_total counter phase, reason Entries added to journal attempt_failures, present after the first failure.
narwhal_served_after_retry_total counter Completed requests whose final attempt followed a failed one.
narwhal_admission_info gauge mode 1 on the series for serving.admission.
narwhal_admission_margin gauge serving.admission_margin.

The four outcome counters export every documented reason other than unclassified from process start, at 0 until it occurs. An unclassified series appears once the router counts one. Sum a counter over its reason or cause label for the total that /narwhal/state reports.

Reading engine lifecycle gauges

The lifecycle gauges report fields of each engine's record in GET /narwhal/lifecycle.

Metric Type Labels Value
narwhal_engine_draining gauge iid 1 when draining is true, otherwise 0.
narwhal_engine_ready_to_stop gauge iid 1 when ready_to_stop is true, otherwise 0.
narwhal_engine_lifecycle_state gauge iid, state 1 on the series whose state label matches the engine's state.

Inspecting scheduling and role control

These metrics track role changes and pool load:

Metric Meaning
narwhal_engine_seats Each engine's seats, labelled by iid and phase (prefill or decode), 0 for no limit.
narwhal_flip_reversals_total Moves whose target role differs from the same engine's previous recorded move, counted from that engine's second move.
narwhal_flips_refused_total Role changes blocked by timing, availability, role pins, role floors, the resident guard, shared-device profile coverage, or advisory mode.
narwhal_pool_load Pool load normalized per phase in Role control, with 1.0 at the phase target.

Reading latency histograms

narwhal_slo_seconds exports the configured ttft and tpot budgets through the metric label. Its slo label, and the slo label on each latency histogram, carry the target that scaled the histogram's bucket edges.

Histogram Measures Bucket scale
narwhal_ttft_seconds Arrival to prefill completion. slo.ttft_s
narwhal_tpot_seconds Mean seconds per output token after prefill completion. slo.tpot_s
narwhal_queue_wait_seconds Wait per original request at each stage it reached: admission, prefill seat, or decode seat. serving.queue_timeout_s, or slo.ttft_s when it is 0
narwhal_seat_seconds Time a request holds an admission seat. slo.ttft_s

Bucket boundaries, as multiples of the bucket scale:

0.025, 0.05, 0.1, 0.2, 0.35, 0.5, 0.7, 1.0, 1.5, 3.0, 10.0, +Inf

Histogram aggregation:

  • Compute quantiles from bucket rates grouped by instance, slo and le, and by stage for narwhal_queue_wait_seconds.
  • Divide a narwhal_ttft_seconds or narwhal_tpot_seconds quantile by the narwhal_slo_seconds series with the matching metric, instance and slo.

Inspecting retained attainment evidence

narwhal_attainment_evidence_pruned_total appears with a kind label of buckets or outcomes after the first buckets age out of the attainment retention window.

Inspecting demand history and decode floor

Demand histories clear on router restart.

Metric Type Labels Meaning
narwhal_decode_floor gauge Configured min_decode.
narwhal_demand_history_cells gauge window: unsized, arrivals, expected_decode, observed_decode, residency Retained demand cohorts.
narwhal_demand_history_cell_limit gauge same window values Maximum retained cohorts.
narwhal_demand_history_observations gauge same window values Original observations represented by the window.
narwhal_demand_history_overflow_observations gauge same window values Observations merged past the cohort limit.

Inspecting consolidation gates

These metrics expose the consolidation evidence that gates decode-to-prefill moves:

Metric Type Labels Meaning
narwhal_demand_evidence_span_seconds gauge Duration of the retained arrival window in seconds.
narwhal_demand_evidence_arrivals gauge Arrival samples in the retained arrival window.
narwhal_demand_evidence_closed gauge 1 after the evidence window closes.
narwhal_demand_evidence_risk_age_seconds gauge Age of the newest risk event in seconds.
narwhal_demand_evidence_short_decode_engines gauge Short-horizon decode demand in engine equivalents.
narwhal_demand_evidence_envelope_decode_engines gauge Conservative decode-demand envelope in engine equivalents.
narwhal_demand_evidence_trend_ratio gauge Short-horizon demand over long-horizon demand, present once a long-horizon estimate exists.
narwhal_demand_evidence_refused gauge gate: risk, evidence, trend 1 when the gate blocks consolidation.
narwhal_demand_evidence_risk_events_total counter kind: first_token_timeout, p_to_d_recovery Risk-event count, present after the first risk event.