SLO attainment and demand accounting¶
SLO attainment¶
attainment reports SLO results per time bucket:
| Field | Meaning |
|---|---|
bucket_s |
Bucket width, controller.monitor_interval_s |
retained_s |
Retention span, four demand windows counted back from the newest bucket |
covered_s |
Age of the oldest retained bucket, capped at retained_s |
buckets |
Retained buckets |
outcomes |
Requests counted in retained buckets |
pruned_buckets |
Buckets aged out of retention |
pruned_outcomes |
Requests in pruned buckets |
outcomes and pruned_outcomes count completed, failed, expired, and predictively refused requests.
Demand accounting¶
One demand window is controller.reactive.window_s seconds long.
Unsized offers¶
demand_history.unsized reports two counts:
| Field | Meaning |
|---|---|
pending |
Request bodies the router is reading |
observations |
Retained offers that ended before workload sizing |
Each unsized offer takes the mean price of the parsed offers in the same span.
Demand completeness¶
Demand in a span is incomplete when any of these is true:
- The fleet has zero engine profiles.
- The span has unsized offers and zero parsed offers.
- An uncapped offer's prompt bucket has zero delivered outputs to estimate its output from.
- An overflow cohort's output cap is zero.
- An expected decode shape has zero capacity at both its bucketed and exact length.
The demand estimate includes prompts longer than the profile's measured range, prefill_max_tokens. The router prices these prompts by extending the prefill fit to their length.
Input-size repricing¶
Demand history records each parsed offer at its arrival timestamp, with this input size:
| Offer | Recorded input size |
|---|---|
| On entry | Local input-size estimate |
| Tokenized after admission | Tokenization result, replacing the estimate |
| Rejected before tokenization | Local input-size estimate |
Bucketing and retention¶
Demand history buckets and retains its evidence as follows:
| Property | Value |
|---|---|
| Bucket width | Smallest of one second, the controller step, and the minimum evidence span |
| Exact request shapes | Up to 128 per bucket |
| Overflow cohort | One per bucket |
| Unsized offers | One shape per bucket |
| Arrival and residency data | Kept for one demand window |
| Completed output observations | Kept for four demand windows |
| Pruning | On each evidence write |
demand_history holds one object for each history (unsized, arrivals, expected_decode, observed_decode, and residency), each with these fields:
| Field | Meaning | Metric, labelled by window |
|---|---|---|
bucket_s |
Bucket width | |
retained_s |
Retention span | |
cells |
Retained cells | narwhal_demand_history_cells |
cell_limit |
Cell limit | narwhal_demand_history_cell_limit |
observations |
Counted observations | narwhal_demand_history_observations |
overflow_observations |
Observations in overflow cohorts | narwhal_demand_history_overflow_observations |
Overflow¶
An overflow cohort:
- keeps every request count
- takes its price from its largest input length and requested output length
- keeps the output estimates learned before the overflow until every overflow cohort leaves output history
- contributes its full count when it crosses a window boundary
Prefill recovery ratio¶
When queued_prefill_s > 0 and at least one engine has the prefill role, the prefill recovery ratio is:
recovery_prefill_ratio = max(
observed prefill pressure,
(resident prefill seconds + queued prefill seconds)
/ (current prefill engine count * TTFT SLO seconds)
)
Otherwise, recovery_prefill_ratio equals the observed prefill pressure.
Decisions made with incomplete demand report both observed phase ratios, recovery_prefill_ratio, and queued_prefill_s.
In those decisions, decision_basis is one of:
Source-pressure and movement-gate checks use demand at full precision. State and journal records round it.
Role floors, cooldown, dwell, KV limits, and the movement and confirmation gates constrain each move.
Decode recovery ratio¶
recovery_decode_ratio equals the observed decode pressure. Role control prices decode capacity from engine profiles without the router's engine seats.
Decisions held before candidate scoring, while they wait for demand history or fleet profiles, report recovery_decode_ratio.
Consolidation evidence¶
demand_evidence fields:
| Field | Meaning |
|---|---|
span_s |
Time span of the counted arrivals |
arrivals |
Arrivals since the later of the latest risk event and max_span_s seconds ago |
required_span_s |
Span that closes the window, controller.reactive.evidence_span_s |
required_arrivals |
Arrival count that closes the window, controller.reactive.evidence_min_arrivals |
max_span_s |
Longest lookback, controller.reactive.evidence_max_span_s |
closed |
true when the window meets required_span_s and required_arrivals, or spans max_span_s |
risk_kind |
Newest risk event: first_token_timeout or p_to_d_recovery |
risk_age_s |
Seconds since the newest risk event |
risk_events |
Risk-event counts by kind |
short_decode_engines |
Decode demand over required_span_s, in engines |
long_decode_engines |
Decode demand from the latest full-window estimate, in engines |
trend_ratio |
short_decode_engines divided by long_decode_engines |
envelope_decode_engines |
Larger of short_decode_engines and long_decode_engines |
blocked_gate |
none, risk, evidence, or trend |