Serving and role control¶
Request admission and bounded serving¶
Global admission¶
| Field | Default | Meaning | Values |
|---|---|---|---|
serving.admission |
"predictive" |
Admission mode. | predictive or open |
serving.admission_margin |
0.0 |
Fraction of the TTFT target added to the admission budget. | Zero or greater |
serving.max_connections |
768 |
Router in-flight limit, and the data connections the router opens to each engine. | At least 1 |
engine.control_connections |
0 |
Size of the control connection pool that all engines share, reserved for health and recovery probes. | 0 for max(4, 2 × engine count), or a positive size |
In predictive mode, the router returns HTTP 429 when a request fails the time to first token (TTFT) check or the decode admission check. open mode disables predictive refusals.
Admission and refusal semantics lists the error and Retry-After value for each predictive refusal.
Before raising serving.max_connections, measure the in-flight load the fleet sustains while healthy. Choosing admission, queue and retry settings gives the measurement procedure.
In-flight limit¶
The in-flight limit is narwhal-serve --max-concurrent when set, otherwise serving.max_connections. At most that many requests hold an admission seat at once. With serving.queue_capacity above 0, up to that many more requests wait for a seat.
The router counts a completion request against the limit from its arrival, before it reads the body, until its response ends. When the requests it counts reach the in-flight limit plus serving.queue_capacity, the router answers each new request with HTTP 429, error type server_overloaded_error and message router in-flight limit reached. The router counts the request as an unsized offer with rejection reason inflight_limit.
/metrics reports the admission seats in use as narwhal_admission_inflight, the limit as narwhal_admission_inflight_limit, and the counted requests as narwhal_http_retained against narwhal_http_retained_limit.
Router saturation¶
The router also answers HTTP 429 with rejection reason saturated when either signal reaches a quarter of slo.ttft_s:
- event-loop lag, the lateness of a 50 ms wake-up probe
- the median time to count tokens and hash prefixes for at least 8 requests sized in the last 2 seconds
/metrics reports the signals as narwhal_router_loop_lag_seconds and narwhal_request_sizing_delay_seconds, and their threshold as narwhal_saturation_threshold_seconds.
Decode admission check¶
The decode admission check projects when each request holds a slot in the decode pool.
The check admits the request outright when the fleet has zero live decode engines, or when any live decode engine's profile or profiled decode_max_requests is unset.
Each request holds one slot. The pool's slot count is the sum of each live decode engine's decode seats.
| Request | Reaches decode | Holds a slot |
|---|---|---|
| Request resident in decode | From now until its projected last token | |
| Request waiting for a decode slot | Now | From the earliest free slot until its projected last token |
| Request in prefill, or the checked request | At its predicted prefill completion | From the earliest free slot after it reaches decode until its projected last token |
| Term | Meaning |
|---|---|
| Slot wait | Time from the checked request's predicted prefill completion to the start of its slot |
| Projected TTFT | Time since arrival plus the prefill placement price, including queueing |
| Request KV | Key-value (KV) tokens for the prompt plus delivered and remaining output |
| Output cap | max_completion_tokens, or max_tokens when max_completion_tokens is unset |
| Bucket | The request's prompt length and output cap, each rounded up to a power of two |
The check expects this much output from each request:
| Request | Expected output |
|---|---|
| Capped, with three or more finished requests in its bucket | Output cap times the bucket's median delivered fraction |
| Capped, with fewer than three finished requests in its bucket | Output cap |
| Uncapped | Median delivered output for its prompt bucket, or the fleet-wide median delivered output |
Remaining output depends on how much output the request has delivered:
| Delivered output | Remaining output |
|---|---|
| Below the expected output | Expected output minus delivered output |
| At or past the expected output | Output cap minus delivered output, or unknown for an uncapped request |
When a request's remaining output is unknown:
- the checked request holds its slot for the instant the slot starts
- every other request holds its slot indefinitely
The check projects generation at these token intervals:
| Request | Token interval |
|---|---|
| Request resident in decode | Its decode engine's interval |
| Every other request | Fleet mean of the engine intervals |
An engine's interval is its profiled token interval times the decode correction.
The check admits the request when all three of these pass:
| Check | Passes when |
|---|---|
| Slot wait | The slot wait is zero, or the projected TTFT plus the slot wait fits the TTFT budget |
| KV tokens | Peak request KV during the checked request's hold fits the sum of the live decode engines' decode KV token bounds, or the request is alone in decode during its hold |
| TPOT | At the start of the request's slot, a live decode engine meets slo.tpot_s with the request and the residents still generating on that engine, or the request misses slo.tpot_s on every idle decode engine |
Retry pricing¶
In predictive mode, an attempt after the first skips the TTFT check and runs the decode admission check, where the slot-wait check uses the retry's predicted prefill completion in place of its projected TTFT. A retry that fails the decode admission check receives HTTP 429 with that check's cause. Its journal attempt entry carries retry_reason not_dispatched.
Waiting, engine seats, and retries¶
| Field | Default | Meaning | Values |
|---|---|---|---|
serving.queue_capacity |
0 |
Maximum requests waiting for admission. | 0 rejects immediately at saturation |
serving.queue_timeout_s |
0.0 |
Most time one request waits for an admission seat and prefill seats combined, capped by the original request deadline. | Positive when serving.queue_capacity is positive |
serving.max_attempts |
2 |
Maximum complete prefill and decode attempts per original request. | 1 to 3 |
serving.retry_base_s |
0.1 |
Initial exponential-backoff ceiling. | Positive |
serving.retry_cap_s |
1.0 |
Maximum backoff ceiling. | At least serving.retry_base_s |
serving.retry_budget |
10 |
Starting and maximum size of the retry-credit pool, with each retry spending one credit when it dispatches. | Zero or greater |
serving.retry_replenish |
0.1 |
Credits added after each successful original request. | 0 to 1 |
serving.max_request_bytes |
4194304 |
Maximum HTTP request-body size. | At least 1 |
serving.max_response_bytes |
16777216 |
Maximum bytes retained for each non-streaming attempt, and for the metadata a stream buffers before its first output. | At least 1 |
Queue waits¶
With serving.queue_capacity above 0, a request can wait at three stages. Each wait has its own bound:
| Wait | The request waits while | The wait ends at | Response when the bound ends the wait |
|---|---|---|---|
| Admission seat | Requests hold every seat under the in-flight limit | serving.queue_timeout_s |
HTTP 504, error type queue_expired |
| Prefill seat | Every engine that can take the prefill leg has its prefill seats full | The part of serving.queue_timeout_s that the admission-seat wait and earlier prefill-seat waits of the request left |
HTTP 504, error type expired |
| Decode seat | Every engine that can take the decode leg has its decode seats full | The producer's KV handoff bound | HTTP 504, error type handoff_expired |
The original request deadline also ends each wait, with HTTP 504 and error type request_expired. Under event-loop lag, reported as narwhal_router_loop_lag_seconds, a wait can run past its bound by about the lag, and a seat that frees in that time still serves the request. The journal records the bound that ended a wait as the expiry reason: queue_timeout, handoff or deadline. A decode-seat wait that ends at the handoff bound with attempts left retries the request.
A request with no queue budget left still takes a free prefill seat, but it does not wait for one.
Waiting requests keep their arrival order. When a seat frees, the release gives it to the waiting requests in that order. The release stops at the first waiting request that cannot take a free seat.
In predictive mode, the router prices an original request's first attempt throughout an admission-seat or prefill-seat wait. The price is the projected TTFT on the cheapest live prefill engine, which is that engine's placement price plus the time since arrival. When the price exceeds the budget, the request leaves the queue with the HTTP 429 of the TTFT check. The router does not price a retry's prefill-seat wait, and open mode never refuses a waiting request.
The router prices an admission-seat wait before it sizes the request. That price uses the local length estimate from engine.chars_per_token, or the array length of a token-ID prompt, and the cold prefill curve without prefix-cache pricing. A waiting request whose prompt is mostly cached can therefore leave the queue with a queue or prompt refusal that its sized, cache-aware price would pass. The router sizes the request when it takes an admission seat, so later prefill-seat waits use the sized price.
A hold ends waiting requests with the HTTP 503 of a router that is not ready: error type standby, Retry-After: 1, and journal reason not_ready.
| Hold begins | Requests that end with the 503 | Requests that continue |
|---|---|---|
| Whole-wave hold | Requests waiting for an admission, prefill or decode seat, and prefilled requests before decode dispatch | Requests with a dispatched decode leg |
| Degraded engine monitoring | Requests waiting for an admission or prefill seat | Prefilled requests, which wait for a decode seat and decode |
| Lost or failed router lease | Requests waiting for an admission or prefill seat | Prefilled requests, which wait for a decode seat and decode |
A request in its prefill leg when a whole-wave hold begins receives the 503 when its prefill completes. The router also returns the 503 to a request that reaches prefill placement during a whole-wave hold. The same applies while the router is standby, fenced or in degraded engine monitoring.
Engine seats¶
The router derives each engine's prefill and decode seats from engine evidence:
| Seats | Value |
|---|---|
| Decode | The lower of the engine's attested sequence limit and its profiled decode_max_requests. |
| Prefill | The number of back-to-back profiled prefills at the mean input length that complete within slo.ttft_s, at least 1. |
The attested sequence limit is the engine's runtime.max_sequences, read from the launch arguments in its verified attestation: --max-num-seqs on vLLM or --max-running-requests on SGLang. The router reads it when it verifies the engine's identity at startup, takeover, and readmission. Without runtime.max_sequences, a vLLM engine takes its decode seats from the profile alone, and an SGLang engine has a limit of 256.
The mean input length covers requests the router sized within the last controller.reactive.window_s. An engine has no prefill seat limit until a request is sized. An engine outside the profile store has no prefill seat limit, and its decode seats come from the attested sequence limit alone.
With serving.queue_capacity above 0, a request waits for an engine with a free seat in its phase. With the queue off, the router places a prefill leg without checking seats, and an engine queues prefill legs above its seats.
With the queue off, the router places a decode leg on an engine with a free decode seat. When every decode engine is full, the router places the leg on the engine whose seat the profiles project to free first.
/narwhal/state reports each engine's seats under seats, and /metrics exports them as narwhal_engine_seats.
KV handoff bound¶
After prefill, a vLLM producer engine holds the request's KV blocks for its NIXL lease, kv_lease_duration. The lease starts when prefill completes.
The router reads each producer's lease from the --kv-transfer-config launch argument in the engine's verified attestation. It derives the producer's handoff bound:
At the launcher's default lease of 30 seconds, the bound is 25 seconds. runtime.kv_lease_s sets the lease.
The router counts handoff age from the moment it receives the producer's prefill response. When the age reaches the bound before decode dispatch, the attempt ends with a handoff expiry. With serving.queue_capacity above 0, a wait for a decode seat also ends at the bound.
When an engine's attestation records no kv_lease_duration, the router applies no handoff bound to that producer, and the original request deadline bounds its handoffs. /narwhal/state reports null for that engine's handoff fields.
SGLang engines have no lease and no handoff bound. The router caps a decode leg's wait for its first token at 30 seconds, the prefill engine's bootstrap timeout.
When serving.max_attempts is greater than 1, the router retries an attempt that fails before visible output with any of these:
- a transient transport error
- an HTTP 408, 429, 500, 502, 503 or 504 response
- an expired KV handoff
Any of these ends the request:
- a permanent error
- starvation of the local data connection pool
- cancellation
- any failure after visible output
A retry avoids each engine whose leg failed earlier in the same request. When no engine remains for a role, the request ends with HTTP 503. Retries gives the placement rules, the retry credit and the exact-count failures that move between engines.
The original request deadline covers tokenization, queue wait, retry backoff, engine work, and client writes.
Tune queueing and retries:
- Read Choosing admission, queue and retry settings for the client outcomes of each value and when to change it.
- Set queue capacity and deadlines from the workload's measured latency and capacity.
- Verify that the backend releases abandoned KV handoffs when the lease expires.
- After any change to queueing, engine launch limits, profiles, KV handoff expiry, retries, or byte limits, measure again.
Streaming failure semantics¶
A streaming response holds its HTTP status and headers until the first output frame.
Engine failure handling gives the client response for each prefill and decode failure.
Request deadline expiry ends a streaming response this way:
| Deadline expires | Result |
|---|---|
| Before the first output frame | The client receives HTTP 504 |
| After the first output frame | The client receives a terminal stream error event with code: expired |
| While client writes are blocked | The router drops the connection immediately |
Treat an error event, or a stream that ends before the success terminator, as a failed response.
Fit client-side retries inside the caller's remaining deadline.
Placement¶
Placement selects the lowest-cost engine that passes the role, availability, exclusion, and projected service-level objective (SLO) filters. Ties go to the lowest instance ID.
When every candidate violates its projected SLO, placement selects the lowest-cost candidate as an unserved placement.
A decode leg avoids its producer, the engine that ran its prefill. Placement puts the decode leg on the producer only when no other engine in the decode pool can take it. The rule applies when a role change moves a producer into the decode pool, including while its requests wait for a decode seat.
A live role change:
- applies to new placements immediately
- leaves resident requests on their current engine and reservation until completion or cancellation
- keeps lifecycle drains, quarantine, ejections, and restart holds in place
Prefix-cache pricing¶
For a request sized from exact token IDs, the router records the leading prompt blocks each engine has cached, from that engine's residency view.
When an engine holds a cached prefix, the router prices prefill on that engine with its warm prefill fit.
These decisions use the warm price:
- which engines meet the TTFT budget, and their placement order
- predictive admission
- resident work
- pool load
- offered demand
- role-split scoring
Offered demand takes warm prices from the engines that run prefill:
| Offered demand in | Warm prices from |
|---|---|
| Demand estimate | Live prefill-role engines |
| Role-split scoring | Each candidate split's prefill engines |
The router prices a request on the cold curve of its full input when any of these holds:
- the router used its local length estimate
- the request carries multimodal content
- the request sets
truncate_prompt_tokens,documents, orreasoning_effort - the fleet config leaves
engine_contractunset - the fleet's
engine_contractenables speculative decoding - the engine's residency is unknown
- the engine's current view drops the prefix before placement
- the engine's profile holds only a cold fit
- the case lies outside the warm fit's measured domain
The request journal records each placement priced with cache evidence in cache_placement.
Request deadlines and engine HTTP behaviour¶
| Field | Default | Meaning | Values |
|---|---|---|---|
serving.request_timeout_s |
600.0 |
End-to-end completion deadline from HTTP ingress through response delivery. | Positive |
serving.prefill_timeout_s |
120.0 |
Elapsed-time deadline for the prefill leg. | Positive, at most serving.request_timeout_s |
engine.first_token_timeout_s |
2.5 |
Deadline to the first decode token. | Positive, at most serving.request_timeout_s |
engine.first_token_calibration_path |
"" |
Path to a completed first-token calibration artifact under runs/. |
|
engine.decode_read_timeout_s |
10.0 |
Maximum silent interval between decode chunks after the first token. | 0 disables the gap limit |
engine.tokenize |
true |
Requests exact token counts for text and chat input from the dialect tokenization endpoint. | |
engine.tokenize_timeout_s |
2.0 |
Elapsed-time deadline for an exact token count. | Positive |
engine.chars_per_token |
3.8 |
Characters per token for the fallback estimate. | Positive |
engine.connect_timeout_s |
10.0 |
TCP connect deadline for engine requests. | Positive |
engine.pool_timeout_s |
5.0 |
Maximum wait for a connection from the data or control connection pool. | Positive |
engine.health_timeout_s |
5.0 |
HTTP I/O timeout for preflight, breaker, readmission, and residency requests. | Positive |
Request and prefill deadlines¶
Set the deadlines from these inputs:
| Field | Set from |
|---|---|
serving.request_timeout_s |
Supported output length and the client deadline |
serving.prefill_timeout_s |
Measurements of the longest admitted inputs under the supported load |
For diagnostic profiling, narwhal-profile --observation-timeout-s sets the probe HTTP timeout.
A prefill, tokenization, or health call ends at the first of its phase, connection, or pool timeouts to expire.
First-token deadline and calibration¶
The engine.first_token_timeout_s window runs from before the decode HTTP stream opens to the first generated token.
The budget for each inference probe leg, prefill and decode, is the larger of engine.first_token_timeout_s and engine.health_timeout_s.
Configure the first-token deadline:
- Run first-token deadline calibration.
- Set
engine.first_token_timeout_sabove the candidate it prints. - Set
engine.first_token_calibration_pathto its artifact.
The calibration artifact binds each engine to the same process generation as its saved profiles.
Preflight and router startup respond to these conditions:
| Condition | Artifact | Preflight and router startup |
|---|---|---|
engine.first_token_calibration_path is empty |
Log a warning | |
| An engine relaunched with the same process generation | Label the engine reused |
|
| An engine's process generation changed | Stale | Fail |
| An engine relaunched during calibration | Insufficient | Fail |
Decode stream gaps¶
Set engine.decode_read_timeout_s from measured inter-chunk gaps and the service's failure budget. Decode timeouts gives the client outcome of a positive limit and of 0.
Disable the gap limit:
Token counting¶
The router counts a request's input tokens this way:
| Input | Token count |
|---|---|
| Completion prompt that is a nonempty, flat list of nonnegative integer token IDs | Local array length |
Text or chat input, with engine.tokenize set to true and a dialect exact-count route |
Exact count from that route, within engine.tokenize_timeout_s |
| Other text or chat input | Estimate from engine.chars_per_token |
When the exact-count call fails, the client receives the engine error before placement.
Measure engine.chars_per_token for the served tokenizer and for every dialect that uses the estimate.
Connection, pool, and health timeouts¶
Set these timeouts from latency measured under the intended load:
| Field | Measured latency |
|---|---|
engine.connect_timeout_s |
Connection setup |
engine.pool_timeout_s |
Pool waits |
engine.health_timeout_s |
Health and identity requests |
These timeouts decide a probe's outcome:
| Probe | Condition | Outcome |
|---|---|---|
| Health or inference probe | Waits longer than engine.pool_timeout_s for a control connection |
Inconclusive |
| Health probe | Times out within 1.5 times engine.health_timeout_s of getting its control connection |
Failed |
| Health probe | Times out later than that | Inconclusive |
An inconclusive probe has this effect for each caller:
| Caller | Inconclusive probe |
|---|---|
| Breaker verification | Engine keeps its current health verdict |
| Liveness probe | Engine keeps its current health verdict |
| Preflight reach | Failed /health check |
| Readmission probe | Engine stays ejected |
| Readmission validation | Failed /health check |
Role control¶
| Field | Default | Meaning | Values |
|---|---|---|---|
controller.advisory |
false |
Holds current roles and records proposed role splits with reasons. | |
controller.monitor_interval_s |
1.0 |
Delay between engine monitoring passes and between residency refreshes. | Positive |
controller.monitor_failure_limit |
5 |
Consecutive engine monitoring passes with a stage failure before degraded state stops new admission. | At least 1 |
controller.min_prefill |
1 |
Minimum live prefill engines that role-controller moves preserve. | At least 1 |
controller.min_decode |
1 |
Minimum live decode engines that role-controller moves preserve. | At least 1 |
controller.thresholds.expand |
1.0 |
SLO-relative pool load that starts reactive expansion. | Positive |
controller.thresholds.shrink |
0.5 |
Maximum projected source load for ordinary consolidation. | Zero or greater, lower than expand |
controller.thresholds.cooldown_s |
10.0 |
Minimum time between prefill-to-decode moves. | Zero or greater |
controller.thresholds.sustained_intervals |
3 |
Confirmations required for moves that need them. | At least 1 |
controller.thresholds.dwell_s |
0.0 |
Minimum time an engine stays in a new role. | Zero or greater |
controller.thresholds.panic_ratio |
0.0 |
Multiple of expand for the prefill-to-decode cooldown bypass. |
0 for off, or at least 1 |
controller.thresholds.flip_resident_guard |
0 |
Maximum resident decode streams allowed on a decode-to-prefill donor. | 0 disables the guard |
controller.flip_history |
1000 |
Maximum role-change records the router retains and exposes in /narwhal/state. |
At least 1 |
Load definitions¶
Prefill load is:
Decode load is:
| Term | Meaning |
|---|---|
| Observed token interval | The engine's average time between tokens. If no tokens completed during the monitoring interval, it uses the time since the last token on an active stream. |
| Corrected idle floor | The profile's zero-contention decode interval times the decode correction |
| Remaining time per output token (TPOT) budget | TPOT target minus the corrected idle floor |
| Decode correction | The engine's ratio of live to profiled decode latency, bounded by controller.reactive.decode_correction_min and controller.reactive.decode_correction_max |
When the corrected idle floor reaches the TPOT target, decode load is the raw interval-to-target ratio.
A load of 1.0 means the phase has reached its target.
Controller decision details call load "pressure", as in the mixed_pressure eligibility rule and the source_pressure_safe flag.
Role floors¶
In a fleet of two or more engines, controller.min_prefill plus controller.min_decode is at most the engine count. A one-engine fleet has both floors at 1.
When pins, drains, quarantine, or health ejections leave too few movable engines for a floor, Narwhal reports a floor breach.
Every role change preserves controller.min_decode.
When live decode engines drop below controller.min_decode, engine monitoring moves one eligible engine to decode per pass.
Floor recoveries skip some move timers:
| Recovery | Cooldown | Per-engine dwell |
|---|---|---|
| Decode floor | Skipped | Applied |
| Prefill floor | Skipped | Skipped |
Both floor recoveries:
- honor pins, availability, floor limits, and advisory mode
- add each applied move to the role-change records that
controller.flip_historycaps
Resident work during role changes¶
A decode-to-prefill move requires:
- the new split and resident decode batches inside the profile's decode domain
- KV capacity for the resident work
Advisory rollout¶
Before allowing role changes in production, run advisory mode against recorded traffic.
Enable advisory mode:
/narwhal/state and Prometheus expose each proposal's prefill and decode counts, caller, reason, and result.
Adjacent-split decisions¶
An adjacent split differs from the current split by one engine moved between prefill and decode.
The worst projected SLO ratio of a split is the largest of its projected TTFT, TPOT, and decode queueing ratios.
| Move | Projected source load |
|---|---|
| Prefill-to-decode | The candidate's projected TTFT ratio |
| Decode-to-prefill | The larger of the candidate's TPOT and decode queueing ratios |
The decode queueing ratio projects the slots of the decode admission check onto the split's decode engines:
max over waiting decode requests and requests in prefill:
slot wait / (slo.ttft_s - time since arrival - time until the request reaches decode)
The split's slot count is its decode engine count times the live mean slot count per decode engine.
A request at or past its TTFT deadline when it reaches decode contributes zero.
The ratio is at least the measured handoff delay times the current decode engine count, divided by the split's decode engine count and by slo.ttft_s.
Adjacent-split decisions use these spans:
| Span | Length | Default |
|---|---|---|
| Window | controller.reactive.window_s |
120 s |
| Settled run | controller.reactive.evidence_span_s |
60 s |
| Confirmation span | controller.reactive.step_s times max(controller.reactive.confirmations, controller.thresholds.sustained_intervals) |
15 s |
| Reversal lookback | controller.reactive.window_s plus controller.reactive.evidence_span_s |
180 s |
Each scored decision records in eligibility_rule the first of these rules that matches:
projected_ttft_recoverymixed_pressuresettled_departuresteady_demandsource_shrink
Ordinary consolidation, the source_shrink rule, requires both:
- projected source load at or below
controller.thresholds.shrink - reduction in the worst projected SLO ratio of at least
controller.reactive.movement_margin
If prefill demand spikes, engines can move from decode to prefill under source_shrink even if decode load is higher than usual (up to controller.thresholds.expand). This requires:
- projected prefill load exceeds
controller.thresholds.expandduring the window or confirmation span - no engine is currently moving to prefill
- no first-token timeouts have occurred within the arrival-evidence window
Triggering this move cancels any pending moves to decode.
The mixed_pressure rule moves one engine when all of these hold:
- projected source load is above
controller.thresholds.shrink - for a move to prefill, the prefill recovery ratio is at or above
controller.thresholds.expand - for a move to decode, the current split's projected decode load is at or above
controller.thresholds.expandand no departure is open - every engine has a profile
- the move improves the worst projected SLO ratio by more than zero and by at least
controller.reactive.movement_margin
The settled run is the time the current split has held without a better adjacent split. It restarts when any of these happens:
- the role controller attempts a move
- an evaluation stops before scoring splits
- for
controller.reactive.step_s, an adjacent split improves the worst projected SLO ratio by at leastcontroller.reactive.movement_marginon window or confirmation-span demand
A departure reverses when all of these hold as it opens:
- the role controller applied at least one adjacent-split move within the preceding reversal lookback
- each of those moves went opposite the departure's direction
- the latest of those moves is at least
controller.reactive.evidence_span_sold
Departures need these settled runs:
| Departure | Settled-run length | Default |
|---|---|---|
| Reversing | controller.reactive.evidence_span_s minus the confirmation span |
45 s |
| Other | controller.reactive.evidence_span_s |
60 s |
The role controller opens a departure when all of these hold:
- the current settled run is at least the departure's settled-run length
- confirmation-span demand for either phase differs from window demand by more than
controller.reactive.demand_rise_tolerancetimes the larger estimate - on confirmation-span demand, an adjacent split improves the worst projected SLO ratio by at least
controller.reactive.movement_margin
The settled_departure rule moves one engine toward that split when both hold on confirmation-span demand:
- projected source load at or below
controller.thresholds.shrink - reduction in the worst projected SLO ratio of at least
controller.reactive.movement_margin
After its first move, a departure leads until either of these has held for controller.reactive.step_s:
- on confirmation-span demand, the best adjacent split lies opposite the departure's direction and improves the worst projected SLO ratio by at least
controller.reactive.movement_margin - confirmation-span demand for each phase is within
controller.reactive.demand_rise_tolerancetimes the larger estimate of window demand
The departure's later moves use this demand and rule:
| Departure | Later move | Demand | Rule |
|---|---|---|---|
| Reversing | From the first move until the lead ends | Confirmation-span demand | source_shrink |
| Other | From controller.reactive.evidence_span_s after the departure opens until the lead ends |
Confirmation-span demand | source_shrink |
| Any | Every other later move | Window demand | First matching rule |
When mixed_pressure applies to the departure's split on window demand, that split keeps window demand and the mixed_pressure rule.
A departure closes at the first of these:
controller.reactive.window_spasses after the departure opens- before the first move, the best adjacent split on confirmation-span demand lies in the other direction
- before the first move, the best adjacent split on confirmation-span demand improves the worst projected SLO ratio by less than
controller.reactive.movement_margin - after the first move, the role controller applies a window-demand move in the departure's direction
- the role controller applies a prefill surge move
After its first move, a departure blocks moves in the opposite direction until it closes, except for prefill surge moves.
Demand is steady when both hold:
- for the last
controller.reactive.evidence_span_s, confirmation-span demand and window demand for each phase differ by at mostcontroller.reactive.demand_rise_tolerancetimes the larger estimate - the arrival-evidence window is closed
The steady_demand rule moves one engine under steady demand when both hold:
- projected source load above
controller.thresholds.shrinkand at or belowcontroller.thresholds.expand - reduction in the worst projected SLO ratio of at least
controller.reactive.movement_margin
Projected-TTFT recovery evaluations leave the settled run, steady_demand_s, and an open departure unchanged.
A standby router or a lifecycle hold pauses role control and restarts the settled run and steady_demand_s.
One confirmation is enough in three cases:
- the move comes from a projected-TTFT recovery evaluation
- the eligibility rule is
settled_departure - all of these hold:
- demand is complete
- the eligibility rule is
source_shrink - the projected source load is at or below
controller.thresholds.shrink - the destination phase's current SLO ratio meets or exceeds
controller.thresholds.expand
Every other move needs this many confirmations:
Any of these resets the confirmation sequence:
- a change in the proposed split
- a change in demand completeness
- a change in the eligibility rule
- a failed eligibility check
Prefill-to-decode moves require:
- projected prefill load at or below
controller.thresholds.shrink, or at or belowcontroller.thresholds.expandundersteady_demand, on both window demand and confirmation-span demand - an elapsed
controller.thresholds.cooldown_s, or an armed cooldown bypass
The cooldown bypass arms after controller.thresholds.sustained_intervals consecutive passes with both:
- decode load at or above
controller.thresholds.panic_ratiotimescontroller.thresholds.expand - prefill load at or below
controller.thresholds.shrink
Evidence gating for decode-to-prefill consolidation¶
Decode-to-prefill consolidation waits for the arrival-evidence window to close. The window closes when either of these holds:
controller.reactive.evidence_span_shas elapsed with at leastcontroller.reactive.evidence_min_arrivalssamplescontroller.reactive.evidence_max_span_shas elapsed under sparse traffic
With enough samples in the short horizon of controller.reactive.evidence_span_s, consolidation pauses when:
A first-token timeout or a prefill-to-decode recovery move resets the arrival-evidence window.
If projected prefill load exceeds controller.thresholds.shrink, moving an engine from decode to prefill bypasses the short-horizon pause and ignores arrival-evidence resets caused by prefill-to-decode moves. It is still blocked if the window was reset by a first-token timeout. When bypassed, evidence_blocked_gate reports which check was skipped.
The source_shrink and settled_departure rules move engines toward decode while the window is open.
The steady_demand rule requires a closed arrival-evidence window for moves in either direction.
Live state and metrics expose the short and long demand estimates, the arrival-evidence window state, and the gate blocking movement.
Reactive-controller parameters¶
| Field | Default | Meaning | Values |
|---|---|---|---|
controller.reactive.window_s |
120.0 |
Demand-estimation window, the maximum time a departure stays open, and part of the reversal lookback. | Positive |
controller.reactive.confirmations |
2 |
Required consecutive identical adjacent proposals for moves that need confirmation, and a factor of the confirmation span. | At least 1 |
controller.reactive.utilization |
0.8 |
Fraction of each engine treated as usable capacity. | Greater than 0, at most 1 |
controller.reactive.min_arrivals |
10 |
Minimum arrival count required for a decision. | At least 1 |
controller.reactive.demand_floor |
0.5 |
Minimum accepted demand signal. | Positive |
controller.reactive.movement_margin |
0.05 |
Required reduction in worst projected SLO ratio before movement. | [0, 1) |
controller.reactive.step_s |
5.0 |
Minimum interval between scheduled adjacent-split evaluations and the unit of the confirmation span. | Positive |
controller.reactive.evidence_span_s |
60.0 |
Minimum recent-arrival span for decode-to-prefill consolidation, steady-demand duration, settled-run length, part of the reversal lookback, the minimum age of the latest move a reversing departure reverses, and the departure age from which other leading departures price later moves on confirmation-span demand. | Positive, at most evidence_max_span_s |
controller.reactive.evidence_max_span_s |
120.0 |
Maximum evidence duration under sparse traffic. | Positive, at least evidence_span_s, at most window_s |
controller.reactive.evidence_min_arrivals |
10 |
Minimum samples within the evidence span before decode-to-prefill consolidation. | At least 1 |
controller.reactive.demand_rise_tolerance |
0.25 |
Maximum accepted short-horizon rise over long-horizon decode demand, and the fraction of the larger estimate within which two demand spans match. | Zero or greater |
controller.reactive.decode_correction_min |
0.5 |
Lower bound on the live-to-profile decode correction. | Positive |
controller.reactive.decode_correction_max |
2.0 |
Upper bound on the live-to-profile decode correction. | At least decode_correction_min |
controller.reactive.decode_correction_alpha |
0.2 |
Fraction of each qualifying observation window applied to the correction. | (0, 1] |
controller.reactive.decode_correction_min_samples |
8 |
Required decode gaps before a window updates the correction. | At least 1 |