Backend execution and failures¶
Disaggregated backend execution¶
Prefill¶
The prefill leg is a non-streaming, one-token completion request to the producer.
Decode¶
Remote decode receives:
- the original prompt or messages
- the requested output limit
- sampling settings
- the validated handoff descriptor
Same-engine decode:
- strips
kv_transfer_paramsfrom the request - uses engine prefix caching or prompt recomputation
Descriptor validation¶
Run preflight to validate the pinned engine contract and role-permitted KV transfers.
Decode rejects a KV handoff descriptor with:
- missing engine identity
- malformed block IDs
- a connector mismatch
- an endpoint mismatch
Request scope and timing¶
Scoped to the original client request:
- handoff-age enforcement
- phase reservations
- retries
- cleanup
KV handoff expiry¶
The KV handoff bound limits the time from prefill completion to decode dispatch. A handoff that reaches the bound ends its attempt:
| Point of expiry | Journal error contains |
|---|---|
| Before decode dispatch | KV handoff expired before decode dispatch |
While waiting for a decode seat, with serving.queue_capacity above 0 |
KV handoff bound reached while waiting for a decode seat |
The expiry is transient, so a request with attempts remaining retries with a fresh prefill. When the request has no attempts left, the router returns HTTP 504 with error type and code handoff_expired and the message KV handoff expired before decode dispatch. The journal records terminal expired with reason handoff.
Durations in the measurement contract:
| Duration | Interval |
|---|---|
ttft_s |
Request arrival at the router to producer HTTP completion |
first_byte_s |
Request arrival at the router to the first generated decode output |
first_byte_s - ttft_s |
Producer completion to the first decode output |
Python API¶
| Object | Contract |
|---|---|
EngineClient.prefill() |
Returns a PrefillResult to pass to EngineClient.decode() |
PrefillResult |
Holds the KV handoff descriptor, producer URL, endpoint, and backend request ID |
result.parameters() |
Returns a detached dictionary |
Engine failure handling¶
A timeout-shaped engine fault maps to HTTP 504, and every other engine fault maps to HTTP 502. A timeout-shaped fault is an engine HTTP 408 or 504 status, or a connect, connection-pool, prefill, first-token, between-token, or exact-count timeout.
Exact-count failures, prefill failures, non-streaming decode failures, and streaming decode failures before the first output return that HTTP error status when the request does not retry. The error type and code both name the request phase in which the fault ended the request:
| Condition | HTTP | Error type |
Error code |
|---|---|---|---|
| The last exact input count fails, or a count fails non-transiently, with a timeout-shaped fault | 504 |
admission |
admission |
| The last exact input count fails, or a count fails non-transiently, with any other fault | 502 |
admission |
admission |
| The prefill leg ends with a timeout-shaped fault and the request does not retry | 504 |
prefill |
prefill |
| The prefill leg ends with any other fault and the request does not retry | 502 |
prefill |
prefill |
| The decode leg ends with a timeout-shaped fault before output starts and the request does not retry | 504 |
decode |
decode |
The decode leg ends with any other fault before output starts and the request does not retry, or a non-streaming response fails assembly or exceeds serving.max_response_bytes |
502 |
decode |
decode |
The client error body carries a generic message, such as Upstream request failed. The engine failure detail goes to the error field of the terminal request record.
A streaming decode failure after the first output ends the HTTP 200 stream with a terminal server-sent event (SSE). Its error type is the request phase, decode, and its code is the terminal state, failed or expired:
An unexpected router error returns HTTP 500 with a plain-text body and no error type. The journal records terminal failed with reason internal.
Input sizing¶
The router sizes each request's input before placement:
| Input | Sizing before placement |
|---|---|
Nonempty prompt array of nonnegative integer token IDs |
Local ID count |
Text or chat, with engine.tokenize on and an exact-count endpoint in the dialect |
Exact count from an engine |
| Other input | Character ratio |
A failed exact count puts that engine into count backoff for 1 second. Each further consecutive failure doubles the backoff, up to 30 seconds, and a successful count resets it. The failure also counts as failure evidence against that engine.
After a transient failure, the router moves the exact count to another live engine that has not failed it for this request. A request tries at most serving.max_attempts engines for its exact count, and these tries spend no retry credit. A non-transient failure, such as an HTTP 400 for the prompt, ends the count on its first engine.
When the last exact count fails, or a count fails non-transiently, the request returns the engine-fault mapping status before placement.
Breaker ejection and readmission¶
When an engine's consecutive failures in one breaker class reach recovery.eject_after, the router ejects the engine or runs a probe that decides ejection or readmission.
Last-engine protection keeps an uncovered suspect in placement during its inference probe.
Successful stream termination¶
A valid engine stream ends with data: [DONE] after generated output.
| Stream | Result | Journal error contains |
|---|---|---|
Closes before [DONE] |
Engine failure | stream ended before the [DONE] terminator |
[DONE] before the first generated token |
HTTP 502 |
stream ended with [DONE] before any token arrived |
Upstream HTTP 200 carrying an error object with an integer code from 400 to 599 |
Engine failure with status code |
Engine error message |
Upstream HTTP 200 carrying any other error object |
Engine failure with status 500 |
Engine error message |
Decode timeouts¶
engine.first_token_timeout_s bounds the time from decode request start, across connection and response headers, to the first generated token. On expiry, the router returns HTTP 504 and the journal error contains no first token within.
engine.decode_read_timeout_s bounds the silence between any two transport chunks after the first token. On expiry, the router returns HTTP 504 and the journal error contains engine went silent between tokens. With engine.decode_read_timeout_s: 0, the original request deadline bounds the stream.
Retries¶
serving.max_attempts sets the prefill/decode attempts per admitted request, from 1 to 3. The default, 2, allows one retry. To choose a value, follow Retries and quarantine.
A fresh attempt starts for a transient fault before visible output when:
- the original request deadline permits it
- retry budget remains
A retry reserves one retry credit when it is scheduled and spends that credit when its prefill dispatches. A retry that ends before dispatch returns its credit.
Each retry receives:
- fresh backend request IDs
- a new KV handoff
- placement that avoids the engines that failed earlier attempts
Retry placement¶
A retry excludes, for each role, every engine whose leg failed earlier in the same request:
| Failed leg | Engines the retry excludes |
|---|---|
| Prefill | That prefill engine |
| Decode, before visible output | That decode engine and the attempt's producer |
Each exclusion lasts for that request only.
When no live engine outside the exclusions can take a role's legs, the request ends with HTTP 503 and error type backend_unavailable. Under aggregate fallback, an unpinned engine of the other role can still take them. The journal records reason no_engine, and the final attempt entry carries retry_reason not_dispatched.
In predictive mode, a retry skips the TTFT check and runs the decode admission check for retries.
With recovery.failure_quarantine_s above 0, placement of the engine whose leg failed follows failure quarantine.