Admission and responses¶
Admission and refusal semantics¶
The time to first token (TTFT) budget is slo.ttft_s * (1 + serving.admission_margin).
Each row gives the HTTP status and error type that a completion route returns for one condition before output starts. A blank code cell means the error body omits code, and null means the body carries "code": null.
| Condition | HTTP | Error type |
Error code |
Retry-After |
|---|---|---|---|---|
The body is not valid JSON or not a JSON object, with param null, or a field the router reads has the wrong type, with param naming the field |
400 |
invalid_request_error |
null |
|
On vLLM, vllm_xargs sets kv_cache_report_mode, kv_transfer_params, or ec_transfer_params; param is vllm_xargs |
400 |
invalid_request_error |
null |
|
On SGLang, the body sets rid, bootstrap_host, bootstrap_port, or bootstrap_room; param names the field |
400 |
invalid_request_error |
null |
|
On SGLang, a /v1/chat/completions body sets return_token_ids, return_prompt_token_ids, or return_meta_info; param names the field |
400 |
invalid_request_error |
null |
|
Unsupported non-streaming audio, modality, or tool request; param names the option |
400 |
invalid_request_error |
null |
|
n > 1 or best_of > 1 |
400 |
invalid_request_error |
||
| Requested model differs from the configured model | 404 |
invalid_request_error |
model_not_found |
|
Request exceeds serving.max_request_bytes |
413 |
request_too_large |
||
Requests counted against the in-flight limit reach it plus serving.queue_capacity; message router in-flight limit reached |
429 |
server_overloaded_error |
1 |
|
Router event-loop lag reaches a quarter of slo.ttft_s |
429 |
server_overloaded_error |
1 |
|
Median token-counting and prefix-hashing time of at least 8 requests in the last 2 seconds reaches a quarter of slo.ttft_s |
429 |
server_overloaded_error |
1 |
|
| Projected TTFT exceeds the budget for a prompt that fits alone, at prefill placement or during an admission-seat or prefill-seat wait | 429 |
server_overloaded_error |
Placement price minus the budget in seconds, excluding the time the request has waited, rounded up, minimum 1 |
|
| Prompt alone exceeds the TTFT budget | 429 |
server_overloaded_error |
||
| Zero prefill engines are live and the decode target holds resident decode work | 429 |
server_overloaded_error |
1 |
|
| The projected TTFT plus the projected decode slot wait exceeds the budget | 429 |
server_overloaded_error |
1 |
|
| The peak KV tokens projected in decode during the request's decode hold exceed live decode capacity | 429 |
server_overloaded_error |
1 |
|
Decode load pushes the request past slo.tpot_s on every live decode engine |
429 |
server_overloaded_error |
1 |
|
| Zero engines are live while the router holds its lease, with engine monitoring healthy and no lifecycle hold | 503 |
backend_unavailable |
backend_unavailable |
1 |
| Router is standby, fenced, in a whole-wave hold, awaiting engine identity validation, or in degraded engine monitoring | 503 |
standby |
standby |
1 |
| A whole-wave hold begins while the request waits for a seat or before its decode dispatch | 503 |
standby |
standby |
1 |
| Degraded engine monitoring begins, or the router loses its lease, while the request waits for an admission or prefill seat | 503 |
standby |
standby |
1 |
| No engine can take a leg of the request before its first prefill dispatch | 503 |
no_schedulable_engines |
no_schedulable_engines |
1 |
| After the first prefill dispatch, no engine outside the request's failed engines can take its decode leg or a retry's leg | 503 |
backend_unavailable |
backend_unavailable |
1 |
The admission-seat wait reaches serving.queue_timeout_s before the original request deadline |
504 |
queue_expired |
||
A prefill-seat wait reaches the remaining serving.queue_timeout_s before the original request deadline |
504 |
expired |
expired |
|
| On vLLM, the KV handoff reaches its bound before decode dispatch, including during a decode-seat wait, with no attempts left | 504 |
handoff_expired |
handoff_expired |
|
| The original request deadline expires before response headers, at any stage, including a seat wait | 504 |
request_expired |
An engine fault before output starts returns HTTP 502 or 504 with the request phase as its error type. A streaming decode failure after output starts ends the HTTP 200 stream with a terminal error event of type decode.
Shorten the prompt or raise slo.ttft_s to clear a 429 for an oversized prompt.
/ready reports the reason for each 503 refusal.
serving.admission selects the admission mode. open enforces the router saturation checks, the in-flight limit, and the queue and engine seat limits. predictive, the default, adds the predictive TTFT, decode-capacity, and slo.tpot_s checks to every open check and limit. To choose a mode, queue setting and retry limit, follow Choosing admission, queue and retry settings.
Admission counters¶
The router keeps these admission counters:
| Counter | Increments on |
|---|---|
offered |
Each request to a completion route |
unsized_offered |
Termination before workload sizing |
rejected |
HTTP 429 capacity refusal or HTTP 503 router-readiness refusal |
refused |
Global predictive refusal |
invalid_requests |
HTTP 400, 404, or 413 refusal before admission |
Journal rows and /metrics split rejected and refused by outcome reason.
Streaming and response assembly¶
Streaming responses¶
Streaming deltas use the engine's response shape under the token-ID rules.
The router relays the engine stream with these rules:
| Engine stream | Client stream |
|---|---|
| Line ends | CRLF, CR and LF each end an SSE line |
data: event |
One client event in compact JSON, with token fields removed unless the client set return_token_ids |
Malformed data: payload |
Relayed as received, counted as zero tokens |
data: events before the first token |
Held until the first token, up to 64 events and serving.max_response_bytes |
| Other lines before the first token | Dropped |
| Other lines after the first token | Relayed, each as its own event |
| One transport read | One client write holding that read's events |
Non-streaming assembly¶
Non-streaming responses assemble the engine stream as follows:
| Engine output | Assembly behaviour |
|---|---|
Chat content |
String deltas concatenate under content |
Chat reasoning |
String deltas concatenate under reasoning |
Chat reasoning_content |
String deltas concatenate under reasoning_content |
Chat refusal |
String deltas concatenate under refusal |
tool_calls |
One call per stream index, returned in index order |
| Tool call fields | ID, function name, and arguments concatenate separately per call |
| Tool call arguments | Returned as engine-generated strings |
Legacy function_call |
Function name and argument fragments concatenate into one message field |
| Chat logprobs | content and refusal arrays concatenate in stream order |
| Text-completion logprobs | tokens, token_logprobs, top_logprobs, and text_offset arrays concatenate in stream order |
Assembly returns HTTP 502 when:
- an unsupported choice or chat-delta field carries a value
- a supported field has a malformed value
- a tool call is missing its ID or function name
Metadata and usage¶
Assembled responses carry the engine's response metadata. finish_reason, stop_reason and, on SGLang, matched_stop come from the last choice that reports each.
usage carries the non-null engine usage, including one from a later usage-only frame. When the engine omits usage, the router computes it from the input length and the measured output token count.
Token identity and output accounting¶
On vLLM, every decode request includes:
On SGLang, a /v1/completions decode request includes "return_token_ids": true, and the launcher sets --stream-interval 1. A /v1/chat/completions decode request on SGLang carries no token-ID field, and the router counts one token for each streamed chunk that carries text, reasoning content, or tool calls.
When the decode request asks for token IDs, every event with text, reasoning, tool calls, or refusals carries a token-ID list of nonnegative integers. A missing or malformed list, including one with a Boolean ID, fails the decode attempt or profiling measurement.
The router reports token_accounting: token_ids on vLLM and SGLang, and token_accounting: reported on llama.cpp.
Clients that set return_token_ids receive the IDs in streaming and assembled responses. On SGLang, the router refuses a /v1/chat/completions request that sets return_token_ids with HTTP 400.