Targets and deployment freeze¶
Setting production SLOs¶
slo.ttft_s sets the time to first token (TTFT) target, and slo.tpot_s sets the time per output token (TPOT) target.
- Run light traffic with accepted profiles.
- Set both targets from the service requirement and the latency distribution measured in step 1.
-
Validate the config:
A TPOT target below the engine's measured per-token time gives zero feasible decode capacity.
The narwhal-check slo gate passes each engine when slo.tpot_s is at or above the profile's token interval at the smallest measured decode cohort, and slo.ttft_s is above the profile's single-token prefill time.
The narwhal-check pace gate compares each engine's prefill pace with the fleet median, its saved profile, or both.
Freezing the deployment under test¶
- Assign a deployment identifier before the load test.
-
Attach the exact artifacts to it:
- the Narwhal release, source revision, and distribution digest
- the fleet, router, and engine configurations
- the profile files, sample store, engine image digest, and engine launcher
- the attestation documents, preflight output, endpoint captures, and deployment-client output
- the router journal, state snapshots, and metrics
-
Record the hosts and SSH tunnel mapping with the identifier.
The offered-rate sweep holds these fixed:
- source revision
- model
- runtime
- profiles
- router targets
- workload shape
- cache policy
- both latency targets
Before the next rate, wait for:
- resident work to finish
- transfer leases to release
Stop the sweep at the first of:
- a run that misses the trial's attainment target
- a tested rate at the intended operating ceiling