Targets and deployment freeze¶
5. Set production SLOs¶
Run light traffic with accepted profiles, then set slo.ttft_s and slo.tpot_s from the service requirement and measured latency distribution.
A TPOT target below the measured per-token floor of the engine shape yields zero feasible decode capacity.
After changing an SLO, run narwhal-check against the edited fleet to test the target with saved profiles and live handoffs:
Pace gate¶
With at least three successful probes, the pace gate compares each engine with the fleet median under a 1.5x slowdown limit. For one or two successful probes, it requires a saved prefill profile and exact usage.prompt_tokens for each engine to apply the same limit.
6. Freeze the deployment under test¶
Before the load test, assign a deployment identifier and attach the exact:
- Narwhal release;
- source revision;
- distribution digest;
- fleet configuration;
- profile files;
- sample store;
- engine image digest;
- engine launcher;
- attestation documents;
- router configuration;
- engine configuration;
- preflight output;
- endpoint captures;
- deployment-client output;
- router journal;
- state snapshots;
- metrics.
Record the workstation host, router host, and SSH tunnel mapping under the same identifier.
Across the offered-rate sweep, vary the request rate while holding these inputs fixed:
- source revision;
- model;
- runtime;
- profiles;
- router targets;
- workload shape;
- cache policy;
- TTFT target;
- TPOT target.
Between rates, drain resident work and transfer leases.
End the sweep when a run with a valid client schedule misses the trial's attainment target, or after testing the intended operating ceiling.
Continue with the synthetic load trial.