Measuring a fleet¶
A fleet measurement reports the highest tested offered rate that meets the fleet's time to first token (TTFT) and time per output token (TPOT) targets.
Running a measurement¶
- Measurement contract and profiling: router and client latency boundaries, and the idle-fleet latency profile.
- Targets and deployment freeze: production TTFT and TPOT targets, and the frozen deployment under test.
- Synthetic load trial: 200 requests each at 0.5 and 1 request/s through the private router tunnel.
- Reconciling and accepting: the client-to-journal join, throughput and client-limit checks, the post-load KV ring check, and deployment acceptance.
Automating benchmark points¶
-
A plan's trial points with readiness and drain checks per point.
-
Journal rows, metrics samples, file digests, and a shareable summary per point.
Placement trials¶
-
Cold-priced and cache-aware arms on one fleet, scored on repeated-prefix and cold-control workloads.
Router throughput¶
-
Relayed frames per router CPU-second for one router process against simulated engines.