Skip to content

Measuring a fleet

A fleet measurement reports the highest tested offered rate that meets the fleet's time to first token (TTFT) and time per output token (TPOT) targets.

Running a measurement

  1. Measurement contract and profiling: router and client latency boundaries, and the idle-fleet latency profile.
  2. Targets and deployment freeze: production TTFT and TPOT targets, and the frozen deployment under test.
  3. Synthetic load trial: 200 requests each at 0.5 and 1 request/s through the private router tunnel.
  4. Reconciling and accepting: the client-to-journal join, throughput and client-limit checks, the post-load KV ring check, and deployment acceptance.

Automating benchmark points

Placement trials

Router throughput