Ordered benchmark points¶
tools/measurement/benchmark_runner.py runs the plan's points in order, with these steps per point:
- Check that
/readyreportsready. - Check that
/v1/modelslists exactly the--modelname. - Wait for an idle router.
- Start the evidence collector when the plan has an
evidenceobject. - Run the point's client.
- Wait for an idle router.
- Write
result.json.
Prerequisites:
- A passing preflight against the current fleet.
- A router reserved for this run.
- A router and engines that stay running through the benchmark.
For a plan with an evidence object, run the runner on a host that reads the router's append-only JSONL journal as a local file and reaches the router and every engine metrics endpoint. Run other plans from the workstation repository root, through the private router tunnel.
Running a plan¶
- Prepare the workload with the load trial helper.
-
Write a private plan at
runs/benchmark-plan.jsonwithworkload.fileand--workloadset to the prepared file:{ "schema": 1, "points": [ { "id": "rate-0_5", "workload": {"file": "runs/load-trial-EXAMPLE/workload/workload.json", "rate_rps": 0.5, "requests": 200}, "client_argv": [ ".venv/bin/python", "tools/measurement/load_trial.py", "run", "--base", "{base}", "--expected-model", "{model}", "--workload", "runs/load-trial-EXAMPLE/workload/workload.json", "--rate", "0.5", "--requests", "200", "--ttft", "2", "--tpot", "0.0333", "--attainment", "0.95", "--out", "{point_dir}/client" ], "client_timeout_s": 900, "drain_timeout_s": 180 }, { "id": "rate-1", "workload": {"file": "runs/load-trial-EXAMPLE/workload/workload.json", "rate_rps": 1, "requests": 200}, "client_argv": [ ".venv/bin/python", "tools/measurement/load_trial.py", "run", "--base", "{base}", "--expected-model", "{model}", "--workload", "runs/load-trial-EXAMPLE/workload/workload.json", "--rate", "1", "--requests", "200", "--ttft", "2", "--tpot", "0.0333", "--attainment", "0.95", "--out", "{point_dir}/client" ], "client_timeout_s": 900, "drain_timeout_s": 180 } ] } -
Run the plan through the private router tunnel:
Running AIPerf points¶
AIPerf 0.13.0 is the standard benchmark client for fleet runs.
-
Install AIPerf 0.13.0 in a Python 3.11 through 3.13 virtual environment on the runner host, with
<aiperf-venv>replaced by the environment path: -
Check the installed version:
Expected output:
-
Add an AIPerf point to the plan's
points, with<tokenizer-path>replaced by the served model's tokenizer directory:{ "id": "aiperf-rate-1", "workload": {"client": "aiperf 0.13.0", "isl": 512, "osl": 256, "rate_rps": 1, "requests": 200}, "client_argv": [ "<aiperf-venv>/bin/aiperf", "profile", "--url", "{base}", "--model", "{model}", "--tokenizer", "<tokenizer-path>", "--endpoint-type", "chat", "--streaming", "--isl", "512", "--osl", "256", "--request-rate", "1", "--request-count", "200", "--ui", "none", "--no-gpu-telemetry", "--artifact-dir", "{point_dir}/aiperf" ], "client_timeout_s": 900, "drain_timeout_s": 180 } -
Set these fields in the plan's
evidenceobject:Field Value client_records{point_dir}/aiperf/profile_export.jsonlidentity.benchmark_client_versionaiperf 0.13.0
Plan fields¶
A plan is a JSON object with these fields:
| Field | Value |
|---|---|
schema |
1 |
points |
Ordered array of points |
evidence |
Optional evidence collector settings |
points[].id |
Unique name of letters, digits, hyphens, and underscores |
points[].workload |
Nonempty object copied into result.json |
points[].client_argv |
Nonempty argument vector that contains {base} and {model} |
points[].client_timeout_s |
Positive client deadline in seconds |
points[].drain_timeout_s |
Positive drain deadline in seconds for each drain wait |
client_argv placeholders:
| Placeholder | Value |
|---|---|
{base} |
The --base URL |
{model} |
The --model served name |
{point_id} |
The point's id |
{point_dir} |
The point's directory |
Bearer-token ingress:
- Export the token in an environment variable.
- Pass the variable name with
--api-key-env NAMEfor the runner's probes, drain checks, and router samples. - Pass client credentials in
client_argv, such as the load trial helper's--api-key-env NAME.
Output and exit status¶
The new private --out directory holds:
| File | Location | Contents |
|---|---|---|
manifest.json |
Output root | Plan, its SHA-256 digest, runner digest, model, URL, and invocation |
result.json |
Point directory | Readiness, client exit status, initial and final drain condition, timestamps, last state snapshot, and evidence diagnostic count |
client.stdout, client.stderr |
Point directory | External client output |
The runner exits 0 when every point reaches completed. It exits 1 when the plan fails validation or a point ends with another condition.
The runner stops at the first point that ends with one of these result.json conditions:
condition |
Cause |
|---|---|
readiness_refused |
/ready returned a status other than 200 with ready |
models_unavailable |
/v1/models returned a status other than 200 |
model_mismatch |
/v1/models listed models other than exactly --model |
probe_error |
A readiness or model probe failed |
initial_drain_timeout, initial_drain_state_error |
The drain wait before the client timed out or failed to read /narwhal/state |
drain_timeout, drain_state_error |
The drain wait after the client timed out or failed to read /narwhal/state |
client_failure |
The client timed out, failed to start, or returned a nonzero exit code |
evidence_error |
The evidence collector failed |