Cache-aware placement trial¶
The trial compares the throughput of requests that meet the service-level objectives (SLOs) between two arms on the same fleet. The baseline arm prices every engine cold. The cache-aware arm prices each engine that holds the request's prefix at its warm fit.
Both arms run with prefix caching on, and the baseline arm turns off cache events with runtime.kv_events: false.
Freezing the comparison¶
Record these in the private execution record before the first measured run:
- the benefit and regression thresholds
- the fleet, model, engine image, launch policy, SLOs, attainment target, and controller settings
- the workload shape, family count, and profile lengths
- the rate pilot rule, the seeds, and the run order
The arms differ only in runtime.kv_events.
Each run's score is its qualified_rps_including_drain and attainment. Refusals and failures count as misses.
Choosing the workload shape¶
L is the input token count of every request in a workload. B is the residency block size, reported as block_size in each engine's residency record in /narwhal/state.
For a repeated-prefix workload on an engine with boundary-state groups, such as Mamba state:
- set
--input-tokenstok*B + swith0 < s < B - set
--prefix-tokensto at leastk*B - expect a cached prefix of
k*Btokens and an uncached suffix ofstokens on the engine holding the family
On an engine with full-attention groups only, the cached prefix is the largest multiple of B within both the shared prefix and L - 1 tokens.
An engine prices a cached prefix with its warm fit when both lengths lie inside its measured warm domain in the profile fields. The domain spans cached_min_prefix_tokens to cached_max_prefix_tokens for the cached prefix and cached_min_suffix_tokens to cached_max_suffix_tokens for the uncached suffix.
Profile both arms with these profiler options:
| Flag | Value |
|---|---|
--cached-prefix-lens |
Lengths that bracket the trial's cached prefix |
--cached-suffix-lens |
Lengths that bracket the trial's uncached suffix |
--decode-input-lens |
Lengths up to the trial's input length |
Preparing each arm¶
The baseline arm sets runtime.kv_events to false, and every sidecar returns HTTP 404 from the residency routes. The cache-aware arm leaves runtime.kv_events unset, and every sidecar serves the residency routes.
For each arm:
- Launch and check every engine as in Validate and start engines.
- Attest the engines as in Attest live engines.
- Profile the engines with the lengths from Choosing the workload shape.
- Calibrate the first-token deadline over every directed engine pair as in Calibrating the first-token deadline.
- Run a passing preflight.
- Start the router with
pin: falseon every engine and the recorded controller settings.
Before each cache-aware run, confirm:
- every engine's
residencyrecord reports"known": true - every profile carries its
cached_fields
Creating the workloads¶
- Set
NARWHAL_TRIAL_URLandTRIAL_DIRas in Creating the trial directory and workload. - From the management workstation, prepare a repeated-prefix workload for each pass and one cold control with the same shape.
Repeated-prefix workload:
.venv/bin/python tools/measurement/load_trial.py prepare \
--base "$NARWHAL_TRIAL_URL" --out "$TRIAL_DIR/repeated-1" \
--input-tokens <input-tokens> --output-tokens <output-tokens> \
--prefix-tokens <prefix-tokens> --families <families> --seed <seed>
Cold control:
.venv/bin/python tools/measurement/load_trial.py prepare \
--base "$NARWHAL_TRIAL_URL" --out "$TRIAL_DIR/control" \
--input-tokens <input-tokens> --output-tokens <output-tokens> \
--prefix-tokens <prefix-tokens> --families 0 --seed <seed>
Replace each <...> value with the recorded choice.
Give the second pass's repeated-prefix workload, repeated-2, a distinct --seed.
Setting the offered rates¶
- In the baseline arm, run a 60-second point at each pilot rate, each with a fresh repeated-prefix workload.
- Take
R_bas the highest pilot rate with attainment at or above the recorded attainment target. - Set the low rate to
1.1 * R_band the high rate to1.3 * R_b, each rounded to 0.5 request/s.
Checking cache-aware pricing¶
- In the cache-aware arm, run one short, unscored point with a repeated-prefix workload.
- Join its
requests.jsonlto the router journal onclient_rid, as in Joining client offers to the router journal. - Compare
predicted_prefill_swithcold_prefill_sin thecache_placementrecord of each row withplaced_cached_tokensabove 0.
When predicted_prefill_s is below cold_prefill_s on every row, start the measured runs. When it is at or above cold_prefill_s on any row, revisit the workload shape and the warm prefill lengths.
Running each point¶
In each arm:
- Run one unscored 120-second warm-up at the low rate.
- Run pass 1:
repeated-1low, control low,repeated-1high, control high. - Run pass 2: control high,
repeated-2high, control low,repeated-2low.
Run one point:
.venv/bin/python tools/measurement/load_trial.py run \
--base "$NARWHAL_TRIAL_URL" --out "$TRIAL_DIR/<arm>-<point>" \
--workload "$TRIAL_DIR/<workload>/workload.json" \
--rate <rate> --requests <requests> --run-seed <run-seed> \
--ttft <ttft-s> --tpot <tpot-s> --attainment <attainment>
Set --requests to 60 seconds of offers at the point's rate. Give each point a distinct --run-seed, shared by both arms.
A run that exits with status 0 or 2 writes summary.json with:
| Field | Meaning |
|---|---|
attainment |
Completed requests within the time to first token (TTFT) and time per output token (TPOT) limits, over all offers |
qualified_rps_including_drain |
Completed requests within the limits per second of the measurement window |
ttft_s, tpot_s |
p50, p95, and p99 over completed requests |
outcomes |
Offers by client outcome, with router refusals in http_error |
Between runs:
- Drain the router.
- Reconcile the run with the journal as in Reconciling and accepting.
Exercising a role change¶
Run one separate, unscored cache-aware point after the measured runs.
A reactive prefill-to-decode move requires every adjacent-split condition.
Run the role-change point:
- Prepare a decode-heavy workload with a larger
--output-tokensand the repeated-prefix workload's--seed,--input-tokens,--prefix-tokens, and--families. - Run it at a rate that raises decode load until
flipsin/narwhal/staterecords the move with"by": "reactive". - Confirm that
pools.prefilllists the remaining prefill engines. - Confirm that prefill placements after the change go to the remaining prefill engines.
- Confirm that journal rows placed on the moved engine before the change complete with that engine as
prefill_iid. - Confirm that the move's
flipsrecord reports a numericdrained_s. - Confirm that the moved engine's
residencyrecord keeps"known": trueand the sameepochacross the change.
Reporting the result¶
For each arm and point, report:
qualified_rps_including_drain,attainment, and refusals by cause- the
ttft_sandtpot_sdistributions - controller decisions and
flips - the prefill engines' prefix-cache hit rate from engine metrics
In the cache-aware arm, report these per rate over journal rows with a cache_placement record and attempts equal to 1:
- the estimation error of
cache_placement.predicted_prefill_sagainstupstream_seconds.prefill - the share of rows where
cache_placement.cold_choice_iiddiffers fromcache_placement.placed_iid
Report these overhead measures:
- residency resynchronisations, from the change in each engine's
residencyresyncsbetween the run'sstate-before.jsonandstate-after.json - sidecar and router CPU and memory, from host process accounting
Compare the arms on the mean of the two passes at each point.
The trial passes when the cache-aware arm meets both thresholds at both rates:
- the benefit threshold on the repeated-prefix workload
- the regression threshold on the control