Skip to content

Cache-aware placement trial

The trial compares the throughput of requests that meet the service-level objectives (SLOs) between two arms on the same fleet. The baseline arm prices every engine cold. The cache-aware arm prices each engine that holds the request's prefix at its warm fit.

Both arms run with prefix caching on, and the baseline arm turns off cache events with runtime.kv_events: false.

Freezing the comparison

Record these in the private execution record before the first measured run:

  • the benefit and regression thresholds
  • the fleet, model, engine image, launch policy, SLOs, attainment target, and controller settings
  • the workload shape, family count, and profile lengths
  • the rate pilot rule, the seeds, and the run order

The arms differ only in runtime.kv_events.

Each run's score is its qualified_rps_including_drain and attainment. Refusals and failures count as misses.

Choosing the workload shape

L is the input token count of every request in a workload. B is the residency block size, reported as block_size in each engine's residency record in /narwhal/state.

For a repeated-prefix workload on an engine with boundary-state groups, such as Mamba state:

  • set --input-tokens to k*B + s with 0 < s < B
  • set --prefix-tokens to at least k*B
  • expect a cached prefix of k*B tokens and an uncached suffix of s tokens on the engine holding the family

On an engine with full-attention groups only, the cached prefix is the largest multiple of B within both the shared prefix and L - 1 tokens.

An engine prices a cached prefix with its warm fit when both lengths lie inside its measured warm domain in the profile fields. The domain spans cached_min_prefix_tokens to cached_max_prefix_tokens for the cached prefix and cached_min_suffix_tokens to cached_max_suffix_tokens for the uncached suffix.

Profile both arms with these profiler options:

Flag Value
--cached-prefix-lens Lengths that bracket the trial's cached prefix
--cached-suffix-lens Lengths that bracket the trial's uncached suffix
--decode-input-lens Lengths up to the trial's input length

Preparing each arm

The baseline arm sets runtime.kv_events to false, and every sidecar returns HTTP 404 from the residency routes. The cache-aware arm leaves runtime.kv_events unset, and every sidecar serves the residency routes.

For each arm:

  1. Launch and check every engine as in Validate and start engines.
  2. Attest the engines as in Attest live engines.
  3. Profile the engines with the lengths from Choosing the workload shape.
  4. Calibrate the first-token deadline over every directed engine pair as in Calibrating the first-token deadline.
  5. Run a passing preflight.
  6. Start the router with pin: false on every engine and the recorded controller settings.

Before each cache-aware run, confirm:

  • every engine's residency record reports "known": true
  • every profile carries its cached_ fields

Creating the workloads

  1. Set NARWHAL_TRIAL_URL and TRIAL_DIR as in Creating the trial directory and workload.
  2. From the management workstation, prepare a repeated-prefix workload for each pass and one cold control with the same shape.

Repeated-prefix workload:

.venv/bin/python tools/measurement/load_trial.py prepare \
  --base "$NARWHAL_TRIAL_URL" --out "$TRIAL_DIR/repeated-1" \
  --input-tokens <input-tokens> --output-tokens <output-tokens> \
  --prefix-tokens <prefix-tokens> --families <families> --seed <seed>

Cold control:

.venv/bin/python tools/measurement/load_trial.py prepare \
  --base "$NARWHAL_TRIAL_URL" --out "$TRIAL_DIR/control" \
  --input-tokens <input-tokens> --output-tokens <output-tokens> \
  --prefix-tokens <prefix-tokens> --families 0 --seed <seed>

Replace each <...> value with the recorded choice.

Give the second pass's repeated-prefix workload, repeated-2, a distinct --seed.

Setting the offered rates

  1. In the baseline arm, run a 60-second point at each pilot rate, each with a fresh repeated-prefix workload.
  2. Take R_b as the highest pilot rate with attainment at or above the recorded attainment target.
  3. Set the low rate to 1.1 * R_b and the high rate to 1.3 * R_b, each rounded to 0.5 request/s.

Checking cache-aware pricing

  1. In the cache-aware arm, run one short, unscored point with a repeated-prefix workload.
  2. Join its requests.jsonl to the router journal on client_rid, as in Joining client offers to the router journal.
  3. Compare predicted_prefill_s with cold_prefill_s in the cache_placement record of each row with placed_cached_tokens above 0.

When predicted_prefill_s is below cold_prefill_s on every row, start the measured runs. When it is at or above cold_prefill_s on any row, revisit the workload shape and the warm prefill lengths.

Running each point

In each arm:

  1. Run one unscored 120-second warm-up at the low rate.
  2. Run pass 1: repeated-1 low, control low, repeated-1 high, control high.
  3. Run pass 2: control high, repeated-2 high, control low, repeated-2 low.

Run one point:

.venv/bin/python tools/measurement/load_trial.py run \
  --base "$NARWHAL_TRIAL_URL" --out "$TRIAL_DIR/<arm>-<point>" \
  --workload "$TRIAL_DIR/<workload>/workload.json" \
  --rate <rate> --requests <requests> --run-seed <run-seed> \
  --ttft <ttft-s> --tpot <tpot-s> --attainment <attainment>

Set --requests to 60 seconds of offers at the point's rate. Give each point a distinct --run-seed, shared by both arms.

A run that exits with status 0 or 2 writes summary.json with:

Field Meaning
attainment Completed requests within the time to first token (TTFT) and time per output token (TPOT) limits, over all offers
qualified_rps_including_drain Completed requests within the limits per second of the measurement window
ttft_s, tpot_s p50, p95, and p99 over completed requests
outcomes Offers by client outcome, with router refusals in http_error

Between runs:

  1. Drain the router.
  2. Reconcile the run with the journal as in Reconciling and accepting.

Exercising a role change

Run one separate, unscored cache-aware point after the measured runs.

A reactive prefill-to-decode move requires every adjacent-split condition.

Run the role-change point:

  1. Prepare a decode-heavy workload with a larger --output-tokens and the repeated-prefix workload's --seed, --input-tokens, --prefix-tokens, and --families.
  2. Run it at a rate that raises decode load until flips in /narwhal/state records the move with "by": "reactive".
  3. Confirm that pools.prefill lists the remaining prefill engines.
  4. Confirm that prefill placements after the change go to the remaining prefill engines.
  5. Confirm that journal rows placed on the moved engine before the change complete with that engine as prefill_iid.
  6. Confirm that the move's flips record reports a numeric drained_s.
  7. Confirm that the moved engine's residency record keeps "known": true and the same epoch across the change.

Reporting the result

For each arm and point, report:

  • qualified_rps_including_drain, attainment, and refusals by cause
  • the ttft_s and tpot_s distributions
  • controller decisions and flips
  • the prefill engines' prefix-cache hit rate from engine metrics

In the cache-aware arm, report these per rate over journal rows with a cache_placement record and attempts equal to 1:

  • the estimation error of cache_placement.predicted_prefill_s against upstream_seconds.prefill
  • the share of rows where cache_placement.cold_choice_iid differs from cache_placement.placed_iid

Report these overhead measures:

  • residency resynchronisations, from the change in each engine's residency resyncs between the run's state-before.json and state-after.json
  • sidecar and router CPU and memory, from host process accounting

Compare the arms on the mean of the two passes at each point.

The trial passes when the cache-aware arm meets both thresholds at both rates:

  • the benefit threshold on the repeated-prefix workload
  • the regression threshold on the control