Skip to content

Engine profiles and capacity

Validating the engine cost model

narwhal-profile measures each engine and writes its cost model as one row in the profile store at profiles.path.

Profile store document:

{
  "schema": "narwhal.profiles",
  "schema_version": 1,
  "meta": {"package": "narwhal-inference", "version": "0.3.1", "git": "<commit>", "source": "sha256:..."},
  "profiles": []
}

The profiler fails the run when an engine's process digest changes between the start and end of its sweep.

The saved digest and the .samples.json sidecar contents depend on the fleet:

Fleet Saved digest .samples.json sidecar keeps
With engine_contract and a launch digest in the attestation Attested launch digest. The full attestation response.
Other fleets with engine_contract Per-process attestation digest. The full attestation response.
Otherwise Digest of the process identity from /version and /metrics. The process identity.

The attested launch digest covers:

  • the contract fields
  • the engine arguments, with the values of --host, --port, --served-model-name, and --kv-events-config removed
  • the image
  • the packages
  • the model configuration and revision
  • the hashes of the launcher, the launch record, and the cache-capture hook
  • the ucx_version and peer_release values from the image check

Preflight, router startup, readmission, and automatic recovery require a fresh profile when an engine's stored measured profile has missing generation evidence or a digest that differs from the engine's live generation. The error names the engine.

If preflight, router startup, or recovery reports profile has no generation evidence:

  1. Re-profile the named engines against the current engine processes.
  2. Keep the .samples.json sidecar.

The refit procedure requires a generation_digest in each saved profile and a generation_evidence object in its sample row. Samples missing either need a fresh sweep.

On a fleet with engine_contract, load a fresh profile store by activating it with router resume. On other fleets, restart the router.

A malformed profile stops the operation with an error that names the file, engine, and field:

profiles.json: profile n4: tpot_slope must be nonnegative

The profile store must hold the same set of iid values as the configured fleet. Combine rows from separate narwhal-profile --only runs with narwhal-profile --merge.

Profile fields

Each profile row has these fields:

Field JSON type Constraint
iid string Nonempty.
generation_digest string sha256: digest of the attested launch, the verified attestation, or the process identity.
ttft_a, ttft_b, ttft_c number Nonnegative coefficients of the quadratic prefill fit.
ttft_block_tokens integer, optional Positive cache block size, set or null together with ttft_split.
ttft_split number, optional Nonnegative extra prefill time for a prompt that ends inside a block past the first, set or null together with ttft_block_tokens.
tpot_slope number Nonnegative increase in decode interval per resident KV token.
tpot_intercept number Nonnegative zero-contention decode interval.
tpot_request_slope number, optional Nonnegative increase in decode interval per active sequence, default 0.
kv_capacity_tokens integer, optional Positive, and at least decode_max_kv_tokens.
decode_min_requests, decode_max_requests integer Positive measured concurrency range with min <= max.
decode_min_kv_tokens, decode_max_kv_tokens integer Positive measured range of resident KV tokens with min <= max.
decode_fit_mape, decode_cv_mape number Nonnegative fit error and leave-one-cell-out cross-validation error.
cached_ttft_a, cached_ttft_b, cached_ttft_c, cached_ttft_d number, optional Nonnegative coefficients of the warm prefill fit.
cached_cv_mape number, optional Nonnegative leave-one-case-out error of the warm prefill fit.
cached_min_prefix_tokens, cached_max_prefix_tokens integer, optional Positive measured cached-prefix range with min <= max.
cached_min_suffix_tokens, cached_max_suffix_tokens integer, optional Positive measured uncached-suffix range with min <= max.
prefill_min_tokens, prefill_max_tokens integer, optional Positive measured prompt-length range with min <= max.
decode_min_output_tokens, decode_max_output_tokens integer, optional Positive measured output-length range with min <= max.
colocated_group string, optional Nonempty shared-device group of a profile measured under one colocated role mix.
colocated_target_role string prefill or decode, required with colocated_group.
colocated_prefill_engines, colocated_decode_engines integer Positive engine counts of the measured role mix, at least two engines in total, required with colocated_group.
colocated_prefill_rps, colocated_decode_rps number Nonnegative measured neighbour load in requests per second, required with colocated_group.
nominal boolean, optional true for a nominal profile, with generation_digest unset. Default false.

Nominal profiles

A nominal profile states an engine's cost from four figures in place of a sweep. Its row sets nominal to true and these fields:

Figure Fields
Prefill time per prompt token, positive ttft_b, with ttft_a and ttft_c at 0
Decode token interval, positive tpot_intercept, with tpot_slope at 0
KV capacity in tokens kv_capacity_tokens and decode_max_kv_tokens, with decode_min_kv_tokens at 1
Largest number of concurrent sequences decode_max_requests, with decode_min_requests at 1

Both fit errors are 0.

Router startup, preflight and automatic recovery pass a nominal profile with no generation check, so the profile stays valid across engine restarts. Preflight reports the stated figures in place of the fit errors. A fleet with engine_contract refuses a nominal profile and names the engine.

Profile loading aborts on true, "96" or 1.5 in an integer field, on NaN or Infinity, on a field outside the profile fields table, and on a partial set of cached_ fields.

The router prices a request with a cached prefix from the cached_ fields, following the prefix-cache pricing rules.

Prefill price derived from the profile

The router prices a cold prompt of n tokens with the prefill fit:

Prompt length Price
Shorter than prefill_min_tokens Fit at prefill_min_tokens
From prefill_min_tokens to prefill_max_tokens Fit at n
Longer than prefill_max_tokens Fit extended to n

Placement, admission, pool load, demand, and role-split scoring all use this price.

Decode capacity derived from the profile

Narwhal limits concurrent decode requests on each fitted engine by the priced context length, context_tokens.

When context_tokens <= 0 or decode_max_requests is null, the decode request limit is zero. Otherwise the limit is the smaller of decode_max_requests and the KV budget divided by context_tokens, with a minimum of 1.

For a profile with kv_capacity_tokens, the KV budget is the smaller of decode_max_kv_tokens and kv_capacity_tokens. Otherwise the KV budget is decode_max_kv_tokens.

To price the token interval for decode request capacity, Narwhal counts at least decode_min_requests requests and at least decode_min_kv_tokens resident KV tokens in the batch.