Engine profiles and capacity¶
Validating the engine cost model¶
narwhal-profile measures each engine and writes its cost model as one row in the profile store at profiles.path.
Profile store document:
{
"schema": "narwhal.profiles",
"schema_version": 1,
"meta": {"package": "narwhal-inference", "version": "0.3.1", "git": "<commit>", "source": "sha256:..."},
"profiles": []
}
The profiler fails the run when an engine's process digest changes between the start and end of its sweep.
The saved digest and the .samples.json sidecar contents depend on the fleet:
| Fleet | Saved digest | .samples.json sidecar keeps |
|---|---|---|
With engine_contract and a launch digest in the attestation |
Attested launch digest. | The full attestation response. |
Other fleets with engine_contract |
Per-process attestation digest. | The full attestation response. |
| Otherwise | Digest of the process identity from /version and /metrics. |
The process identity. |
The attested launch digest covers:
- the contract fields
- the engine arguments, with the values of
--host,--port,--served-model-name, and--kv-events-configremoved - the image
- the packages
- the model configuration and revision
- the hashes of the launcher, the launch record, and the cache-capture hook
- the
ucx_versionandpeer_releasevalues from the image check
Preflight, router startup, readmission, and automatic recovery require a fresh profile when an engine's stored measured profile has missing generation evidence or a digest that differs from the engine's live generation. The error names the engine.
If preflight, router startup, or recovery reports profile has no generation evidence:
- Re-profile the named engines against the current engine processes.
- Keep the
.samples.jsonsidecar.
The refit procedure requires a generation_digest in each saved profile and a generation_evidence object in its sample row. Samples missing either need a fresh sweep.
On a fleet with engine_contract, load a fresh profile store by activating it with router resume. On other fleets, restart the router.
A malformed profile stops the operation with an error that names the file, engine, and field:
The profile store must hold the same set of iid values as the configured fleet. Combine rows from separate narwhal-profile --only runs with narwhal-profile --merge.
Profile fields¶
Each profile row has these fields:
| Field | JSON type | Constraint |
|---|---|---|
iid |
string | Nonempty. |
generation_digest |
string | sha256: digest of the attested launch, the verified attestation, or the process identity. |
ttft_a, ttft_b, ttft_c |
number | Nonnegative coefficients of the quadratic prefill fit. |
ttft_block_tokens |
integer, optional | Positive cache block size, set or null together with ttft_split. |
ttft_split |
number, optional | Nonnegative extra prefill time for a prompt that ends inside a block past the first, set or null together with ttft_block_tokens. |
tpot_slope |
number | Nonnegative increase in decode interval per resident KV token. |
tpot_intercept |
number | Nonnegative zero-contention decode interval. |
tpot_request_slope |
number, optional | Nonnegative increase in decode interval per active sequence, default 0. |
kv_capacity_tokens |
integer, optional | Positive, and at least decode_max_kv_tokens. |
decode_min_requests, decode_max_requests |
integer | Positive measured concurrency range with min <= max. |
decode_min_kv_tokens, decode_max_kv_tokens |
integer | Positive measured range of resident KV tokens with min <= max. |
decode_fit_mape, decode_cv_mape |
number | Nonnegative fit error and leave-one-cell-out cross-validation error. |
cached_ttft_a, cached_ttft_b, cached_ttft_c, cached_ttft_d |
number, optional | Nonnegative coefficients of the warm prefill fit. |
cached_cv_mape |
number, optional | Nonnegative leave-one-case-out error of the warm prefill fit. |
cached_min_prefix_tokens, cached_max_prefix_tokens |
integer, optional | Positive measured cached-prefix range with min <= max. |
cached_min_suffix_tokens, cached_max_suffix_tokens |
integer, optional | Positive measured uncached-suffix range with min <= max. |
prefill_min_tokens, prefill_max_tokens |
integer, optional | Positive measured prompt-length range with min <= max. |
decode_min_output_tokens, decode_max_output_tokens |
integer, optional | Positive measured output-length range with min <= max. |
colocated_group |
string, optional | Nonempty shared-device group of a profile measured under one colocated role mix. |
colocated_target_role |
string | prefill or decode, required with colocated_group. |
colocated_prefill_engines, colocated_decode_engines |
integer | Positive engine counts of the measured role mix, at least two engines in total, required with colocated_group. |
colocated_prefill_rps, colocated_decode_rps |
number | Nonnegative measured neighbour load in requests per second, required with colocated_group. |
nominal |
boolean, optional | true for a nominal profile, with generation_digest unset. Default false. |
Nominal profiles¶
A nominal profile states an engine's cost from four figures in place of a sweep. Its row sets nominal to true and these fields:
| Figure | Fields |
|---|---|
| Prefill time per prompt token, positive | ttft_b, with ttft_a and ttft_c at 0 |
| Decode token interval, positive | tpot_intercept, with tpot_slope at 0 |
| KV capacity in tokens | kv_capacity_tokens and decode_max_kv_tokens, with decode_min_kv_tokens at 1 |
| Largest number of concurrent sequences | decode_max_requests, with decode_min_requests at 1 |
Both fit errors are 0.
Router startup, preflight and automatic recovery pass a nominal profile with no generation check, so the profile stays valid across engine restarts. Preflight reports the stated figures in place of the fit errors. A fleet with engine_contract refuses a nominal profile and names the engine.
Profile loading aborts on true, "96" or 1.5 in an integer field, on NaN or Infinity, on a field outside the profile fields table, and on a partial set of cached_ fields.
The router prices a request with a cached prefix from the cached_ fields, following the prefix-cache pricing rules.
Prefill price derived from the profile¶
The router prices a cold prompt of n tokens with the prefill fit:
| Prompt length | Price |
|---|---|
Shorter than prefill_min_tokens |
Fit at prefill_min_tokens |
From prefill_min_tokens to prefill_max_tokens |
Fit at n |
Longer than prefill_max_tokens |
Fit extended to n |
Placement, admission, pool load, demand, and role-split scoring all use this price.
Decode capacity derived from the profile¶
Narwhal limits concurrent decode requests on each fitted engine by the priced context length, context_tokens.
When context_tokens <= 0 or decode_max_requests is null, the decode request limit is zero. Otherwise the limit is the smaller of decode_max_requests and the KV budget divided by context_tokens, with a minimum of 1.
For a profile with kv_capacity_tokens, the KV budget is the smaller of decode_max_kv_tokens and kv_capacity_tokens. Otherwise the KV budget is decode_max_kv_tokens.
To price the token interval for decode request capacity, Narwhal counts at least decode_min_requests requests and at least decode_min_kv_tokens resident KV tokens in the batch.