Configuration reference¶
The fleet control service reads its private configuration from config/fleet-control.local.json, or from the file named by --config or NARWHAL_CONTROL_CONFIG. To create it, follow Setting up the control service.
The service validates the whole file at startup and reports every problem in one message. It rejects unknown keys at every level. Relative paths resolve against the directory where the service starts.
Top-level keys¶
| Key | Default | Meaning |
|---|---|---|
host |
127.0.0.1 |
Listener address. Must be a loopback address or localhost |
port |
8020 |
Listener port, 1 through 65535 |
token_env |
NARWHAL_CONTROL_TOKEN |
Environment variable that holds the bearer token |
runs_dir |
runs/fleet-control |
Directory for the action log and session run records |
fleet |
NARWHAL_FLEET when set |
Baseline fleet configuration file. Required when NARWHAL_FLEET is unset |
hooks |
none; required | Hook commands by name |
router |
see Router | The router whose state, lifecycle and readiness routes the service calls |
prometheus_url |
absent | Prometheus base URL, http or https, that the service asks for firing alerts |
load |
absent | AIPerf settings and the workload library. Without it, load-job routes refuse requests or are absent |
console |
absent | Grafana settings for the console. Without it, the console shows no charts or dashboard links |
The bearer token in token_env must be at least 32 characters long, without leading or trailing whitespace.
Hooks¶
Each entry under hooks names one command:
| Key | Default | Meaning |
|---|---|---|
argv |
none | Command and arguments, a non-empty list of strings |
timeout_s |
900 |
Seconds before the service stops the command |
The service runs these hooks:
| Hook | Runs for | Must |
|---|---|---|
engine_pause |
Engine pause | Pause the engine named by NARWHAL_CONTROL_ENGINE |
engine_resume |
Engine resume, restoring the baseline | Resume that engine |
engine_stop |
Engine stop | Stop that engine |
engine_start |
Engine start, restoring the baseline | Start that engine |
router_restart |
Overlay, restoring the baseline | Restart the router with the fleet configuration at NARWHAL_CONTROL_FLEET |
cold_restart |
Cold restart | Restart every engine and the router with the fleet configuration at NARWHAL_CONTROL_FLEET |
Hook execution¶
A hook succeeds when it exits 0 within its timeout. The service runs each hook in its own process group with standard input closed, and writes the hook's standard output and standard error to a log in the session directory. At the timeout, the service sends SIGTERM to the process group, then SIGKILL 5 seconds later.
Each hook inherits the service's environment, minus the bearer token variable, plus these variables:
| Variable | Set for | Value |
|---|---|---|
NARWHAL_CONTROL_SESSION |
Every hook | Session ID |
NARWHAL_CONTROL_RUN_DIR |
Every hook | Absolute path of the session directory |
NARWHAL_CONTROL_BASELINE |
Every hook | Absolute path of the session's copy of the baseline fleet configuration |
NARWHAL_CONTROL_ACTION |
Engine hooks | pause, resume, stop or start |
NARWHAL_CONTROL_ENGINE |
Engine hooks | Engine ID |
NARWHAL_CONTROL_ENGINE_URL |
Engine hooks | The engine's url from the baseline fleet configuration, or empty |
NARWHAL_CONTROL_FLEET |
router_restart, cold_restart |
Absolute path of the fleet configuration the restarted processes must load |
Hooks run in the service's working directory, so relative paths in argv resolve against it.
Reference engine hook¶
tools/fleet_control/engine_hook.py controls the engine containers that narwhal-engine launches, for every engine backend that runs in a container, and refuses native launches. Use it for all four engine hooks, and pass the hook name without engine_ as the action:
"engine_start": {
"argv": [".venv/bin/python", "-m", "tools.fleet_control.engine_hook",
"--launch-dirs", "config/fleet-control.launch-dirs.json", "start"],
"timeout_s": 900
}
--launch-dirs names a JSON file that maps each engine ID to the absolute path of its launch directory:
{
"n1": "/srv/narwhal/runs/engine-launch-20260101T000000Z-1",
"n2": "/srv/narwhal/runs/engine-launch-20260101T000000Z-2"
}
The hook reads the container ID that narwhal-engine start writes to container.id in the launch directory, then runs docker pause, docker unpause, docker stop or docker start on that container. Keep each engine's launch directory in place while the engine runs.
After docker start, the hook polls the health route of the endpoint in launch.json every 2 seconds. The start succeeds when the route returns 200, and fails if the container stops or the route does not return 200 within --health-timeout seconds (600 by default).
To run a command after the engine answers its health route, such as an attestation sidecar restart, add -- and the command after start. The hook runs the command with NARWHAL_CONTROL_LAUNCH_DIR set to the engine's launch directory. The start fails if the command exits non-zero. Set timeout_s on engine_start to more than --health-timeout plus the command's run time.
The hook exits 2 before it runs Docker when it refuses the request, for example when the engine has no entry in the map or the launch is native. Any later failure exits 1. The hook prints the reason to stderr.
Router¶
| Key | Default | Meaning |
|---|---|---|
router.url |
http://127.0.0.1:8000 |
Router base URL, http or https. AIPerf load jobs also send their requests here |
router.timeout_s |
600 |
Limit for lifecycle calls, for the readiness wait after a restart or restore, and for an engine's return to service after start or resume |
router.journal |
absent | Absolute path of the router's request journal on this host. The file need not exist at startup |
router.journal_max_bytes |
268435456 (256 MiB) |
Largest journal extract copied for one session or load job, a positive number of bytes |
Engine state reads use the shorter of 10 seconds and router.timeout_s.
Set router.journal to the path the router writes: the value of narwhal-serve --journal, or by default journal.jsonl beside profiles.path. Sessions and load jobs record a journal extract only when router.journal is set.
Load¶
| Key | Default | Meaning |
|---|---|---|
load.aiperf |
none | AIPerf executable |
load.model |
none | Served model name that AIPerf requests |
load.tokenizer |
none | Tokenizer name or path for AIPerf |
load.endpoint_type |
chat |
chat or completions |
load.streaming |
true |
Whether AIPerf streams responses |
load.goodput |
{} |
AIPerf metric tag mapped to its positive goodput threshold |
load.grace_period_s |
none | Seconds AIPerf waits for in-flight responses after a timed job ends |
load.extra_args |
[] |
Further arguments appended to the AIPerf command |
load.workloads |
none | The workload library: at least one named workload |
Workloads¶
Workload names use letters, digits, ., _ and -, up to 64 characters. Each workload has a kind:
| Kind | Requests |
|---|---|
synthetic |
AIPerf generates prompts with one input and output length distribution |
mixed |
AIPerf generates prompts in several sizes, each with a share of the requests |
multi_turn |
AIPerf generates conversations. Each turn resends the conversation so far |
public_dataset |
AIPerf sends recorded chat prompts from a public dataset |
timestamped_trace |
AIPerf replays a mooncake_trace JSONL file at its recorded timestamps |
prefix_trace |
AIPerf replays a mooncake_trace JSONL file whose records share prompt prefixes through hash_ids, paced by the job |
Every kind accepts these keys:
| Key | Default | Meaning |
|---|---|---|
label |
the workload name | Text in the console's workload list, at most 60 characters |
description |
none | Text in the console hint, at most 300 characters. Without it, the hint summarizes the workload's settings |
ignore_eos |
false |
When true, each request sets ignore_eos, and the engine generates the full requested output length |
cache_bust |
none | Adds a marker unique to each session at system_prefix, system_suffix, first_turn_prefix or first_turn_suffix. A prefix marker stops prefix-cache reuse between sessions |
cancel_percent |
none | Percentage of requests that the client cancels, above 0 and at most 100 |
cancel_after_s |
0 |
Seconds after sending at which the client cancels a request. Requires cancel_percent |
The other keys depend on the kind:
| Key | Kinds | Default | Meaning |
|---|---|---|---|
isl |
synthetic, multi_turn |
required | Mean input tokens of each request or turn |
osl |
synthetic, multi_turn |
required | Mean output tokens requested by each request or turn |
isl_stddev |
synthetic, multi_turn |
0 |
Standard deviation of the input tokens |
osl_stddev |
synthetic, multi_turn |
0 |
Standard deviation of the output tokens |
mix |
mixed |
required | Request sizes, described below |
turns |
multi_turn |
required | Mean turns per conversation, an integer of at least 2 |
turns_stddev |
multi_turn |
0 |
Standard deviation of the turns per conversation |
turn_delay_s |
multi_turn |
0 |
Mean seconds between turns |
turn_delay_stddev_s |
multi_turn |
0 |
Standard deviation of the seconds between turns |
dataset |
public_dataset |
required | sharegpt |
file |
timestamped_trace, prefix_trace |
required | Path of a mooncake_trace JSONL file |
block_size |
timestamped_trace, prefix_trace |
none | Tokens each hash ID stands for. Required for prefix_trace |
system_prompt_tokens |
synthetic, mixed, multi_turn |
none | Length of one system prompt that every request shares |
prefix_prompts |
synthetic, mixed, multi_turn |
none | Number of shared prefixes. Each request starts with one of them |
prefix_tokens |
synthetic, mixed, multi_turn |
none | Length of each shared prefix |
Each mix entry holds:
| Key | Default | Meaning |
|---|---|---|
isl |
required | Mean input tokens |
osl |
required | Mean output tokens requested |
isl_stddev |
0 |
Standard deviation of the input tokens |
osl_stddev |
0 |
Standard deviation of the output tokens |
percent |
required | Share of the requests, above 0 and at most 100 |
The service checks these rules when it starts:
- The
percentvalues of amixadd up to 100. prefix_promptsandprefix_tokensappear together.- A workload sets
system_prompt_tokensorprefix_prompts, but not both. - With
load.endpoint_typeset tocompletions, the library holds nomulti_turnorpublic_datasetworkload and no workload that setssystem_prompt_tokensorcache_bust. - When a workload sets
ignore_eos,load.extra_argsleaves out--extra-inputs.
Each input and output length must fit the served model's context length.
AIPerf downloads the ShareGPT dataset from Hugging Face on the first public_dataset job and keeps it in .cache/aiperf/ under the service's working directory. Later jobs reuse that copy.
aiperf synthesize agentic-code writes a mooncake_trace file of coding-agent sessions. Use it as a prefix_trace workload with block_size 512.
Set each job's inputs to suit its workload kind, as in Start a job.
Console¶
| Key | Default | Meaning |
|---|---|---|
console.grafana_url |
none; required | Grafana base URL as the operator's browser reaches it, or a path such as / for Grafana on the console's origin. No credentials, query or fragment |
console.dashboard_uid |
narwhal-router |
Dashboard UID for embedded panels and dashboard links |
console.panels |
none | Non-empty list of distinct panel IDs that the standalone console embeds, in display order |
console.from |
now-15m |
Panel time range start, now or now-<n><unit> with unit s, m, h, d, w, M or y |
console.refresh |
5s |
Panel refresh interval, <n><unit> with unit s, m, h or d |
console.embed_in_grafana |
false |
Whether pages from the console.grafana_url origin and the console's own origin may frame the console |
console.auto_connect |
false |
Whether the console page carries the token and connects on load |
console.trusted_hosts |
none | Host names or IPv4 addresses, besides loopback, for which the console page carries the token. Requires console.auto_connect |
console.grafana_url is Grafana's address as your workstation browser sees it, such as http://127.0.0.1:13000. When one proxy serves Grafana and the console, set it to /.
The example configuration embeds the Requests, Latency, Time to first token, Time per output token, Engine role history and Fleet events panels of the shipped dashboard. If those panels show as empty frames, follow Charts in the standalone console are empty frames.
Environment variables¶
| Variable | Read by | Meaning |
|---|---|---|
NARWHAL_CONTROL_CONFIG |
The service | Configuration file when --config is not given |
NARWHAL_FLEET |
The service | Baseline fleet configuration file when fleet is unset |
NARWHAL_CONTROL_TOKEN |
The service, make observe |
Bearer token. The service reads the variable that token_env names; make observe always reads NARWHAL_CONTROL_TOKEN |
NARWHAL_CONTROL_CONSOLE_URL |
make observe |
Console address that the dashboard's Fleet control link opens, when it differs from http://127.0.0.1:18020/console |
NARWHAL_CONTROL_METRICS_URL |
make observe |
Control service address as Prometheus reaches it, for the dashboard annotations |