narwhal dev¶
narwhal dev runs a development fleet on one workstation, on Linux natively or under WSL2. With setup, the fleet holds llama.cpp engines and a router, on a GPU or on the CPU, and the quick start covers that path. With init, the fleet holds native vLLM or SGLang engines on one NVIDIA CUDA GPU, their attestation sidecars and a router. To set up the host for those, follow Narwhal dev. A template pins the model, the runtime, the memory budget and the ports. init takes one of three installed templates, and setup one of the two llama.cpp templates:
| Template | Engine | Engines | Model input |
|---|---|---|---|
small-cuda-v1.json, the default |
vLLM | 1 prefill, 1 decode | GGUF file |
small-cuda-sglang-v1.json |
SGLang with Mooncake | 1 prefill, 1 decode | Checkpoint directory |
reference-v1.json, for an RTX 5090 |
vLLM | 2 prefill, 2 decode | GGUF file |
direct-ernie45-03b-v1.json, the default of setup |
llama.cpp | 2 prefill, 2 decode | GGUF file that setup downloads |
direct-qwen35-08b-v1.json |
llama.cpp | 2 prefill, 2 decode | GGUF file that setup downloads |
Lifecycle¶
Run the subcommands with the Python environment that ran setup or init:
narwhal dev setup
narwhal dev up
narwhal dev verify
narwhal dev ask "What is the capital of France?"
narwhal dev watch
narwhal dev status
narwhal dev down
For vLLM or SGLang engines, narwhal dev init takes the place of setup.
| Subcommand | Operation | Reports |
|---|---|---|
setup |
Downloads the engine build and model file that a template pins into assets under the instance directory, checks their SHA-256 values, and runs the init checks for a template that starts its engines directly. On a stopped instance, an option that differs from the instance changes it in place and keeps the downloaded files. |
initialized, reconfigured, or reused |
init |
Checks the model files, runtime packages, GPU, network interface and ports, then writes template.json, instance.json, engine-launch.json and fleet.json to the instance directory. |
initialized, or reused |
up |
Prepares, checks and starts the engines with narwhal-engine, captures their attestations, starts the attestation sidecars, profiles the engines and starts the router. For llama.cpp engines, starts each engine process, writes it a nominal profile from concurrent short requests to every engine and starts the router. While another instance's fleet is up, up starts nothing and names that instance; --alongside starts the fleet beside it. When the instance does not exist, or a setup option is given, up runs setup with those options first; an instance from init takes no setup option. |
launched |
verify |
Records KV evidence for every directed pair, sends a routed 2 + 3 request that must return 5, saves router and engine metrics, and checks the router's /ready. For llama.cpp engines, sends one routed completion that must return text and checks /ready. |
ready |
ask |
Takes one prompt as its argument, after -- when the prompt starts with a hyphen. Sends it through the router and prints the reply, its time to first token, its tokens per second and the engine that served each leg. With --repeat, sends that many requests, of the given prompt or of the built-in prompts when none is given, and prints the time to first token and the time per output token at the median and the 95th percentile, the throughput and the count of failed requests. |
answered, or completed |
watch |
Redraws, from the router's /narwhal/state and /metrics, each engine's role and requests in flight, the time to first token and the time per output token at the median and the 95th percentile, the throughput and the role switches. The times are the router's: arrival to the end of prefill, and the mean time per output token after it, so they differ from the client's view that ask reports. Output tokens are counted when a request completes. It runs until interrupted and prints the view in place of a result document. With --once or --format json it takes two readings one interval apart and prints that frame as its result. |
observed |
status |
Reads the recorded processes, health routes and saved evidence. | A state from the next table |
down |
Sends SIGTERM to each recorded process group and SIGKILL to members still running 10 seconds later. With --remove, then deletes the instance directory with its downloads and logs. |
stopped, or removed |
status reports one of these states:
| State | Condition |
|---|---|
starting |
up is running. |
launched |
Every engine, sidecar and router process is running, every engine and the router answer /health, and the router answers /ready. |
ready |
The launched conditions hold, and the KV evidence of the last verify still verifies against the running engines. |
degraded |
problems lists a failed condition, a surviving process, or a failed verify. |
stopped |
The last generation stopped and none of its processes survive, or up has never run. |
up requires an instance that has never started or whose last generation stopped, through down or the rollback of a failed up. When the template's profile sets neighbour traffic, as the vLLM templates do, up profiles every role split from 1 prefill engine to one less than the engine count with --colocated, and merges the splits when there are several. Otherwise, as with the SGLang template, up profiles only the opening split, where the SGLang pair sweep measures both roles of each engine.
Running init on an existing instance changes nothing. When the settings you supply match the saved instance, init reports reused. When a setting conflicts, init exits with code 2 and lists the conflicting options. To change a setting, run init with a new --instance directory.
To authenticate engine requests, export NARWHAL_ENGINE_API_KEY before up and keep it set for later commands. up passes it to the engines, and sets engine.engine_api_key_env in the fleet configuration so that the profiler, preflight and router send it.
A failed verify writes its reason, evidence directory and failure time to verification_failure in lifecycle.json and to failure.json in its evidence directory. status reports degraded with that verification_failure until a later verify succeeds or down stops every process.
Output and exit codes¶
In text mode, stdout carries the result as one JSON document, and stderr carries preparation and profiling progress and error diagnostics. To save the result, redirect stdout, as in narwhal dev up > result.json.
| Outcome | Exit code |
|---|---|
initialized, reconfigured, reused, starting, launched, ready, stopped, removed, observed, answered, or completed with every request served |
0 |
degraded, a failed up, verify or down, a failed ask request, or completed with a failed request |
1 |
Invalid ask or watch arguments, a failed setup download or check, a failed init check, an unreadable instance or lifecycle record, a missing runtime package, or a Python environment other than the one that ran setup or init |
2 |
With --format json, the command prints a versioned command result, and the exit code follows its status. A degraded instance exits with code 3, a failed step or routed check during up or verify and a failed single ask request exit with code 4, and an ask --repeat run with a failed request exits with code 1.
Options¶
--instance and --format apply to every subcommand. --model-name and --accelerator apply to setup and up, --repeat, --concurrency and --max-tokens to ask, --alongside to up, --remove to down, --interval and --once to watch, --engine-count and --port-base to setup, up and init, and the other options to init. An omitted setup or init option takes its value from the saved instance or the template. narwhal --version prints the installed distribution version.
| Option | Default | Description | Valid values |
|---|---|---|---|
--instance |
runs/dev |
Private instance directory. | |
--format |
text |
Output format, either text or json for versioned command results. |
|
--template |
small-cuda-v1.json |
Template file with versioned model, tokenizer, runtime, profiling and memory settings. | |
--model |
The template's pinned file or snapshot in the Hugging Face cache | Model to serve, a GGUF file for a vLLM template or a checkpoint directory for an SGLang template. | Matches the template's checksums, and for a GGUF file its file name |
--model-dir |
The template's pinned tokenizer snapshot in the Hugging Face cache | Directory with the tokenizer files and config.json. |
Matches the template's checksums |
--gpu |
The only GPU that nvidia-smi lists |
Physical GPU UUID. | |
--alongside |
Off | Starts the fleet while another instance's fleet is up. Each fleet takes its own memory. | Flag |
--remove |
Off | Deletes the instance directory after down stops its fleet. |
Flag |
--engine-count |
Template value, 2 in the small templates and 4 in the llama.cpp templates |
Number of engines. | 2 to 8 |
--port-base |
Template ports | Base of the port layout below. | Every port within 1 to 65535 and distinct |
--engine-memory-mib |
Template value, 2867 in the small templates |
Each engine's GPU memory in MiB, which init converts to a fraction of the GPU it finds. On SGLang it is an upper bound, since the engine applies the fraction to the memory free at its start. |
A positive integer |
--device-memory-mib |
Template value, 7168 in the small vLLM template and 9728 in the small SGLang template |
GPU memory in MiB that bounds the sum of the engines' memory and the memory growth during startup. | At least --engine-count times --engine-memory-mib, and at most the GPU's total |
--gpu-memory-utilization |
Template value, 0.1 in the reference template |
Each engine's fraction of total GPU memory on vLLM, or of the GPU memory free at engine start on SGLang. Pass it with --device-allowance on a template that states MiB. |
Above 0 and at most 1 |
--device-allowance |
Template value, 0.5 in the reference template |
Fraction of total GPU memory that bounds the sum of engine fractions and the memory growth during startup. | At least --engine-count times --gpu-memory-utilization, and at most 1 |
--interface |
eth0 |
Network interface for KV transfer. | An interface with exactly one IPv4 address |
--engine-binary |
None | Engine server executable, for a template that starts its engines directly. | An executable file of the build the template pins |
--no-accelerator |
Off | Keeps the model on the CPU, for a template that starts its engines directly. | A flag with no value |
--accelerator |
auto |
Engine build that setup downloads. auto selects by the host's devices. |
auto, none for the CPU, or a build the template pins |
--interval |
1.0 |
Seconds between redraws of watch. |
A positive number |
--once |
Off | Makes watch print one frame as its result and exit. |
A flag with no value |
--repeat |
None | Number of requests ask sends, of the given prompt or of the built-in prompts. |
A positive integer |
--concurrency |
1 |
Requests ask keeps in flight with --repeat. |
A positive integer |
--max-tokens |
128 |
Output token limit of each ask request. |
A positive integer |
--model-name |
The default template's model | Served name of the model setup prepares. |
A model that an installed template starting its engines directly serves |
--port-base P puts the router at P and starts the engine HTTP, attestation and KV side-channel ranges at P+1, P+101 and P+201, one port per engine. A template that starts its engines directly uses the router port and the engine HTTP range alone. The KV side channel is the NIXL side channel on vLLM and the bootstrap server on SGLang. The installed templates put the router at 18000 and start the ranges at 18101, 18201 and 5701. init and up require every port to be free on 127.0.0.1.
init also checks the GPU:
| Check | Requirement |
|---|---|
| Free memory | Free VRAM covers the fleet's allowance plus the template's gpu.reserve_mib: about 7,680 MiB for the small vLLM template and about 10,240 MiB for the small SGLang template, and --device-allowance times total VRAM plus 2,048 MiB for the reference template. |
| GPU product | The GPU name matches the template's gpu.product when the template sets it. |
| Total memory | Total VRAM is at least the template's gpu.minimum_total_mib when the template sets it. |
To serve another model or runtime, write a custom template and select it with --template.
Role-split replay¶
The RTX 5090 reference template's role_cycle holds three seeded workloads that drive the four-engine fleet's controller from 2P:2D to 1P:3D, back to 2P:2D, to 3P:1D and back to 2P:2D. To replay them, run this from a checkout against an instance that reports ready:
The replay writes its records to a new cycle-* directory in the instance, or to --out. Its summary.json sets passed when all of these hold:
- The controller's reactive moves produce exactly the expected splits on the same engine processes.
- Each workload runs on schedule with every request completed or rejected for its TTFT.
- Each workload meets the SLO where the recipe requires it.
The command exits with code 0 when the replay passes and 2 when it fails.
Run directories and recovery¶
Each up writes a run-* directory in the instance. It holds:
- the router's fleet configuration with the engine contract
- the command line and log of each stage
- one launch directory per engine
- the profiles
- whole-device VRAM samples in
up-memory.jsonlandverify-memory.jsonl - the request journal
On vLLM, each launch directory also holds the engine's cache-layout.json.
Each verify writes a verify-* directory in the run directory with the preflight log, the KV evidence in kv.json, the routed response in completion.json, and the router and engine metrics.
When up fails, it stops the processes it started and records the failure in lifecycle.json. To recover:
- Read the log of the stage that the error names.
- Repair the cause.
- Run
narwhal dev downand confirm that it reportsstopped. - Run
narwhal dev up.
To find a stage's time budget or handle processes that survive down, follow Stage deadlines and recovery.