Narwhal dev¶
Narwhal dev runs two to eight independent engines and a router on one workstation, on Ubuntu natively or under WSL2. The quick start runs llama.cpp engines on a GPU or the CPU. This page covers vLLM and SGLang engines on one NVIDIA CUDA GPU, with engine attestation, measured profiles and KV transfer evidence. An instance directory, runs/dev by default, holds the model choice, runtime, GPU allocation, ports, profiles, and process identities. Model files come from the Hugging Face cache or from paths given to narwhal dev init.
A template selects the backend, model, runtime pins, engine count, and memory budget. Narwhal installs three for vLLM and SGLang, and two for llama.cpp engines:
| Template | Backend and KV transfer | Model | Engines |
|---|---|---|---|
small-cuda-v1.json, the default |
vLLM 0.30.0 with NIXL over UCX | Qwen3.5-0.8B Q4_K_M GGUF | 2 |
small-cuda-sglang-v1.json |
SGLang 0.5.21 with Mooncake over TCP | Qwen3.5-0.8B checkpoint in bfloat16 | 2 |
reference-v1.json |
vLLM 0.30.0 with NIXL over UCX | Qwen3.5-0.8B Q4_K_M GGUF | 4 |
The two small templates start one prefill and one decode engine on an NVIDIA GPU. The vLLM template needs about 7,680 MiB of free VRAM and the SGLang template about 10,240 MiB. The four-engine reference template pins the RTX 5090.
Preparing Ubuntu or WSL2¶
-
Install the NVIDIA driver for your host:
- On native Ubuntu, install a driver compatible with the CUDA runtime you plan to use.
- On WSL2, install the Windows driver from NVIDIA's CUDA on WSL guide.
-
Keep the checkout, the model, the virtual environment, and the instance directory on the Linux filesystem.
- Run every command from an Ubuntu shell, inside WSL2 on a Windows host.
-
Show the GPU and the IPv4 interfaces:
-
Record the name of the interface for KV transfer,
eth0by default. Pick one with exactly one IPv4 address that is not a bridge such asdocker0. On a bridge, vLLM engines fail to start withNIXL_ERR_BACKEND.
Installing the runtime and model¶
For the default vLLM template, install the pinned vLLM, Torch, NIXL, Transformers, GGUF loader, model, and tokenizer with the CUDA runtime and model steps. For the SGLang template, follow Running SGLang engines.
Template and GPU allocation¶
A template states the GPU memory of its engines in MiB or as fractions of the GPU.
The two small templates state it in MiB: allocation.engine_memory_mib is 2867 for each engine, and allocation.device_memory_mib bounds all engines together and whole-device memory growth during startup. It is 7168 in the vLLM template, whose engines each take about 450 MiB in addition to engine_memory_mib, and 9728 in the SGLang template, whose engines each take about 1,700 MiB for CUDA graphs and runtime in addition to it. init converts both to fractions of the GPU it finds, rounded up to four places, and writes them beside the MiB values in the instance's template.json. For the vLLM template the fractions are 0.35 and 0.875 on a GPU of 8 GiB, and 0.088 and 0.2199 on a GPU of 32 GiB. --engine-memory-mib and --device-memory-mib override the two values, and the device memory must cover the engines' memory.
The reference template states fractions: allocation.gpu_memory_utilization for each engine and allocation.device_allowance for all of them. --gpu-memory-utilization and --device-allowance set them. On a template that states MiB, pass both fraction options to size the engines as fractions, and on the reference template pass both MiB options to size them in MiB.
A vLLM engine receives its fraction as --gpu-memory-utilization, the share of total VRAM for the process, covering model weights, runtime overhead, and KV cache. An SGLang engine receives it as --mem-fraction-static, which SGLang applies to the VRAM that is free when the engine starts, so an engine on a GPU with other memory in use receives less than its MiB value, and the second engine receives a smaller budget than the first.
Under WSL2, every SGLang engine receives the first engine's budget, so the engines can overcommit the GPU. Initialize a new --instance with a lower --engine-memory-mib until every engine starts.
init requires free VRAM of the fleet's allowance plus the template's gpu.reserve_mib: about 7,680 MiB for the small vLLM template and about 10,240 MiB for the small SGLang template. During up, each engine starts only when free VRAM covers its fraction of total VRAM, and startup fails when whole-device memory use grows by more than the device allowance times total VRAM.
The small templates set development SLO targets of 5 seconds for time to first token (TTFT) in slo.ttft_s and 500 ms for time per output token (TPOT) in slo.tpot_s. Replace these placeholder targets with values measured on your card.
A custom --template sets the model, runtime, context length, profiling sweep, and reserve. Its gpu.product field pins the GPU product name, and gpu.minimum_total_mib pins the minimum total VRAM. An SGLang template with the mooncake connector also sets runtime.decode_cuda_graph_memory_gb, the memory a prefill-launched engine reserves when it switches to decode.
runtime.max_num_seqs caps each engine's running requests. An SGLang template accepts 1 to 256 and defaults to 256. If up stops at attestation with The engine serves at most <n> running requests, set runtime.max_num_seqs to <n> and initialize a new --instance from the template.
Create a custom template:
-
Export an installed template:
-
Edit the exported file.
- Initialize a fresh instance with
narwhal dev init --template runs/small-cuda-template.json --instance runs/dev-custom.
Initializing and verifying an instance¶
Run the lifecycle commands with the interface name from ip:
interface=eth0 # replace with the interface reported by ip
narwhal dev init --interface "$interface"
narwhal dev up
narwhal dev verify
narwhal dev status
init runs these steps:
- Checks the model and tokenizer checksums.
- Selects a CUDA GPU by UUID and checks its free VRAM.
- Checks the runtime package versions, imports the engine runtime, and checks the GGUF plugin hashes that a vLLM template pins.
- Checks the network interface and ports.
- Writes the private instance.
up runs these steps:
- Checks the model and tokenizer hashes, and the GGUF plugin hashes that a vLLM template pins.
- Checks that the ports are free.
- Checks and starts the engines.
- Captures attestations and starts the attestation sidecars.
- Profiles the engines. A template with neighbour load settings in
profile, such as the vLLM templates, profiles every role split under that load. A template without them, such as the SGLang template, profiles only the opening split. - Starts the router.
verify runs these steps:
- Runs preflight over every eligible directed KV path.
- Sends a routed arithmetic request.
- Saves the router and engine metrics.
- Checks router readiness, and reports
readywhen every check passes.
Run every later lifecycle command in the Python environment that ran init.
Each command writes the lifecycle result to stdout as one JSON document, and preparation, profiling, and error diagnostics to stderr.
For scripts, --format json returns versioned command results and automation exit codes.
Read the router's state and metrics at its default address:
The WSL2 monitoring setup forwards these metrics to Prometheus and Grafana on a separate host.
--port-base on narwhal dev init selects a different port layout, and --instance on any command targets another instance.
Running SGLang engines¶
The installed small-cuda-sglang-v1.json template runs one prefill and one decode SGLang engine with the Qwen3.5-0.8B checkpoint, each running at most 4 requests at once. The engines transfer KV through Mooncake over TCP. The checkpoint mixes linear-attention and full-attention layers, so the template launches the engines with --mamba-radix-cache-strategy no_buffer and --disable-overlap-schedule, as SGLang runtime fields describes.
-
Install the system libraries that Mooncake needs:
-
Install a CUDA 13 toolkit from NVIDIA's CUDA installation guide for Linux, or from the CUDA on WSL guide under WSL2. Export the toolkit root as
CUDA_HOMEin each shell that runs the lifecycle commands, and check thatnvccreports release 13 andlib64/libcudart.soexists: -
Install Narwhal and the pinned packages in a fresh virtual environment:
-
Download the checkpoint into the Hugging Face cache:
-
Export the template and start an instance from it:
mkdir -p runs python - <<'PYTHON' > runs/small-cuda-sglang-template.json from importlib.resources import files print(files('narwhal.dev').joinpath('small-cuda-sglang-v1.json').read_text()) PYTHON narwhal dev init --template runs/small-cuda-sglang-template.json \ --instance runs/dev-sglang --interface "$interface" narwhal dev up --instance runs/dev-sglang narwhal dev verify --instance runs/dev-sglangUnder WSL2, when an engine fails to start, lower
--engine-memory-mibas Template and GPU allocation describes.
Inspecting and stopping the instance¶
The run directory that status prints holds:
| Content | Path |
|---|---|
| Routed response | verify-*/completion.json |
| Whole-device memory samples | up-memory.jsonl, verify-memory.jsonl |
| Request journal | journal.jsonl |
| Teardown record | teardown.json |
| Model loading and engine requests | engine-*/startup.log |
| Probe and fit outcomes | profile-*.log |
| Runtime, profile, and transfer checks | verify-*/preflight.log |
Stop the instance:
down reports stopped for the recorded process groups.
Each up writes a new run directory.
For a runtime change or engine restart:
- Run
down. - Run
up. - Run
verify.
Stage deadlines and recovery¶
dev up and dev verify run subprocess stages with time budgets. NARWHAL_STAGE_TIMEOUT_SECONDS sets the budget for every stage.
NARWHAL_STAGE_<NAME>_TIMEOUT_SECONDS sets the budget for one stage. <NAME> is the stage name in uppercase, with hyphens replaced by underscores.
Stages and default budgets:
| Stage | Work | Default budget |
|---|---|---|
engine-<n> |
Runtime check for engine n |
300 seconds |
native-start-shared |
Native shared startup | 180 seconds per engine |
attest-<n> |
Attestation capture for engine n |
300 seconds |
profile-<p>p<d>d |
Profiling a role split with p prefill and d decode engines |
300 seconds |
profile-merge |
Merging the role-split profiles, when up profiles more than one split |
300 seconds |
preflight |
Directed KV preflight during verify |
300 seconds |
Override the startup and preflight budgets:
NARWHAL_STAGE_NATIVE_START_SHARED_TIMEOUT_SECONDS=900 narwhal dev up
NARWHAL_STAGE_PREFLIGHT_TIMEOUT_SECONDS=120 narwhal dev verify
A stage budget covers the stage's work, helper imports, and subprocess startup. Each profiled role split gets a fresh stage budget.
Native shared startup allows 180 seconds for each engine health check. Each HTTP health request times out after 2 seconds, and the routed verification request times out after 30 seconds.
narwhal dev stops the stage's supervised processes when:
- a stage budget expires
narwhal devreceives SIGINT (Ctrl-C) during a stagenarwhal devreceives SIGTERM during a stage
Cleanup runs in two steps:
narwhal devsends SIGTERM and waits up toNARWHAL_STAGE_CLEANUP_GRACE_SECONDS, 10 seconds by default.narwhal devsends SIGKILL to surviving processes and waits up toNARWHAL_STAGE_KILL_GRACE_SECONDS, 5 seconds by default.
Both grace periods add to the execution budget.
Files next to each stage log:
| File | Content |
|---|---|
*.command.json |
The stage's command line |
*.stdout, *.stderr |
The stage's private output |
*.stage.json |
The budget, wall-clock start, elapsed time, exit status, process identities, cleanup escalation, and surviving PIDs |
When up fails, startup rolls back and lifecycle.json records the original failure and teardown errors.
When verify fails, the attempt directory stays. status reports degraded with the reason until a later verify succeeds or down completes.
Recover from a failed stage, with PATH as the instance directory:
- Read the failed stage's output.
- Run
narwhal dev status --instance PATH. - Stop the process generation with
narwhal dev down --instance PATH. - Run
narwhal dev up --instance PATH.
If status or down lists surviving PIDs, inspect each one by hand against its recorded boot ID, start tick, and process group.
Contributing another GPU recipe¶
Contribute a template for a new small CUDA GPU:
- Follow the checks and pull request workflow.
- Submit the working template with qualification evidence.
The qualification evidence records:
- the GPU, driver, model, and runtime versions
- the free-memory reserve and peak startup memory
- both directed KV transfers
- the routed completion
A contribution for a GPU from another vendor includes:
- a measured template
- optionally, discovery, memory accounting, engine launch, and a compatible transfer runtime