Quick start on one workstation¶
narwhal dev up downloads a llama.cpp engine build and a small model, then starts four engines and a router on one machine. The engines run on an NVIDIA GPU, or on the CPU when the machine has none. The fleet hands each request's KV cache from a prefill engine to a decode engine and moves engines between the two roles as the load changes.
Requirements¶
| ERNIE-4.5-0.3B on the CPU | ERNIE-4.5-0.3B on a GPU | Qwen3.5-0.8B on the CPU | Qwen3.5-0.8B on a GPU | |
|---|---|---|---|---|
| Download | 0.3 GB | 1.0 GB | 0.6 GB | 1.3 GB |
| Disk after setup | 0.3 GB | 2.1 GB | 0.6 GB | 2.3 GB |
| Memory four engines use | 1.5 GB of host memory | 4.4 GB of GPU memory | 2.4 GB of host memory | 5.5 GB of GPU memory |
Available host memory setup requires for four engines |
1,800 MiB | No check | 2,800 MiB | No check |
setup, downloads included, on a fast connection |
4 s | 16 s | 5 s | 8 to 26 s |
First up after setup |
2 s | 24 s | 3 s | 62 s |
Later up |
2 s | 4 s | 3 s | 5 s |
ERNIE-4.5-0.3B is the default model. Every figure was measured on Linux hosts, the GPU columns on an NVIDIA GPU with the CUDA 12 build of the engine. The steps on this page also ran end to end under WSL2, on the CPU and on an NVIDIA GPU. Both model files are 4-bit GGUF quantizations. While the fleet serves, each run keeps handed-over KV state in files under the instance directory, a few megabytes for ERNIE-4.5-0.3B and up to 0.5 GB for Qwen3.5-0.8B.
The host needs:
- Ubuntu or a comparable Linux distribution on x86-64, or Windows with WSL2 and an Ubuntu distribution
- Python 3.11.4 or newer with
venv - For a GPU, an NVIDIA driver that supports CUDA 12.
setupthen downloads the CUDA 12 build of the engine together with its runtime libraries.
A request's prefill computes the KV cache of its prompt, and its decode generates the reply's tokens from that cache. Narwhal runs the two on different engines.
Installing¶
Run every command in a Linux shell. On Windows, use a WSL2 Ubuntu shell.
Run every later command from this directory with the virtual environment active. The instance directory runs/dev is relative to it.
Starting and removing in one command each¶
On its first run, up sets up the instance in runs/dev and then starts the fleet. It takes the options of setup, so narwhal dev up --engine-count 8 changes a stopped fleet to eight engines and starts it.
down --remove stops the fleet and deletes runs/dev with its downloads and logs. python -m pip uninstall narwhal-inference then removes Narwhal itself.
The sections below take the same steps one at a time.
Setting up the instance¶
setup writes the instance to runs/dev. It downloads the engine build and the model file into runs/dev/assets, checks each file's SHA-256 against the value the template pins, and checks the host:
{
"status": "initialized",
"instance": "/home/you/runs/dev",
"model": "ERNIE-4.5-0.3B",
"engine_build": "none",
"downloaded": [
"llama-b12365-bin-ubuntu-x64.tar.gz",
"ERNIE-4.5-0.3B-PT-Q4_K_M.gguf"
]
}
engine_build is cuda12 when setup finds an NVIDIA driver, and none for the CPU build.
| To change | Pass |
|---|---|
| The model | --model-name Qwen3.5-0.8B |
| The engine build | --accelerator none for the CPU, or --accelerator cuda for the CUDA 13.5 build, for a driver with CUDA 13.5 support |
| The number of engines, two to eight | --engine-count 2 |
| The ports, when 18000 or 18101 to 18104 are taken | --port-base 19000 |
A second setup on the same directory with the same settings reports reused and downloads nothing.
Changing a setting¶
To change the number of engines, the model, the engine build or the ports, stop the fleet and start it with the option. The instance keeps the files it has downloaded and downloads a file only when the new setting needs one. An option left out keeps its value. setup with the same option changes the instance without starting it, and reports reconfigured.
One fleet runs at a time. up on a second instance directory while a fleet is up reports a fleet is already up in … and starts nothing, because a second fleet takes its own share of memory beside the first. narwhal dev up --alongside starts it anyway.
Starting the fleet¶
up starts the engines, sends each one short request to measure its speed, and starts the router at http://127.0.0.1:18000. Half of the engines, rounded down, start in the prefill role and the rest in the decode role.
{
"status": "launched",
"run": "/home/you/runs/dev/run-03020490194f",
"router": "http://127.0.0.1:18000"
}
Sending a prompt¶
{
"status": "answered",
"reply": "The capital of France is Paris. \n\nFrance is known for its rich culture, historical landmarks, and vibrant culture. Paris is the economic, cultural, and political center of France.",
"ttft_s": 0.033,
"output_tokens": 41,
"tokens_per_second": 292.2,
"engines": {
"prefill": "n1",
"decode": "n3"
}
}
engines names the engine that computed the prompt and the engine that generated the reply from the handed-over KV cache. Both models have fewer than one billion parameters. Their replies are fluent, and their answers to simple factual questions are sometimes wrong.
The router serves the OpenAI completion and chat routes, so most OpenAI clients that use those routes work against it:
curl -s http://127.0.0.1:18000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model": "ERNIE-4.5-0.3B", "messages": [{"role": "user", "content": "Say hello."}]}'
For Qwen3.5-0.8B, add "chat_template_kwargs": {"enable_thinking": false} to the body, as ask does, to turn that model's reasoning output off.
Sending many prompts and watching the fleet¶
In a second shell, change to the same directory, activate the same virtual environment and open the live view, which runs until you press Ctrl+C:
In the first shell, send 2,000 built-in prompts with eight in flight:
The live view shows each engine's role and requests, the latency and the throughput. Long replies load the decode engines. In most runs on the measured hosts, the router moved one prefill engine to the decode role within 30 seconds. Reading the live view explains each figure.
When the prompts finish, ask prints their latency spread and throughput:
{
"status": "completed",
"requests": 2000,
"failed": 0,
"concurrency": 8,
"seconds": 91.2,
"ttft_s": {"p50": 0.074, "p95": 0.485},
"tpot_s": {"p50": 0.005, "p95": 0.006},
"requests_per_second": 21.9,
"output_tokens_per_second": 876.6,
"replies_without_token_count": 0
}
This output is from the measured CPU host, where the run took 91 seconds. On the measured GPU host it took 78 seconds.
Stopping¶
down stops the engines and the router. The instance directory keeps the downloaded files and the last run's run-… directory with engine-N.log, router.log, the request journal and the slot files. The next up removes the runs before it. narwhal dev up starts the fleet again. narwhal dev down --remove also deletes runs/dev.
Troubleshooting¶
| Message | Action |
|---|---|
port 18000 is unavailable on 127.0.0.1 from setup |
Repeat setup with --port-base and a free port that has four free ports after it. The downloaded files are kept. |
host memory check failed |
Repeat setup with --engine-count 2. A Qwen3.5-0.8B instance can instead be set up in a new --instance directory with the default model. |
holds other files; name an empty or new instance directory |
Pass --instance with a new directory. |
SHA-256 … differs from the pinned |
Run setup again. It removed the file and downloads it once more. |
process exited before http://127.0.0.1:18101/health became healthy during up |
Run narwhal dev status and read engine-N.log in the directory its run field names. N is the port in the message less 18100, so 18101 is engine-1.log. When the log reports CUDA error, as the cuda build does on a driver without CUDA 13.5 support, run narwhal dev down, then narwhal dev setup --accelerator cuda12 or --accelerator none. |
a fleet is already up in … during up |
Stop that fleet with the narwhal dev down --instance … command the message names, then repeat up. |
run narwhal dev down … before changing the instance's settings |
Run narwhal dev down, then repeat the setup command. |
no instance here; run narwhal dev up or narwhal dev init |
Run narwhal dev up, or change to the directory where you ran it. |
the fleet is already up during up |
The fleet is running. Use it, or run narwhal dev down to stop it. |
run this instance with the Python environment used by dev init |
Activate the virtual environment that ran setup. |
Differences from a deployed fleet¶
The quick-start fleet gives each engine a profile from a few short requests. Its admission is open, so the router queues a request that the predictive admission of a deployed fleet would refuse with HTTP 429 when its predicted latency exceeds the targets. It has no engine attestation, profiling sweep or KV transfer evidence, which the deployment stages produce. llama.cpp engines describes the engine settings, the KV handoff and the metrics.