llama.cpp engines¶
The llamacpp backend routes requests over llama-server processes. narwhal dev starts them directly from a template with "start": "direct", and the quick start covers the commands. The backend serves development fleets on one host. A deployed fleet uses vLLM or SGLang.
Templates¶
Narwhal installs two templates that start llama.cpp engines:
| Template | Model | Engines | Host memory setup and init require for each engine on the CPU |
|---|---|---|---|
direct-ernie45-03b-v1.json, the default of setup |
ERNIE-4.5-0.3B Q4_K_M GGUF | 4 | 450 MiB |
direct-qwen35-08b-v1.json |
Qwen3.5-0.8B Q4_K_M GGUF | 4 | 700 MiB |
narwhal dev setup --model-name selects one by the model it serves. narwhal dev init --template PATH --engine-binary PATH --model PATH initializes an instance from files already on the host.
| Field | Meaning |
|---|---|
schema, schema_version |
narwhal.dev-template and 1. |
start |
direct for engines that start as plain processes. |
setup_default |
true on the template setup uses when --model-name is omitted. |
model.repository, model.revision, model.filename, model.sha256 |
The model file setup downloads and the checksum setup, init and up require. |
model.served_name |
The model name the router and the engines serve. |
engine.url, engine.binary |
The download URL pattern of an engine archive, with {name} for the archive's name, and the server's file name inside it. |
engine.builds |
The archives and checksums of each engine build by machine type. The keys are the values of --accelerator, and none is the CPU build. detect lists the paths whose presence selects the build under --accelerator auto. The installed templates pin none, cuda12 for CUDA 12.8 and cuda for CUDA 13.5, and auto selects cuda12 on a host with an NVIDIA driver. |
runtime.backend |
llamacpp. |
runtime.build |
The llama.cpp build number the server must report. |
runtime.slots |
Requests one engine serves at once. |
runtime.context_tokens |
Context tokens of each slot. |
runtime.memory_mib |
Available host memory required for each engine by setup with the none build and by init with --no-accelerator. |
runtime.extra_args |
Further llama-server arguments. |
request |
Optional fields narwhal dev ask adds to each request body. The Qwen3.5-0.8B template turns that model's reasoning output off. |
allocation.engine_count, ports, slo, controller |
The engine count, the router and first engine port, and the router's SLO and role-control settings. |
serving |
Optional serving settings of the fleet. The installed templates set admission to open. |
Engine command line¶
up starts each engine with these llama-server arguments:
| Argument | Value |
|---|---|
--model, --alias |
The model file and model.served_name |
--host, --port |
127.0.0.1 and the engine's port |
--parallel |
runtime.slots |
--ctx-size |
runtime.context_tokens times runtime.slots, which the server divides among its slots |
--threads |
The host's usable CPUs less one, divided among the engines and rounded down, from 1 to 8 |
--n-gpu-layers |
all, or 0 with --no-accelerator or the CPU build |
--cache-ram |
0, which turns off the server's host prompt cache |
--slot-save-path |
The run's slot-files directory, shared by every engine |
--metrics, --slots, --no-webui |
Set |
The server's shared libraries load from the directory of the engine binary.
KV handoff¶
A request's prefill and decode legs each take one server slot through id_slot and wait while every slot is taken. After the prefill reply, the router saves the producer's slot to a file with POST /slots/<id>?action=save. Before the decode stream opens, it restores that file into the consumer's slot with action=restore, and the consumer computes any prompt tokens the restored state lacks and generates the reply.
A producer keeps as many slot files as it has had handoffs in flight at once and reuses their names. down leaves them in the run directory.
One llama-server process serves both roles, so a role switch changes the router's placement and sends the engine nothing.
Profiles¶
up writes each engine a nominal profile. It sends every engine runtime.slots requests of about a hundred prompt tokens and sixteen output tokens concurrently, so each engine is measured while its neighbours on the shared device serve. An engine's profile holds the mean prefill time per token and the mean token interval of its replies, the context of all its slots as KV capacity, and runtime.slots as the request limit.
Token counts and metrics¶
The router counts a prompt with the engine's /tokenize, after rendering chat messages with /apply-template. Stream chunks carry text without token IDs, so the router reads each reply's output token count from the last chunk and reports token_accounting: reported.
The engines publish llamacpp: series on /metrics. The router reads llamacpp:prompt_tokens_cached_total for cache hits. The engines publish no KV capacity, cache block size or cache events, so prefix-cache pricing stays off for this backend.
Limits¶
- Engine attestation, the profiling sweep of
narwhal-profileand KV transfer evidence apply to vLLM and SGLang fleets. narwhal-engineand the deployment stages launch vLLM and SGLang engines.- The backend runs on x86-64 Linux and under WSL2, with every engine of a fleet on one host.