Discover the deployment¶
Loading the private environment¶
-
In a fresh management checkout, install Narwhal and activate its environment:
-
Copy
.env.exampleto.env. - Restrict
.envto mode 0600. - Fill every field.
-
Load
.envwith shell tracing disabled:
Set one NARWHAL_NODE_<n>_SSH destination for each engine <n>. Discovery requires at least two engines. NARWHAL_ROUTER_SSH sets the router's destination and defaults to the lowest-numbered engine's destination.
A destination is user@host or an OpenSSH alias carrying the username, port, identity, and jump host. Roles that share a destination share one host entry and credential.
For password authentication, set the destination variable with a _PASSWORD suffix, such as NARWHAL_NODE_1_SSH_PASSWORD. Key authentication uses the configured identity or the SSH agent.
Default service URLs use the global address on NARWHAL_FABRIC_INTERFACE. The engine listens at http://<address>:<NARWHAL_ENGINE_PORT>, and the attestation service at http://<address>:<NARWHAL_ATTEST_PORT>/v1/attestation.
Per-node overrides:
| Override | Use when |
|---|---|
NARWHAL_NODE_<n>_IP |
The interface has several global addresses |
NARWHAL_NODE_<n>_URL |
The engine service is reachable through another address |
NARWHAL_NODE_<n>_ATTESTATION_URL |
The attestation service is reachable through another address |
NARWHAL_NODE_<n>_ENGINE_PORT, NARWHAL_NODE_<n>_ATTEST_PORT |
The service is bound to a different port |
Staging and identifying the checkpoint¶
Set NARWHAL_ENGINE_MODEL_NAME to the served model name and NARWHAL_MODEL_DIR to the checkpoint directory on each engine host.
For an empty directory with a Hugging Face source:
- Set
MODEL_REPO_IDto the repository. - Set
MODEL_REVISIONto the full commit SHA. -
Stage the same snapshot on every engine host:
-
Record the repository ID and commit SHA in the private record.
Discovery compares the path, size, and SHA-256 of every regular file in the model directories across replicas. The comparison excludes the root README.md and the .cache/huggingface/ metadata. A replica mismatch stops discovery with an error that names the first differing file.
Running discovery and access checks¶
Each engine host needs:
- Python 3
- Docker
iprocminfoornvidia-smi- the pinned engine image
- the checkpoint at
NARWHAL_MODEL_DIR - the run directory at
NARWHAL_RUN_DIR
From the management checkout:
python3 tools/deployment/discover_deployment.py --out runs/discovery/first-deploy
. config/deployment.env
python3 tools/deployment/deploy_hosts.py plan
python3 tools/deployment/deploy_hosts.py check-access
Discovery prints Generated private configuration. Load config/deployment.env before preparation. plan prints each host ID and its roles. check-access logs in once per physical host with the pinned key and saves the hostname output under runs/access-<id>/.
Open a login shell on a role's host:
--role accepts router or engine-<n>.
The host-key store is NARWHAL_SSH_KNOWN_HOSTS, default config/ssh.known_hosts. Discovery records each new host key and matches stored keys. Other deployment commands match every host key against the store.
Discovery output¶
Discovery also sets launch settings from the checkpoint. It adds --trust-remote-code when config.json or tokenizer_config.json has an auto_map entry. For a model with convolutional SSM transfer state, it sets VLLM_SSM_CONV_STATE_LAYOUT=DS on vLLM, or on an SGLang mooncake engine adds --mamba-radix-cache-strategy no_buffer and --disable-overlap-schedule unless NARWHAL_ENGINE_ARGS already sets --mamba-radix-cache-strategy or NARWHAL_PREFIX_CACHING is false.
Discovery writes the generated configuration at mode 0600:
| File | Content |
|---|---|
config/hosts.local.json |
Host IDs, SSH destination and password variable names, and the router and engine roles on each host |
config/ssh.known_hosts |
Server public keys from authenticated management access |
config/engine-launch.local.json |
GPU allocation, image and model metadata, transfer devices, launch policy |
config/engine-launch.sources.json |
Inspection and policy records for each role |
config/fleet.json |
Model, measured hardware and TP shape, engine URL references, initial roles, initial latency targets, profile path |
config/deployment.env |
Generated paths, fabric addresses, engine and attestation URLs, per-engine image and hash values |
Keep the generated config/ files together.
The discovery output directory, mode 0700:
| File | Content |
|---|---|
engine-<n>.json |
GPU products and device mappings, image identity and package versions, model-config hash and dtype, custom-code and convolutional-state flags, fabric interface addresses, image runtime environment |
engine-<n>-checkpoint.json |
Path, size, and SHA-256 of every checkpoint file, the exclusion policy, and model_tree_sha256 |
logs/ |
SSH command logs |
engine-<n>-inputs.json, engine-launch.json, hosts.json |
Probe inputs and the candidate launch and host records |
manifest.json |
Hashes of the generated config/ files, the engine roles, and the shared model_tree_sha256 |
Reuse an inspected fleet:
- Reload
.envandconfig/deployment.env. - Verify access.
- Prepare a new deployment run.
Rerun discovery after a change to the hardware, model, image, checkpoint, launch policy, route, transport, or another discovery input:
- Archive the generated
config/files. - Run discovery into a new output directory.
Confirming the launch policy¶
A GPU host with one engine role gives that engine every detected GPU, with tensor parallelism set to that count. For several engine roles on one GPU host, declare disjoint NARWHAL_NODE_<n>_GPU_IDS lists. Every replica must have the same accelerator product, GPU count, and TP size.
Default serving policy:
| Setting | Default |
|---|---|
| Transfer | TCP on NARWHAL_FABRIC_INTERFACE |
| Model dtype | The checkpoint's dtype, otherwise bfloat16 |
| KV dtype | automatic |
| Requested cache block | 128 tokens |
| Execution | eager |
| Maximum context | The model's max_position_embeddings, capped at 16,384 tokens |
| Maximum sequences | 8 |
| GPU memory utilisation | 0.9 |
| Initial TTFT limit | 10 s |
| Initial TPOT limit | 0.125 s |
| Initial prefill pool | The lowest-numbered engine |
| Initial decode pool | The remaining engines |
| Profile path | runs/profiles.json in the installed checkout |
Override the policy in .env before discovery:
| Field | Meaning |
|---|---|
NARWHAL_ENGINE_BACKEND, NARWHAL_ENGINE_CONNECTOR |
Fleet engine.backend (vllm, the default, or sglang) and engine.connector |
NARWHAL_DECODE_CUDA_GRAPH_MEMORY_GB |
GB of decode CUDA graph memory for each SGLang engine, required with the mooncake connector |
NARWHAL_GPU_IDS, NARWHAL_TENSOR_PARALLEL_SIZE |
Comma-separated GPU indices or NVIDIA UUIDs, and the replica TP size |
NARWHAL_MODEL_DTYPE, NARWHAL_BLOCK_SIZE |
Model dtype (bfloat16 or float16) and requested cache block size |
NARWHAL_CONTEXT_LENGTH, NARWHAL_MAX_SEQUENCES |
runtime.context_length and runtime.max_sequences |
NARWHAL_PREFIX_CACHING, NARWHAL_KV_EVENTS, NARWHAL_TOKENIZER |
runtime.prefix_caching and runtime.kv_events, each true or false, and runtime.tokenizer |
NARWHAL_ENGINE_ARGS |
JSON array that replaces the default runtime.extra_args |
NARWHAL_ENGINE_ENV |
JSON object of launcher-supported environment fields that overrides the image environment |
NARWHAL_TRANSFER_TRANSPORT, NARWHAL_TRANSFER_NET_DEVICES, NARWHAL_TRANSFER_DEVICES |
Transport (ucx_tcp or ucx_rdma), UCX network devices (HCA:port entries for RDMA), and a JSON array of RDMA device paths |
NARWHAL_TTFT_S, NARWHAL_TPOT_S |
Initial TTFT and TPOT limits, in seconds |
NARWHAL_ENGINE_KEY |
Engine API key that discovery names in engine.engine_api_key_env |
Per-engine overrides use NARWHAL_NODE_<n>_<field>, such as NARWHAL_NODE_2_GPU_IDS.
With NARWHAL_ENGINE_BACKEND=sglang, discovery writes the SGLang runtime fields and requires the sglang, torch and mooncake-transfer-engine versions installed in the image, including CUDA builds such as mooncake-transfer-engine-cuda13. The cache block, execution, maximum sequence and GPU memory defaults apply to vLLM only. For SGLang, NARWHAL_ENGINE_ARGS defaults to an empty list, and discovery writes runtime.max_sequences only when you set NARWHAL_MAX_SEQUENCES, from 1 to 256. Without it, the launcher uses 256.
Discovery writes the maximum context to runtime.context_length and NARWHAL_ENGINE_ARGS to runtime.extra_args. It fails when NARWHAL_ENGINE_ARGS sets a flag that one of the launch settings for every backend controls.
Access failure handling¶
A failed deployment command prints <host-id>: blocked at <operation>; inspect its private command log. Record the first failing host and gate in the private record.
If an access variable is missing, set the named .env field.
If a host key is new or changed:
- Verify the destination and fingerprint independently, such as through the provider console.
- Replace the entry in
NARWHAL_SSH_KNOWN_HOSTS.
If authentication or the connection fails, check the username, credential, route, SSH port, and firewall.