Skip to content

Discover the deployment

Loading the private environment

  1. In a fresh management checkout, install Narwhal and activate its environment:

    make setup
    . .venv/bin/activate
    
  2. Copy .env.example to .env.

  3. Restrict .env to mode 0600.
  4. Fill every field.
  5. Load .env with shell tracing disabled:

    set +x
    set -a
    . ./.env
    set +a
    

Set one NARWHAL_NODE_<n>_SSH destination for each engine <n>. Discovery requires at least two engines. NARWHAL_ROUTER_SSH sets the router's destination and defaults to the lowest-numbered engine's destination.

A destination is user@host or an OpenSSH alias carrying the username, port, identity, and jump host. Roles that share a destination share one host entry and credential.

For password authentication, set the destination variable with a _PASSWORD suffix, such as NARWHAL_NODE_1_SSH_PASSWORD. Key authentication uses the configured identity or the SSH agent.

Default service URLs use the global address on NARWHAL_FABRIC_INTERFACE. The engine listens at http://<address>:<NARWHAL_ENGINE_PORT>, and the attestation service at http://<address>:<NARWHAL_ATTEST_PORT>/v1/attestation.

Per-node overrides:

Override Use when
NARWHAL_NODE_<n>_IP The interface has several global addresses
NARWHAL_NODE_<n>_URL The engine service is reachable through another address
NARWHAL_NODE_<n>_ATTESTATION_URL The attestation service is reachable through another address
NARWHAL_NODE_<n>_ENGINE_PORT, NARWHAL_NODE_<n>_ATTEST_PORT The service is bound to a different port

Staging and identifying the checkpoint

Set NARWHAL_ENGINE_MODEL_NAME to the served model name and NARWHAL_MODEL_DIR to the checkpoint directory on each engine host.

For an empty directory with a Hugging Face source:

  1. Set MODEL_REPO_ID to the repository.
  2. Set MODEL_REVISION to the full commit SHA.
  3. Stage the same snapshot on every engine host:

    hf download "$MODEL_REPO_ID" --revision "$MODEL_REVISION" --local-dir "$NARWHAL_MODEL_DIR"
    
  4. Record the repository ID and commit SHA in the private record.

Discovery compares the path, size, and SHA-256 of every regular file in the model directories across replicas. The comparison excludes the root README.md and the .cache/huggingface/ metadata. A replica mismatch stops discovery with an error that names the first differing file.

Running discovery and access checks

Each engine host needs:

  • Python 3
  • Docker
  • ip
  • rocminfo or nvidia-smi
  • the pinned engine image
  • the checkpoint at NARWHAL_MODEL_DIR
  • the run directory at NARWHAL_RUN_DIR

From the management checkout:

python3 tools/deployment/discover_deployment.py --out runs/discovery/first-deploy
. config/deployment.env
python3 tools/deployment/deploy_hosts.py plan
python3 tools/deployment/deploy_hosts.py check-access

Discovery prints Generated private configuration. Load config/deployment.env before preparation. plan prints each host ID and its roles. check-access logs in once per physical host with the pinned key and saves the hostname output under runs/access-<id>/.

Open a login shell on a role's host:

python3 tools/deployment/deploy_hosts.py shell --role engine-1

--role accepts router or engine-<n>.

The host-key store is NARWHAL_SSH_KNOWN_HOSTS, default config/ssh.known_hosts. Discovery records each new host key and matches stored keys. Other deployment commands match every host key against the store.

Discovery output

Discovery also sets launch settings from the checkpoint. It adds --trust-remote-code when config.json or tokenizer_config.json has an auto_map entry. For a model with convolutional SSM transfer state, it sets VLLM_SSM_CONV_STATE_LAYOUT=DS on vLLM, or on an SGLang mooncake engine adds --mamba-radix-cache-strategy no_buffer and --disable-overlap-schedule unless NARWHAL_ENGINE_ARGS already sets --mamba-radix-cache-strategy or NARWHAL_PREFIX_CACHING is false.

Discovery writes the generated configuration at mode 0600:

File Content
config/hosts.local.json Host IDs, SSH destination and password variable names, and the router and engine roles on each host
config/ssh.known_hosts Server public keys from authenticated management access
config/engine-launch.local.json GPU allocation, image and model metadata, transfer devices, launch policy
config/engine-launch.sources.json Inspection and policy records for each role
config/fleet.json Model, measured hardware and TP shape, engine URL references, initial roles, initial latency targets, profile path
config/deployment.env Generated paths, fabric addresses, engine and attestation URLs, per-engine image and hash values

Keep the generated config/ files together.

The discovery output directory, mode 0700:

File Content
engine-<n>.json GPU products and device mappings, image identity and package versions, model-config hash and dtype, custom-code and convolutional-state flags, fabric interface addresses, image runtime environment
engine-<n>-checkpoint.json Path, size, and SHA-256 of every checkpoint file, the exclusion policy, and model_tree_sha256
logs/ SSH command logs
engine-<n>-inputs.json, engine-launch.json, hosts.json Probe inputs and the candidate launch and host records
manifest.json Hashes of the generated config/ files, the engine roles, and the shared model_tree_sha256

Reuse an inspected fleet:

  1. Reload .env and config/deployment.env.
  2. Verify access.
  3. Prepare a new deployment run.

Rerun discovery after a change to the hardware, model, image, checkpoint, launch policy, route, transport, or another discovery input:

  1. Archive the generated config/ files.
  2. Run discovery into a new output directory.

Confirming the launch policy

A GPU host with one engine role gives that engine every detected GPU, with tensor parallelism set to that count. For several engine roles on one GPU host, declare disjoint NARWHAL_NODE_<n>_GPU_IDS lists. Every replica must have the same accelerator product, GPU count, and TP size.

Default serving policy:

Setting Default
Transfer TCP on NARWHAL_FABRIC_INTERFACE
Model dtype The checkpoint's dtype, otherwise bfloat16
KV dtype automatic
Requested cache block 128 tokens
Execution eager
Maximum context The model's max_position_embeddings, capped at 16,384 tokens
Maximum sequences 8
GPU memory utilisation 0.9
Initial TTFT limit 10 s
Initial TPOT limit 0.125 s
Initial prefill pool The lowest-numbered engine
Initial decode pool The remaining engines
Profile path runs/profiles.json in the installed checkout

Override the policy in .env before discovery:

Field Meaning
NARWHAL_ENGINE_BACKEND, NARWHAL_ENGINE_CONNECTOR Fleet engine.backend (vllm, the default, or sglang) and engine.connector
NARWHAL_DECODE_CUDA_GRAPH_MEMORY_GB GB of decode CUDA graph memory for each SGLang engine, required with the mooncake connector
NARWHAL_GPU_IDS, NARWHAL_TENSOR_PARALLEL_SIZE Comma-separated GPU indices or NVIDIA UUIDs, and the replica TP size
NARWHAL_MODEL_DTYPE, NARWHAL_BLOCK_SIZE Model dtype (bfloat16 or float16) and requested cache block size
NARWHAL_CONTEXT_LENGTH, NARWHAL_MAX_SEQUENCES runtime.context_length and runtime.max_sequences
NARWHAL_PREFIX_CACHING, NARWHAL_KV_EVENTS, NARWHAL_TOKENIZER runtime.prefix_caching and runtime.kv_events, each true or false, and runtime.tokenizer
NARWHAL_ENGINE_ARGS JSON array that replaces the default runtime.extra_args
NARWHAL_ENGINE_ENV JSON object of launcher-supported environment fields that overrides the image environment
NARWHAL_TRANSFER_TRANSPORT, NARWHAL_TRANSFER_NET_DEVICES, NARWHAL_TRANSFER_DEVICES Transport (ucx_tcp or ucx_rdma), UCX network devices (HCA:port entries for RDMA), and a JSON array of RDMA device paths
NARWHAL_TTFT_S, NARWHAL_TPOT_S Initial TTFT and TPOT limits, in seconds
NARWHAL_ENGINE_KEY Engine API key that discovery names in engine.engine_api_key_env

Per-engine overrides use NARWHAL_NODE_<n>_<field>, such as NARWHAL_NODE_2_GPU_IDS.

With NARWHAL_ENGINE_BACKEND=sglang, discovery writes the SGLang runtime fields and requires the sglang, torch and mooncake-transfer-engine versions installed in the image, including CUDA builds such as mooncake-transfer-engine-cuda13. The cache block, execution, maximum sequence and GPU memory defaults apply to vLLM only. For SGLang, NARWHAL_ENGINE_ARGS defaults to an empty list, and discovery writes runtime.max_sequences only when you set NARWHAL_MAX_SEQUENCES, from 1 to 256. Without it, the launcher uses 256.

Discovery writes the maximum context to runtime.context_length and NARWHAL_ENGINE_ARGS to runtime.extra_args. It fails when NARWHAL_ENGINE_ARGS sets a flag that one of the launch settings for every backend controls.

Access failure handling

A failed deployment command prints <host-id>: blocked at <operation>; inspect its private command log. Record the first failing host and gate in the private record.

If an access variable is missing, set the named .env field.

If a host key is new or changed:

  1. Verify the destination and fingerprint independently, such as through the provider console.
  2. Replace the entry in NARWHAL_SSH_KNOWN_HOSTS.

If authentication or the connection fails, check the username, credential, route, SSH port, and firewall.

Install the approved revision