Validate and start engines¶
Inspecting every engine host¶
Run these checks in the installed engine-role shell on every engine host.
NARWHAL_ENGINE_LAUNCH_CONFIG selects the role's launch record.
Print the allocation and launch policy:
python3 - <<'PY_LAUNCH'
import json
import os
from pathlib import Path
record = json.loads(Path(os.environ["NARWHAL_ENGINE_LAUNCH_CONFIG"]).read_text())
for field in ("role", "accelerator", "gpu_ids", "tensor_parallel_size",
"accelerator_devices", "network_mode", "transfer", "environment"):
print(f"{field}: {json.dumps(record[field])}")
PY_LAUNCH
Retain this output.
Read the physical accelerator identity and count:
(
set -e
vendors=
for device in /sys/bus/pci/devices/*; do
test -r "$device/class" && test -r "$device/vendor" || continue
case "$(cat "$device/class")" in
0x03*|0x12*) ;;
*) continue ;;
esac
case "$(cat "$device/vendor")" in
0x10de) vendors="$vendors nvidia" ;;
0x1002) vendors="$vendors amd" ;;
esac
done
if test -z "$vendors"; then
printf '%s\n' 'Inspect GPU PCI device exposure on this engine host.' >&2
exit 1
fi
case "$vendors" in
*nvidia*) nvidia-smi --query-gpu=pci.bus_id,name,uuid --format=csv ;;
esac
case "$vendors" in
*amd*) rocminfo ;;
esac
)
Record the accelerator product and the visible physical GPU count. On ROCm, count the rocminfo agents with Device Type set to GPU, and read the product from Marketing Name.
In the router shell, check the hardware fields in runs/deployment/fleet.json against the host observations:
| Field | Must equal |
|---|---|
hardware.accelerator |
The observed accelerator product |
hardware.accelerators_per_engine |
The number of gpu_ids in each replica's launch record |
hardware.tensor_parallel |
Each replica's tensor_parallel_size |
Every selected index or UUID must be visible on its engine host.
Verify the local artifacts and planned listeners before launch:
test "$(docker image inspect "$NARWHAL_ENGINE_IMAGE" --format '{{.Id}}')" = "$NARWHAL_ENGINE_IMAGE"
test "$(sha256sum "$NARWHAL_MODEL_DIR/config.json" | cut -d' ' -f1)" = "$NARWHAL_MODEL_CONFIG_SHA256"
test -d "$NARWHAL_RUN_DIR"
ip address show dev "$NARWHAL_FABRIC_INTERFACE"
ss -ltnp
A passing test command is silent and exits with status 0.
- If
ss -ltnpshows a planned HTTP, attestation, vLLM NIXL side-channel, SGLang bootstrap, or transfer port in use, free the port. - If
NARWHAL_ENGINE_IMAGEholds a registry digest, compare the runtime's resolved digest with the configured digest. - If the provisioned checkpoint changed, rerun discovery.
Check every declared device path:
python3 - <<'PY_DEVICES'
import json
import os
from pathlib import Path
record = json.loads(Path(os.environ["NARWHAL_ENGINE_LAUNCH_CONFIG"]).read_text())
paths = record["accelerator_devices"] + record["transfer"]["devices"]
missing = [path for path in paths if not Path(path).exists()]
for path in paths:
print(f"{path}: {'missing' if path in missing else 'present'}")
if missing:
raise SystemExit("Restore the declared device exposure before launch.")
PY_DEVICES
For ROCm containers:
- The container requires
/dev/kfdand the selected DRI devices. - A
/dev/drimapping exposes every DRI device on the host. - Verify the user, group, and container-user permissions on each device file.
Deriving cache-equivalence groups¶
From the management shell with config/deployment.env loaded:
python3 - <<'PY_CACHE_GROUPS'
import hashlib
import json
import os
from collections import defaultdict
from pathlib import Path
launches = json.loads(Path(os.environ["NARWHAL_LAUNCH_CONFIG"]).read_text())["engines"]
groups = defaultdict(list)
for role, launch in sorted(launches.items()):
observed = json.loads(Path(launch["sources"]["runtime"]).read_text())
inputs = {
"image_id": observed["image_id"],
"model_config_sha256": observed["model_sha256"],
"accelerator": launch["accelerator"],
"tensor_parallel_size": launch["tensor_parallel_size"],
"gpu_visibility_env": launch["gpu_visibility_env"],
"runtime": launch["runtime"],
"transport": launch["transfer"]["transport"],
}
signature = hashlib.sha256(json.dumps(inputs, sort_keys=True).encode()).hexdigest()[:16]
groups[signature].append(role)
for signature, roles in sorted(groups.items()):
print(f"{signature}: representative={roles[0]}; roles={','.join(roles)}")
PY_CACHE_GROUPS
Each output line names a signature group, its representative role, and its member roles.
Preparing, checking, and starting each engine¶
The launcher builds one launch plan per engine with these properties:
| Property | Value |
|---|---|
| Serving command | vLLM: python3 -m vllm.entrypoints.openai.api_server inside NARWHAL_ENGINE_IMAGE. SGLang: python3 -m sglang.launch_server, started by the Narwhal launch wrapper inside NARWHAL_ENGINE_IMAGE |
| Model mount | NARWHAL_MODEL_DIR, read-only at /model |
| Devices | The configured devices with the selected TP allocation |
| KV connector | vLLM: NixlConnector with kv_role=kv_both, the UCX backend, enforce_handshake_compat=true, and kv_load_failure_policy=fail. SGLang: the mooncake transfer backend; every engine launches as prefill with --enable-pd-role-switch |
| Side-channel address and port | The role environment, for the vLLM NIXL side channel or the SGLang bootstrap server |
| Transport | UCX_TLS of tcp,sm,self,<gpu> for ucx_tcp or rc,sm,self,<gpu> for ucx_rdma |
| Prefix caching | On by default, turned off through runtime.prefix_caching |
| Cache events | Published by the engine over private IPC sockets under /tmp/narwhal-<uid>/ while prefix caching and runtime.kv_events are on |
| Cache event opt-out | Prefix caching and cache events documentation |
For the capacity trial, set NARWHAL_PREFIX_CACHING=false in .env before discovery, or set runtime.prefix_caching to false in the launch records before preparing the launch plans.
To turn prefix caching off after launch:
- Derive the cache-equivalence groups again.
- Prepare a fresh launch directory.
- Restart each engine.
- Repeat the work for an engine restart with a changed attested
launch_digestfrom the deployment sequence.
For a separate tokenizer at PATH, set NARWHAL_TOKENIZER=PATH in .env before discovery, or set runtime.tokenizer to PATH before preparing the launch plan.
When the image, model, dtype, cache policy, TP allocation, or model arguments change:
- Derive the groups.
- For vLLM, capture the live cache layout.
When the workload assumptions change, recalculate the fabric budget.
In every engine-role shell, prepare the launch plan:
umask 077
mkdir -p runs
export ENGINE_RUN="runs/engine-launch-$(date -u +%Y%m%dT%H%M%SZ)-$$"
narwhal-engine prepare --out "$ENGINE_RUN"
python3 -m json.tool "$ENGINE_RUN/launch.json"
The launcher prints Prepared <role>; review launch.json and run the image check.
Inspect launch.json for:
- the immutable image
- the complete serving command
- the mounts and device mappings
- the endpoint
- the application revision
- the source hashes
Keep container.env private.
Validate the image and plan:
A passing check confirms, for both backends:
- the immutable image identity and runtime identity
- the pinned package versions
- the custom-code requirements
- the plan hash
- the resolved prefix-caching setting and cache-event endpoints, matched against the plan
For vLLM, the check also confirms:
- the connector
- the tokenizer and its custom-code metadata
- the DS convolutional-state layout, when required
For SGLang, the check also parses the serving arguments.
The check writes checked.json.
A failed check names the failing package, tokenizer, or identity check.
The check uses the tokenizer at runtime.tokenizer, or the model directory when that field is unset.
For a container engine, the check logs to image-check.log and resolves the tokenizer path in the image and its mounts, including /model for NARWHAL_MODEL_DIR. For a native engine, the check logs to runtime-check.log and resolves the tokenizer path on the host. Each check appends its attempt identifier, the launch-plan hash, and the subprocess output to the check log.
When the launch-plan hash changes, prepare a fresh launch directory. When the launcher changes, prepare a new deployment package in Install the approved revision.
Start the engines:
- Recheck the planned ports with
ss -ltnp. -
Start each engine and follow its log:
-
Watch the log until the HTTP endpoints come up.
- Stop the log follower with Ctrl+C.
When startup fails, inspect the container named by container.id:
Find the cause of the failure before creating another plan.
When vLLM exits requesting trust_remote_code=True or VLLM_SSM_CONV_STATE_LAYOUT=DS, recover in this order:
- Retain the failed attempt's evidence.
- Rerun discovery from the corrected approved revision into fresh outputs.
- Prepare a new deployment run.
- Install engine 1 into the new checkout.
- Validate engine 1 before updating the rest.
- Compare the old and new image IDs, model-config hashes, accelerator and TP allocations, runtime settings, and transport.
Docker command deadlines¶
Docker commands run within the stage budgets in Stage deadlines and recovery. Cleanup after a timeout or cancellation follows Docker command reconciliation.
Before you reuse a failed deployment:
- Inspect the retained partial output.
- Follow the stage recovery procedure.
Proving the live HTTP process¶
- Wait for the engine's
/healthendpoint to return HTTP 200. -
Run the probe from the same engine-role shell:
python3 - <<'PY_ENGINE' import hashlib import json import os from pathlib import Path from urllib.request import Request, urlopen run = Path(os.environ["ENGINE_RUN"]) plan_data = (run / "launch.json").read_bytes() plan = json.loads(plan_data) checked = json.loads((run / "checked.json").read_text()) assert checked["plan_sha256"] == hashlib.sha256(plan_data).hexdigest(), "Repeat the image check for this plan" sglang = plan.get("engine") == "sglang" expected_version = checked["engine_version"] headers = {"Content-Type": "application/json"} if os.environ.get("NARWHAL_ENGINE_API_KEY"): headers["Authorization"] = "Bearer " + os.environ["NARWHAL_ENGINE_API_KEY"] def probe(path, filename=None, body=None): data = json.dumps(body).encode() if body is not None else None request = Request(plan["endpoint"].rstrip("/") + path, data=data, headers=headers) with urlopen(request, timeout=60) as response: content = response.read() if filename: with (run / filename).open("xb") as output: output.write(content) return content probe("/health", "health.txt") if sglang: # /server_info also returns the engine's API key; keep only the version. version = {"version": json.loads(probe("/server_info"))["version"]} with (run / "version.json").open("x") as output: json.dump(version, output) else: version = json.loads(probe("/version", "version.json")) assert version["version"] == expected_version, ( f"engine reported version {version['version']!r}; checked image expects {expected_version!r}" ) models = json.loads(probe("/v1/models", "models.json")) assert os.environ["NARWHAL_ENGINE_MODEL_NAME"] in {m["id"] for m in models["data"]} if sglang: print("Engine health, version and model passed.") else: metrics = probe("/metrics", "metrics.txt").decode() assert any(line.startswith("process_start_time_seconds ") for line in metrics.splitlines()) completion = json.loads(probe("/v1/completions", "completion.json", { "model": os.environ["NARWHAL_ENGINE_MODEL_NAME"], "prompt": "The sea is", "max_tokens": 32, "temperature": 0, })) assert completion["choices"][0]["text"] print("Engine health, version, model, process identity and completion passed.") PY_ENGINE
For a vLLM engine, the probe checks /version, the process start in /metrics, and one completion. For an SGLang engine, it reads the version from /server_info and stops after the model list.
If the probe reports another version, compare the container that serves the endpoint with the recorded image and container ID. To rerun the probe after a repair, use new capture filenames or a new launch directory.
- Leave each serving container running through the workload trial.
- Keep the engines idle during fabric qualification and profiling.
Capturing the live cache layout¶
For a vLLM engine, capture the cache layout in each engine-role shell:
capture-cache writes cache-layout.json with one record per TP rank.
Compare the resolved layouts and page geometry within each signature group. Matching engines share one fabric budget. An engine that differs gets a separate budget.
narwhal-engine rejects capture-cache for an SGLang plan. For an SGLang engine, skip the capture and size its fabric budget from measured handoffs.
In each engine-role shell, create the fabric run directory and check the budget tool: