Skip to content

Validate and start engines

Inspecting every engine host

Run these checks in the installed engine-role shell on every engine host.

NARWHAL_ENGINE_LAUNCH_CONFIG selects the role's launch record.

Print the allocation and launch policy:

python3 - <<'PY_LAUNCH'
import json
import os
from pathlib import Path
record = json.loads(Path(os.environ["NARWHAL_ENGINE_LAUNCH_CONFIG"]).read_text())
for field in ("role", "accelerator", "gpu_ids", "tensor_parallel_size",
              "accelerator_devices", "network_mode", "transfer", "environment"):
    print(f"{field}: {json.dumps(record[field])}")
PY_LAUNCH

Retain this output.

Read the physical accelerator identity and count:

(
  set -e
  vendors=
  for device in /sys/bus/pci/devices/*; do
    test -r "$device/class" && test -r "$device/vendor" || continue
    case "$(cat "$device/class")" in
      0x03*|0x12*) ;;
      *) continue ;;
    esac
    case "$(cat "$device/vendor")" in
      0x10de) vendors="$vendors nvidia" ;;
      0x1002) vendors="$vendors amd" ;;
    esac
  done
  if test -z "$vendors"; then
    printf '%s\n' 'Inspect GPU PCI device exposure on this engine host.' >&2
    exit 1
  fi
  case "$vendors" in
    *nvidia*) nvidia-smi --query-gpu=pci.bus_id,name,uuid --format=csv ;;
  esac
  case "$vendors" in
    *amd*) rocminfo ;;
  esac
)

Record the accelerator product and the visible physical GPU count. On ROCm, count the rocminfo agents with Device Type set to GPU, and read the product from Marketing Name.

In the router shell, check the hardware fields in runs/deployment/fleet.json against the host observations:

Field Must equal
hardware.accelerator The observed accelerator product
hardware.accelerators_per_engine The number of gpu_ids in each replica's launch record
hardware.tensor_parallel Each replica's tensor_parallel_size

Every selected index or UUID must be visible on its engine host.

Verify the local artifacts and planned listeners before launch:

test "$(docker image inspect "$NARWHAL_ENGINE_IMAGE" --format '{{.Id}}')" = "$NARWHAL_ENGINE_IMAGE"
test "$(sha256sum "$NARWHAL_MODEL_DIR/config.json" | cut -d' ' -f1)" = "$NARWHAL_MODEL_CONFIG_SHA256"
test -d "$NARWHAL_RUN_DIR"
ip address show dev "$NARWHAL_FABRIC_INTERFACE"
ss -ltnp

A passing test command is silent and exits with status 0.

  • If ss -ltnp shows a planned HTTP, attestation, vLLM NIXL side-channel, SGLang bootstrap, or transfer port in use, free the port.
  • If NARWHAL_ENGINE_IMAGE holds a registry digest, compare the runtime's resolved digest with the configured digest.
  • If the provisioned checkpoint changed, rerun discovery.

Check every declared device path:

python3 - <<'PY_DEVICES'
import json
import os
from pathlib import Path
record = json.loads(Path(os.environ["NARWHAL_ENGINE_LAUNCH_CONFIG"]).read_text())
paths = record["accelerator_devices"] + record["transfer"]["devices"]
missing = [path for path in paths if not Path(path).exists()]
for path in paths:
    print(f"{path}: {'missing' if path in missing else 'present'}")
if missing:
    raise SystemExit("Restore the declared device exposure before launch.")
PY_DEVICES

For ROCm containers:

  • The container requires /dev/kfd and the selected DRI devices.
  • A /dev/dri mapping exposes every DRI device on the host.
  • Verify the user, group, and container-user permissions on each device file.

Deriving cache-equivalence groups

From the management shell with config/deployment.env loaded:

python3 - <<'PY_CACHE_GROUPS'
import hashlib
import json
import os
from collections import defaultdict
from pathlib import Path

launches = json.loads(Path(os.environ["NARWHAL_LAUNCH_CONFIG"]).read_text())["engines"]
groups = defaultdict(list)
for role, launch in sorted(launches.items()):
    observed = json.loads(Path(launch["sources"]["runtime"]).read_text())
    inputs = {
        "image_id": observed["image_id"],
        "model_config_sha256": observed["model_sha256"],
        "accelerator": launch["accelerator"],
        "tensor_parallel_size": launch["tensor_parallel_size"],
        "gpu_visibility_env": launch["gpu_visibility_env"],
        "runtime": launch["runtime"],
        "transport": launch["transfer"]["transport"],
    }
    signature = hashlib.sha256(json.dumps(inputs, sort_keys=True).encode()).hexdigest()[:16]
    groups[signature].append(role)
for signature, roles in sorted(groups.items()):
    print(f"{signature}: representative={roles[0]}; roles={','.join(roles)}")
PY_CACHE_GROUPS

Each output line names a signature group, its representative role, and its member roles.

Preparing, checking, and starting each engine

The launcher builds one launch plan per engine with these properties:

Property Value
Serving command vLLM: python3 -m vllm.entrypoints.openai.api_server inside NARWHAL_ENGINE_IMAGE. SGLang: python3 -m sglang.launch_server, started by the Narwhal launch wrapper inside NARWHAL_ENGINE_IMAGE
Model mount NARWHAL_MODEL_DIR, read-only at /model
Devices The configured devices with the selected TP allocation
KV connector vLLM: NixlConnector with kv_role=kv_both, the UCX backend, enforce_handshake_compat=true, and kv_load_failure_policy=fail. SGLang: the mooncake transfer backend; every engine launches as prefill with --enable-pd-role-switch
Side-channel address and port The role environment, for the vLLM NIXL side channel or the SGLang bootstrap server
Transport UCX_TLS of tcp,sm,self,<gpu> for ucx_tcp or rc,sm,self,<gpu> for ucx_rdma
Prefix caching On by default, turned off through runtime.prefix_caching
Cache events Published by the engine over private IPC sockets under /tmp/narwhal-<uid>/ while prefix caching and runtime.kv_events are on
Cache event opt-out Prefix caching and cache events documentation

For the capacity trial, set NARWHAL_PREFIX_CACHING=false in .env before discovery, or set runtime.prefix_caching to false in the launch records before preparing the launch plans.

To turn prefix caching off after launch:

  1. Derive the cache-equivalence groups again.
  2. Prepare a fresh launch directory.
  3. Restart each engine.
  4. Repeat the work for an engine restart with a changed attested launch_digest from the deployment sequence.

For a separate tokenizer at PATH, set NARWHAL_TOKENIZER=PATH in .env before discovery, or set runtime.tokenizer to PATH before preparing the launch plan.

When the image, model, dtype, cache policy, TP allocation, or model arguments change:

  1. Derive the groups.
  2. For vLLM, capture the live cache layout.

When the workload assumptions change, recalculate the fabric budget.

In every engine-role shell, prepare the launch plan:

umask 077
mkdir -p runs
export ENGINE_RUN="runs/engine-launch-$(date -u +%Y%m%dT%H%M%SZ)-$$"
narwhal-engine prepare --out "$ENGINE_RUN"
python3 -m json.tool "$ENGINE_RUN/launch.json"

The launcher prints Prepared <role>; review launch.json and run the image check.

Inspect launch.json for:

  • the immutable image
  • the complete serving command
  • the mounts and device mappings
  • the endpoint
  • the application revision
  • the source hashes

Keep container.env private.

Validate the image and plan:

narwhal-engine check --run "$ENGINE_RUN"

A passing check confirms, for both backends:

  • the immutable image identity and runtime identity
  • the pinned package versions
  • the custom-code requirements
  • the plan hash
  • the resolved prefix-caching setting and cache-event endpoints, matched against the plan

For vLLM, the check also confirms:

  • the connector
  • the tokenizer and its custom-code metadata
  • the DS convolutional-state layout, when required

For SGLang, the check also parses the serving arguments.

The check writes checked.json.

A failed check names the failing package, tokenizer, or identity check.

The check uses the tokenizer at runtime.tokenizer, or the model directory when that field is unset.

For a container engine, the check logs to image-check.log and resolves the tokenizer path in the image and its mounts, including /model for NARWHAL_MODEL_DIR. For a native engine, the check logs to runtime-check.log and resolves the tokenizer path on the host. Each check appends its attempt identifier, the launch-plan hash, and the subprocess output to the check log.

When the launch-plan hash changes, prepare a fresh launch directory. When the launcher changes, prepare a new deployment package in Install the approved revision.

Start the engines:

  1. Recheck the planned ports with ss -ltnp.
  2. Start each engine and follow its log:

    narwhal-engine start --run "$ENGINE_RUN"
    export ENGINE_CONTAINER="$(cat "$ENGINE_RUN/container.id")"
    docker logs --follow "$ENGINE_CONTAINER"
    
  3. Watch the log until the HTTP endpoints come up.

  4. Stop the log follower with Ctrl+C.

When startup fails, inspect the container named by container.id:

docker inspect "$ENGINE_CONTAINER"
docker logs "$ENGINE_CONTAINER"

Find the cause of the failure before creating another plan.

When vLLM exits requesting trust_remote_code=True or VLLM_SSM_CONV_STATE_LAYOUT=DS, recover in this order:

  1. Retain the failed attempt's evidence.
  2. Rerun discovery from the corrected approved revision into fresh outputs.
  3. Prepare a new deployment run.
  4. Install engine 1 into the new checkout.
  5. Validate engine 1 before updating the rest.
  6. Compare the old and new image IDs, model-config hashes, accelerator and TP allocations, runtime settings, and transport.

Docker command deadlines

Docker commands run within the stage budgets in Stage deadlines and recovery. Cleanup after a timeout or cancellation follows Docker command reconciliation.

Before you reuse a failed deployment:

  1. Inspect the retained partial output.
  2. Follow the stage recovery procedure.

Proving the live HTTP process

  1. Wait for the engine's /health endpoint to return HTTP 200.
  2. Run the probe from the same engine-role shell:

    python3 - <<'PY_ENGINE'
    import hashlib
    import json
    import os
    from pathlib import Path
    from urllib.request import Request, urlopen
    run = Path(os.environ["ENGINE_RUN"])
    plan_data = (run / "launch.json").read_bytes()
    plan = json.loads(plan_data)
    checked = json.loads((run / "checked.json").read_text())
    assert checked["plan_sha256"] == hashlib.sha256(plan_data).hexdigest(), "Repeat the image check for this plan"
    sglang = plan.get("engine") == "sglang"
    expected_version = checked["engine_version"]
    headers = {"Content-Type": "application/json"}
    if os.environ.get("NARWHAL_ENGINE_API_KEY"):
        headers["Authorization"] = "Bearer " + os.environ["NARWHAL_ENGINE_API_KEY"]
    def probe(path, filename=None, body=None):
        data = json.dumps(body).encode() if body is not None else None
        request = Request(plan["endpoint"].rstrip("/") + path, data=data, headers=headers)
        with urlopen(request, timeout=60) as response:
            content = response.read()
        if filename:
            with (run / filename).open("xb") as output:
                output.write(content)
        return content
    probe("/health", "health.txt")
    if sglang:
        # /server_info also returns the engine's API key; keep only the version.
        version = {"version": json.loads(probe("/server_info"))["version"]}
        with (run / "version.json").open("x") as output:
            json.dump(version, output)
    else:
        version = json.loads(probe("/version", "version.json"))
    assert version["version"] == expected_version, (
        f"engine reported version {version['version']!r}; checked image expects {expected_version!r}"
    )
    models = json.loads(probe("/v1/models", "models.json"))
    assert os.environ["NARWHAL_ENGINE_MODEL_NAME"] in {m["id"] for m in models["data"]}
    if sglang:
        print("Engine health, version and model passed.")
    else:
        metrics = probe("/metrics", "metrics.txt").decode()
        assert any(line.startswith("process_start_time_seconds ") for line in metrics.splitlines())
        completion = json.loads(probe("/v1/completions", "completion.json", {
            "model": os.environ["NARWHAL_ENGINE_MODEL_NAME"], "prompt": "The sea is",
            "max_tokens": 32, "temperature": 0,
        }))
        assert completion["choices"][0]["text"]
        print("Engine health, version, model, process identity and completion passed.")
    PY_ENGINE
    

For a vLLM engine, the probe checks /version, the process start in /metrics, and one completion. For an SGLang engine, it reads the version from /server_info and stops after the model list.

If the probe reports another version, compare the container that serves the endpoint with the recorded image and container ID. To rerun the probe after a repair, use new capture filenames or a new launch directory.

  • Leave each serving container running through the workload trial.
  • Keep the engines idle during fabric qualification and profiling.

Capturing the live cache layout

For a vLLM engine, capture the cache layout in each engine-role shell:

narwhal-engine capture-cache --run "$ENGINE_RUN"

capture-cache writes cache-layout.json with one record per TP rank.

Compare the resolved layouts and page geometry within each signature group. Matching engines share one fabric budget. An engine that differs gets a separate budget.

narwhal-engine rejects capture-cache for an SGLang plan. For an SGLang engine, skip the capture and size its fabric budget from measured handoffs.

In each engine-role shell, create the fabric run directory and check the budget tool:

umask 077
mkdir -p runs
export FABRIC_RUN="$(mktemp -d runs/fabric-XXXXXX)"
test "$(sha256sum "$NARWHAL_FABRIC_BUDGET_TOOL" | cut -d' ' -f1)" = "$NARWHAL_FABRIC_BUDGET_SHA256"

Qualify the transfer fabric