Deploying a fleet¶
Requirements¶
Deploy one model per fleet. Its engines need a compatible KV layout, KV transfer between every eligible producer and consumer, and one active role controller. vLLM transfers KV through NIXL kv_both. SGLang transfers KV through the mooncake connector.
| Host | Responsibilities | Required software and inputs |
|---|---|---|
| Management workstation | Generate private configuration, inspect hosts, package and install the approved revision, run trials, and retain evidence. | Git, Bash, Python 3.11+ with venv, Make, OpenSSH, your site's access tools, and sshpass for password authentication. The private .env, approved revision, engine image, model paths, fabric interface, run directory, ports, and credentials. |
| Router and observability host | Run the router and checks, and host Prometheus and Grafana. | Python 3.11+ with venv, Git, Make, curl, Docker Engine, and Compose. The verified bundle and revision, engine and attestation URLs, API credential, model settings, and SLOs. |
| Engine host | Run vLLM or SGLang and the attestation sidecar, and capture cache and transfer evidence. | Python 3.11+ with venv, Git, Make, curl, accelerator driver, container runtime, and transfer devices. The accelerator allocation, TP shape, pinned image, checkpoint, launch policy, and fabric addresses and ports. |
Concurrency¶
Inspect engine hosts, start engines with disjoint GPU allocations, and attest running engines after the fabric check in parallel. Start engines that share GPUs one at a time. Measure one directed fabric edge at a time with idle engines.
Per-engine and per-host rules:
- Compare each engine allocation with the GPU inventory collected on that engine host.
- Install once per physical host.
- Capture each vLLM engine's live cache layout.
Deployment record¶
Create a private deployment record before the first command. For every stage, retain the host or stable host alias, source revision, starting state, executed commands, exit status, and generated artifacts.
On failure, record the state before changing it, retain the failed evidence, and remove only files and processes from that attempt. Replace private addresses, paths, and credentials with stable aliases before exporting evidence.
Deployment sequence¶
Record each gate's result before the next gate.
-
Freeze inputs and discover the deployment.
-
Package and install the approved revision.
-
Validate and start every engine.
-
Prove the transfer fabric against the serving cache.
-
Attest the live engine processes.
-
Profile engines and run preflight
Profile once and run the live KV contract.
-
Start the service and validate capacity through the private path.
Rerunning after a change¶
| Change | Repeat |
|---|---|
| Offered rate or request count within the profiled range | Drain the router and run the next trial point with the existing profiles, fabric samples, and preflight. |
| SLO or first-token deadline | Run narwhal-check with the revised limits against saved profiles and live handoffs, then load the edited fleet. |
| Router restart with unchanged fleet configuration | Restart the router. |
Engine restart with unchanged attested launch_digest |
Capture the live cache layout on vLLM and the attestation, then run full preflight against saved profiles and the first-token calibration. |
| Engine restart with changed attestation or only an attestation digest | Capture the live cache layout on vLLM and the attestation, profile the new process generation, recalibrate the first-token deadline, then run full preflight. |
| Changed cache geometry | Recalculate the fabric budget, from cache-layout.json on vLLM or from measured handoffs on SGLang. |
| Changed fabric route, host assignment, or transport | Measure affected directed links against the source budget, recalibrate the first-token deadline, and exercise the live KV paths in preflight. |
Evidence and recovery index¶
Record these inputs and artifacts in the private deployment record:
| Stage | Inputs | Evidence to retain |
|---|---|---|
| Discovery and access | .env, installed image and model, GPU inspection utilities, launch policy, SSH route and key |
.env, generated JSON, host-key database, runs/discovery/<run>/, runs/access-<id>/ |
| Package and install | Approved commit, role mapping, per-engine allocation, host prerequisites | runs/deployment-env/<run>/, remote ~/Narwhal-deploy/<id>/, role files, fleet config, install marker |
| Host and engine validation | PCI accelerator identity, visible devices, allocation, TP size, image identity, checkpoint tree digest, paths, and ports | Engine role env, launch record, discovery checkpoint manifests, model_tree_sha256, ENGINE_RUN, image-check and HTTP captures |
| Fabric | Peer addresses, transport, representative process, live cache layout, workload budget | Cache layout, budget, link fingerprints, route files, directed samples, comparisons, edge matrix |
| Attestation | Checked live process, model dimensions, and sidecar endpoint, with the vLLM protocol version, cache layout, transfer mode, and handshake policy | engine-attestation.json, all ENGINE_RUN captures, router fleet before and after attestation |
| Profiling | Idle engines, fixed runtime, generated sequence limits, prompt lengths, and concurrency | Fleet config, profiling limits, profile and sample files |
| Preflight | Current processes, profiles, first-token calibration, fleet contract, and SLOs | Router environment, fleet config, calibration artifact, private check output |
| Router | Bind address, model, engine count, and role split | Router env, fleet config, endpoint captures |
| Capacity trial | Workstation load client, router observability stack, SSH path, fixed runtime and cache policy, workload, and thresholds | Tunnel logs, Compose discovery, runs/load-trial-<id>/, manifests, request records, summaries, state and network snapshots |
Correct a failed stage from its evidence:
- Discovery and access: fix the named environment field, missing host utility, image inspection issue, credential, route, or independently verified changed host key.
- Package and install:
- Create a new prepared run from corrected input.
- Resume dependency installation after fixing the host cause.
- Host and engine validation:
- Restore device exposure or artifacts, free the planned ports, or repair the differing checkpoint file.
- Repeat discovery.
- Fabric:
- Identify the root cause in route, binding, listener, firewall, HCA/GID, MTU, retransmissions or RDMA counters, CPU saturation, or concurrent traffic.
- Re-sample.
- Attestation:
- Inspect the bound capture and source.
- Restart only the affected process or sidecar.
- Rerun finalization.
- Profiling:
- Correct the failing engine or sweep.
- Write new samples to a fresh profile path.
- Preflight: follow the reported engine, transfer leg, or budget into the matching troubleshooting path.
- Router: inspect the listener's process, the address family, the router error, and the upstream engine error.
- Capacity trial:
- Resolve local port conflicts, remote listeners, scrape failures, client saturation, scheduler lag, and reconciliation gaps.
- Attribute any remaining SLO miss to serving capacity.
Runtime and tooling references¶
-
Fleet fields and defaults.
-
Profiling and service-level objective (SLO) calibration.
-
Prometheus and Grafana setup.
-
Symptom-based diagnosis.