Engine restart and process replacement¶
Prerequisites:
- a management host with
curl, Python 3, and access to the router's private control endpoints - a complete
engine_contractin the fleet configuration - external admission closed during replacement and profile activation
Before draining, record:
- the stop and start commands for the engine and its attestation sidecar
- their process or container identities
- the immutable build
Set ROUTER_URL to the fleet router and replace e0 with the engine ID. Create a private run directory:
ROUTER_URL='http://router:8000'
RUN_DIR="runs/restart-$(date -u +%Y%m%dT%H%M%SZ)"
umask 077
mkdir -p "$RUN_DIR"
Restart one engine¶
This procedure applies when recovery.engine_restart_policy is individual.
Drain the engine¶
Drain e0 with a 300-second deadline:
curl -fsS -X POST "$ROUTER_URL/narwhal/lifecycle/drain" \
-H 'content-type: application/json' \
-d '{"engines":["e0"],"deadline_s":300}' \
> "$RUN_DIR/individual-drain-response.json"
Repeat until it prints e0 ready to stop:
curl -fsS "$ROUTER_URL/narwhal/lifecycle" \
> "$RUN_DIR/individual-drained.json"
python3 - "$RUN_DIR/individual-drained.json" <<'PY'
import json
import sys
engine = json.load(open(sys.argv[1]))["engines"]["e0"]
assert engine["ready_to_stop"], engine
assert not engine["accepts_new"], engine
assert engine["resident"] == {"prefill": 0, "decode": 0}, engine
assert engine["old_process_start"] is not None, engine
print("e0 ready to stop")
PY
Stop the engine when the check passes.
Replace the process¶
- Restart the engine and its attestation sidecar through their configured process manager.
- Check that the engine's
/healthreturns HTTP 200. - Check that
/v1/modelslists the configured model and that the engine identity describes the replacement process:/versionandprocess_start_time_secondsin/metricson vLLM, or/server_infoand the attestation sidecar's/v1/processon SGLang. - Check that the restarted attestation sidecar serves attestation bound to that process.
- Retain process manager output and new engine and sidecar identities.
- If the engine's process generation changed, activate replacement profiles while the engine stays excluded.
Request readmission¶
Request readmission for e0:
curl -fsS -X POST "$ROUTER_URL/narwhal/lifecycle/readmit" \
-H 'content-type: application/json' \
-d '{"engines":["e0"]}' \
> "$RUN_DIR/individual-readmitted.json"
Readmission runs these checks in order, each named by its checks entry:
- The engine and each required peer pass four checks:
health:/healthreturns HTTP 200.attestation <fingerprint>: attestation matches the live process and theengine_contract.profile generation: every loaded profile variant is bound to the engine's live process generation.model:/v1/modelslists the configured model.
- The replacement engine passes
new process identity: its process start is newer than its recorded drain identity. On vLLM, it also passesgeneration: a direct completion probe returns a choice. On SGLang, the fabric handoff in step 3 tests generation. - Each role-permitted engine pair completes a fabric validation KV handoff, recorded as
fabric produce to <peer>andfabric consume from <peer>. On an SGLang fleet, the router pairs the replacement engine with the first live engine in the other role. With themooncakeconnector, the router switches the replacement engine to its router role before the handoff. - The replacement engine passes
final health: its/healthreturns HTTP 200. - The identities, attestation, and profile bindings of all participants match step 1.
When every check passes, readmission returns HTTP 200 with accepts_new: true, and the engine state becomes active. A failed check returns HTTP 409, and fenced router control during validation returns HTTP 503. Both leave the engine blocked.
On HTTP 409, read the cause from engines.<id>.error and checks in GET /narwhal/lifecycle.
Verify the response and retain its checks:
python3 - "$RUN_DIR/individual-readmitted.json" <<'PY'
import json
import sys
engine = json.load(open(sys.argv[1]))["engines"]["e0"]
assert engine["state"] == "active" and engine["accepts_new"], engine
assert engine["new_process_start"] > engine["old_process_start"], engine
assert {"health", "profile generation", "model", "new process identity", "final health"} <= set(engine["checks"]), engine
assert any(check.startswith("attestation ") for check in engine["checks"]), engine
print("e0 readmitted:", engine["checks"])
PY
Fabric checks by engine role:
| Engine role | Expected fabric checks |
|---|---|
| Pinned to prefill | fabric produce to <peer> |
| Pinned to decode | fabric consume from <peer> |
| Unpinned on vLLM | fabric produce to <peer> and fabric consume from <peer> |
| Unpinned on SGLang | fabric produce to <peer> when its router role is prefill, fabric consume from <peer> when it is decode |
Verify placement:
- Send a routed request.
- Verify its engine placement and terminal outcome in the request journal.
Recover an unplanned ejection¶
When an ejected engine's /health returns HTTP 200 and its attestation sidecar responds, the router's automatic recovery validates it. When every check passes and the profiles match the running process, the engine returns to placement.
If the engine stays ejected with lifecycle state active while its attestation sidecar refuses connections:
- Start the attestation sidecar through its process manager.
- Wait for the next recovery probe.
If a restarted engine on a host shared with other KV-transfer engines logs waiting for KV peers to release it:
- Wait up to 180 seconds for peer memory release.
- Read the engine's release rounds in
peer_release.<id>fromGET /narwhal/state.
If a restarted engine on a host shared with other KV-transfer engines fails at startup with free GPU memory below its budget, restart the engine wave.
If the engine's process generation changed:
- Wait for lifecycle state
blocked. - Measure and activate replacement profiles.
- Request readmission.
If any other check fails:
- Repair the blocked engine.
- Request readmission.
Recover loss of every placement peer¶
In a fleet with an engine_contract, one failed member holds the whole wave when every placement peer is lost.
- Repair the failing check.
-
Request whole-wave readmission:
When an individual hold has removed the last available peer, promote it to a whole-wave hold and follow Restarting an engine wave:
Restart an engine wave¶
Set recovery.engine_restart_policy to whole_wave when engine builds share peer state across the fleet.
Under whole_wave:
- a confirmed ejection, process change, or identity failure holds the whole wave
- the router's
/readyreturns HTTP 503 until whole-wave readmission completes
Drain the wave¶
Drain the wave with a 600-second deadline:
curl -fsS -X POST "$ROUTER_URL/narwhal/lifecycle/drain" \
-H 'content-type: application/json' \
-d '{"wave":true,"deadline_s":600}' \
> "$RUN_DIR/wave-drain-response.json"
Repeat until it prints wave ready to stop:
curl -fsS "$ROUTER_URL/narwhal/lifecycle" > "$RUN_DIR/wave-drained.json"
curl -sS -o "$RUN_DIR/wave-ready-body.txt" -w '%{http_code}\n' \
"$ROUTER_URL/ready" > "$RUN_DIR/wave-ready-status.txt"
python3 - "$RUN_DIR/wave-drained.json" "$RUN_DIR/wave-ready-status.txt" <<'PY'
import json
import sys
state = json.load(open(sys.argv[1]))
assert open(sys.argv[2]).read().strip() == "503"
assert state["wave"]["active"] and state["wave"]["ready_to_stop"], state
assert not state["router"]["ready"], state
for iid, engine in state["engines"].items():
assert engine["ready_to_stop"] and not engine["accepts_new"], (iid, engine)
assert engine["resident"] == {"prefill": 0, "decode": 0}, (iid, engine)
assert engine["old_process_start"] is not None, (iid, engine)
print("wave ready to stop:", state["wave"]["id"])
PY
Stop the engines through their process manager when the check passes.
Restart the fleet¶
- Restart every engine and attestation sidecar from one immutable build with the recorded process manager commands.
- Retain old and new identities.
- Repeat the process replacement checks for every member.
- When every attestation sidecar serves valid attestation, activate replacement profiles for each engine whose process generation changed.
Request whole-wave readmission:
curl -fsS -X POST "$ROUTER_URL/narwhal/lifecycle/readmit" \
-H 'content-type: application/json' \
-d '{"wave":true}' > "$RUN_DIR/wave-readmitted.json"
Every wave member returns to placement together when every member passes every readmission check.
Verify the completed wave:
python3 - "$RUN_DIR/wave-readmitted.json" <<'PY'
import json
import sys
state = json.load(open(sys.argv[1]))
assert not state["wave"]["active"] and state["router"]["ready"], state
for iid, engine in state["engines"].items():
assert engine["state"] == "active" and engine["accepts_new"], (iid, engine)
assert engine["new_process_start"] > engine["old_process_start"], (iid, engine)
assert {"health", "profile generation", "model", "new process identity", "final health"} <= set(engine["checks"]), (iid, engine)
assert any(check.startswith("attestation ") for check in engine["checks"]), (iid, engine)
print(iid, engine["checks"])
PY
curl -fsS -o /dev/null "$ROUTER_URL/ready"
Check each member's fabric checks against its role in the fabric checks table under Request readmission.
If readmission fails:
- Read each member's
errorandchecksfromGET /narwhal/lifecycle. - Repair the failure.
- Request whole-wave readmission again.
Recover an unplanned whole-wave hold¶
Drain the wave before replacing any process during an unplanned whole-wave hold.
Save the initial hold state:
curl -fsS "$ROUTER_URL/narwhal/lifecycle" > "$RUN_DIR/unplanned-hold.json"
python3 - "$RUN_DIR/unplanned-hold.json" <<'PY'
import json
import sys
state = json.load(open(sys.argv[1]))
assert state["wave"]["active"] and not state["router"]["ready"], state
assert all(not engine["accepts_new"] for engine in state["engines"].values()), state
for iid, engine in state["engines"].items():
print(iid, engine["old_process_start"], engine["error"])
PY
Request the drain and save its response:
curl -sS -o "$RUN_DIR/unplanned-drain.json" -w '%{http_code}\n' \
-X POST "$ROUTER_URL/narwhal/lifecycle/drain" \
-H 'content-type: application/json' \
-d '{"wave":true,"deadline_s":600}' \
> "$RUN_DIR/unplanned-drain-status.txt"
python3 - "$RUN_DIR/unplanned-drain.json" <<'PY'
import json
import sys
state = json.load(open(sys.argv[1]))
print("drain error:", state["error"])
for iid, engine in state["engines"].items():
print(iid, engine["old_process_start"], engine["error"])
PY
In the response, engines.<id>.old_process_start holds the process start recorded by this drain.
When the drain records every identity, it returns HTTP 200 with each member's old_process_start. Verify wave.ready_to_stop and follow Restarting the fleet.
When an identity read fails, the drain returns HTTP 503 with old_process_start set to null for that member. Repair the identity read and retry the wave drain.
If an engine stopped during identity collection:
- Start that engine through its process manager during the whole-wave hold.
- Wait for its process identity endpoints to answer.
- Retry the wave drain.
- Repeat the drain check until
wave.ready_to_stop = true. - Restart the fleet, including every process started for identity collection.
- Request whole-wave readmission.
Activating replacement profiles¶
Prerequisites:
- healthy replacement processes and attestation sidecars on hold
- routed requests finished before profiling, preflight, or router replacement
- a shell on the router host, in the environment and working directory of the router's current launch
If the run directory is on another host:
- Copy it to the router host.
- Set
RUN_DIRandROUTER_URLto a path and address that work from the router host.
For a warm standby router:
- Stop it through its process manager before replacing the active router.
- Restart it with the same final profile store after readmission.
Refresh profiles¶
Re-profile the replaced engines and merge their rows into the current store with the recorded measurement recipe.
Prepare the activation configuration¶
Set FLEET to the current fleet file and FRESH_PROFILES to the complete profile store for the current processes, with every shared-GPU role variant the original fleet requires. Replace both example paths:
FLEET='config/fleet.production.json'
FRESH_PROFILES='runs/replacement/profiles.json'
ACTIVATION_FLEET="$RUN_DIR/fleet-activation.json"
Write the activation fleet file:
python3 - "$FLEET" "$FRESH_PROFILES" "$RUN_DIR" <<'PY'
import json
import sys
from pathlib import Path
source, profiles, run = map(Path, sys.argv[1:])
fleet = json.loads(source.read_text())
assert profiles.is_file(), profiles
fleet.setdefault("profiles", {})["path"] = str(profiles.resolve())
fleet.setdefault("recovery", {})["state_path"] = str((run / "resume-state.json").resolve())
with (run / "fleet-activation.json").open("x") as output:
json.dump(fleet, output, indent=2)
output.write("\n")
PY
The activation fleet file keeps engines unchanged and sets profiles.path to FRESH_PROFILES and recovery.state_path to $RUN_DIR/resume-state.json. Resume reads a state handoff with schema_version 1.
Run full preflight¶
Run full preflight on the idle fleet:
If the log reports process differs from first-token calibration:
- Recalibrate the first-token deadline.
- Set
engine.first_token_calibration_pathin$FLEETand$ACTIVATION_FLEETto the new artifact. - Run full preflight again.
When the command exits 0, keep the log, profiles, and sample files.
Capture the held state¶
Capture the state handoff:
curl -fsS "$ROUTER_URL/narwhal/state" > "$RUN_DIR/activation-idle.json"
python3 - "$RUN_DIR/activation-idle.json" <<'PY'
import json
import sys
state = json.load(open(sys.argv[1]))
assert state["admission"]["inflight"] == 0, state["admission"]
PY
curl -fsS "$ROUTER_URL/narwhal/handoff" > "$RUN_DIR/activation-handoff.json"
python3 - "$ACTIVATION_FLEET" "$RUN_DIR/activation-handoff.json" <<'PY'
import json
import sys
from pathlib import Path
fleet = json.load(open(sys.argv[1]))
saved = Path(sys.argv[2])
handoff = json.loads(saved.read_text())
assert handoff["schema"] == "narwhal.handoff" and handoff["schema_version"] == 1
assert set(handoff["engines"]) == {engine["iid"] for engine in fleet["engines"]}
assert handoff["model"] == fleet["model"]
lifecycle = handoff["lifecycle"]
policy = fleet.get("recovery", {}).get("engine_restart_policy", "individual")
assert lifecycle["engine_restart_policy"] == policy
held = [row for row in lifecycle["records"] if row["state"] != "active"]
assert held, "no lifecycle hold to resume"
for row in held:
assert row["state"] != "validating", row
if row["restart_required"]:
assert row["old_process_start"] is not None, row
with Path(fleet["recovery"]["state_path"]).open("xb") as output:
output.write(saved.read_bytes())
print("saved holds:", [row["iid"] for row in held])
PY
Keep activation-handoff.json as pre-restart evidence. resume-state.json is the copy the replacement router reads and updates.
Restart the router with resume enabled¶
- Stop the old router through its process manager.
- Wait for that process to exit.
- Start the replacement through the process manager with
--fleet "$ACTIVATION_FLEET" --resumeand the router's existing serving options.
A standalone router on the default loopback bind and port starts with:
narwhal-serve --fleet "$ACTIVATION_FLEET" --resume \
--host 127.0.0.1 --port 8000 --journal "$RUN_DIR/router-activation.jsonl"
Startup requires these to match the live process generation:
- the loaded profiles
- the engine attestation
- the first-token calibration artifact, when
engine.first_token_calibration_pathis set
Verify the hold before readmission¶
Set the same ROUTER_URL and RUN_DIR in another terminal.
Compare the resumed holds with the handoff:
curl -fsS "$ROUTER_URL/narwhal/lifecycle" > "$RUN_DIR/activation-resumed.json"
python3 - "$RUN_DIR/activation-handoff.json" "$RUN_DIR/activation-resumed.json" <<'PY'
import json
import sys
saved = json.load(open(sys.argv[1]))["lifecycle"]
live = json.load(open(sys.argv[2]))
assert live["engine_restart_policy"] == saved["engine_restart_policy"]
assert live["wave"]["id"] == saved["wave_id"]
for before in saved["records"]:
if before["state"] == "active":
continue
after = live["engines"][before["iid"]]
assert after["state"] != "active" and not after["accepts_new"], after
assert after["old_process_start"] == before["old_process_start"], after
assert after["restart_required"] == before["restart_required"], after
if saved["wave_id"]:
assert live["wave"]["active"] and not live["router"]["ready"], live
print("lifecycle holds and drain identities preserved")
PY
During an individual hold, /ready returns HTTP 200 and the held engine reports accepts_new: false.
Complete the activation:
- Request individual readmission or whole-wave readmission.
- Verify the readmission checks and a routed request.
- Reopen external admission.
Detect process replacement¶
When recovery.liveness_every is above 0, a liveness sweep probes each engine's /health every recovery.liveness_every * controller.monitor_interval_s seconds. recovery.liveness_every defaults to 10 and must be above 0 under whole_wave.
An engine that answers the sweep leaves placement for recovery when one of these checks fails:
- the engine identity:
/versionandprocess_start_time_secondson vLLM, or/server_infoand the sidecar's process start on SGLang - the process-bound attestation and loaded profile bindings
- the process start against the accepted identity