Release drills¶
Validating every release¶
Run the release drills on an idle fleet with external admission closed:
- Run the restart drill for your
recovery.engine_restart_policy. - Test router failover through the production supervisor and load balancer.
- Restore service.
Keep the release, fleet configuration, profiles, engine build, supervisor commands, and drill results in the private deployment directory.
An earlier drill run that meets the pass conditions counts for this release. If an earlier drill used a launcher other than the production supervisor, repeat it with the production supervisor.
Release drill pass conditions¶
The individual engine restart drill passes when:
- The drain completes.
- Stale profiles block readmission.
- Fresh profiles load with the hold preserved.
- A routed request reaches the returned engine.
The whole-wave restart drill passes when:
- The router withdraws readiness before the first stop.
- One failed member holds the whole wave.
- Every member returns together after validation.
The unplanned whole-wave hold drill passes when:
- The drain records current identities for the whole excluded wave before member replacement and readmission.
The router failover drill passes when:
- The load balancer selects one lease holder.
- Roles and cumulative counters survive takeover.
- The previous primary stays fenced.
Individual restart drill¶
Use recovery.engine_restart_policy = individual and the prerequisites in Restarting one engine.
- Drain the engine.
- Replace it through the production supervisor with a launch that changes its attested
launch_digest, or with any relaunch when its sidecar reports an attestation digest only. - Save the pre-stop drain observation, both process identities, and the supervisor output.
- Request readmission with the previous profile loaded, through
curlwith--failremoved. - Keep the rejection: HTTP 409, the profile generation error, and
accepts_new = false. - Activate the fresh profiles.
- Request explicit readmission.
- Save the successful response.
- Confirm it shows
state = active,accepts_new = true, a newer process start, and every readmission check passing. - Send a routed request.
- Confirm its placement includes the returned engine.
- Match the client response and request ID to the journal's terminal event and the router's cumulative counters.
- Keep state snapshots from before and after the routed request.
Whole-wave drill¶
Use recovery.engine_restart_policy = whole_wave and the setup in Engine restart and process replacement, including ROUTER_URL and RUN_DIR.
Draining and restarting¶
- Drain the wave.
- Save the HTTP 503 readiness reading.
- Save the drain identities.
- Confirm zero resident work and
wave.ready_to_stop = true. - Restart every engine from the recorded build through its supervisor.
- Restart every attestation sidecar from the recorded build through its supervisor.
- While the wave remains held, activate replacement profiles for each member whose process generation changed.
- If step 7 activated profiles, verify that resume preserves every drain identity and the whole-wave hold.
Failing one member¶
- Stop one member's attestation sidecar with the recorded supervisor command.
- Confirm its engine stays healthy.
- Confirm the other sidecars keep running.
-
Request readmission, saving the failure response:
curl -sS -o "$RUN_DIR/wave-failed.json" -w '%{http_code}\n' \ -X POST "$ROUTER_URL/narwhal/lifecycle/readmit" \ -H 'content-type: application/json' -d '{"wave":true}' \ > "$RUN_DIR/wave-failed-status.txt" curl -sS -o /dev/null -w '%{http_code}\n' "$ROUTER_URL/ready" \ > "$RUN_DIR/wave-failed-ready-status.txt" python3 - "$RUN_DIR/wave-failed.json" \ "$RUN_DIR/wave-failed-status.txt" "$RUN_DIR/wave-failed-ready-status.txt" <<'PY' import json import sys state = json.load(open(sys.argv[1])) assert open(sys.argv[2]).read().strip() == "409" assert open(sys.argv[3]).read().strip() == "503" assert state["wave"]["active"] and not state["router"]["ready"], state for iid, engine in state["engines"].items(): assert engine["state"] == "blocked" and not engine["accepts_new"], (iid, engine) print(iid, engine["error"], engine["checks"]) PY -
Confirm the stopped sidecar's member reports an attestation failure.
- Confirm every other member reports
another engine in the wave failed validation.
Sampling the hold¶
- In a second terminal, set the same
ROUTER_URLandRUN_DIRvalues. -
Start this sampler while the whole-wave hold is active:
python3 - "$ROUTER_URL" "$RUN_DIR/wave-samples.jsonl" <<'PY' import json import sys import time import urllib.request url = sys.argv[1].rstrip("/") + "/narwhal/lifecycle" with open(sys.argv[2], "x") as output: while True: with urllib.request.urlopen(url, timeout=10) as response: state = json.load(response) output.write(json.dumps({"at": time.time(), "lifecycle": state}) + "\n") output.flush() engines = state["engines"] accepting = {iid for iid, engine in engines.items() if engine["accepts_new"]} assert not accepting or accepting == set(engines), state if not state["wave"]["active"]: assert accepting == set(engines) and state["router"]["ready"], state assert all(engine["state"] == "active" for engine in engines.values()), state print("complete wave readmitted") break assert not accepting and not state["router"]["ready"], state time.sleep(0.25) PYAfter a complete readmission, the sampler prints
complete wave readmittedand exits. A partial readmission fails its assertion, and Ctrl+C stops the sampler early.
Recovering and verifying¶
- Restart the failed member's sidecar with the attestation inputs from profiling.
- Leave the replacement engine running.
- Check that its attestation endpoint returns the digest from its loaded profile.
- Check that the whole-wave hold remains active.
- Request and verify whole-wave readmission.
- Verify that every member passed every readmission check.
- Find the journal's
engine_lifecycleevent withaction = wave_readmitted. - Send a routed request.
- Match its client result to the journal and the router's cumulative counters.
- Save the samples, readmission response, and lifecycle events.
Unplanned whole-wave hold drill¶
whole_wave requires a liveness sweep: recovery.liveness_every above 0, default 10.
- Set
recovery.engine_restart_policy = whole_wave. - Wait until every engine is admitted and idle.
- Stop one attestation sidecar through the production supervisor.
- Wait for the router to withdraw readiness.
- Wait for the router to exclude every member.
- Save the
process_excludedandrestart_requiredlifecycle events. - Recover the unplanned whole-wave hold.
- Check that the drain recorded identities before member replacement.
- Verify full readmission and a routed request.
Already-stopped-engine case:
- Stop an engine before requesting the drain.
- Keep the drain response and the process start time from identity collection.
- Replace that process after the drain.
- Record the trigger and the branch tested.
If the router has ejected the stopped engine at the drain request, the drain succeeds with the last process identity the router verified. If the stopped engine is in placement at the drain request, the drain fails and names the missing identity. Retry the drain after the router ejects the engine.
Restoring service after the drill¶
- Confirm the final profile store covers the serving fleet's current process generations.
- Keep the measurements activated for readmission.
- Keep the evidence for unchanged engines.
- If the drill used an isolated subset:
- Complete the subset's held readmission.
- Assemble the complete serving fleet's profile store.
- Run preflight with the original fleet configuration and that profile store.
- Stop the temporary routers before starting the serving router.
- Confirm the intended router controls the complete fleet.
- Confirm
/readyreturns HTTP 200. - Send a routed request.
- Match its result to the journal and the router's counters.
- Restore external admission.
A later process replacement that changes an engine's process generation needs fresh measurements and another activation. For a later router replacement with an outstanding hold on the same fleet, follow profile activation with resume and keep both journals and the state snapshots.