Engine and whole-wave recovery¶
Prerequisites:
- the management host from Engine restart and process replacement for lifecycle requests
- a complete
engine_contractin the fleet configuration
Engine failure¶
One engine failed unexpectedly¶
- Check that the router ejected or quarantined the engine.
- Check that the other engines receive work.
- Save the engine boot log.
- Save the supervisor exit reason.
For a planned restart, follow Restarting one engine. When recovery.engine_restart_policy is whole_wave, follow Whole-wave recovery. When it is individual, recover the engine with these steps:
- Start the engine.
- Verify its endpoints per Replacing the process.
- Restart its attestation sidecar through the configured process manager.
- Read the sidecar log.
- Repair each named endpoint or contract failure.
- If the engine's process generation changed:
- Activate fresh profiles while preserving its hold.
- Request readmission.
- Follow
/narwhal/lifecycleuntil readmission completes. - Confirm
accepts_new: true.
If the lifecycle state becomes blocked:
- Repair the failure named in
engines.<id>.error. - Send
POST /narwhal/lifecycle/readmitwith body{"engines":["<id>"]}.
Whole-wave recovery¶
Triggers for whole-wave recovery:
recovery.engine_restart_policyiswhole_wave- a stale-peer assertion
- a transfer stall that kills a peer
- a process replacement that invalidates shared peer state
Steps for whole-wave recovery:
- Start a whole-wave drain.
- If drain identity capture fails, follow Recovering an unplanned whole-wave hold.
- Wait until
wave.ready_to_stopistrue. - Confirm
/readyreturns HTTP 503. - Stop every engine process tree through the external supervisor.
- Verify that only the intended worker processes hold accelerator memory.
- Launch every engine from the same immutable image and launch contract.
- Start a fresh attestation sidecar for each engine.
- While the hold is in effect, activate one replacement profile store covering every engine whose process generation changed.
- Request whole-wave readmission.
- Wait for fabric validation to pass.
- Confirm
/readyreturns HTTP 200. - Restore ingress.
- Run the drills in Validating every release.