Router failover and rollback¶
Router failover¶
Primary router failed¶
- Query
/readyand/narwhal/lifecycleon both routers. - Identify the lease holder: its
/narwhal/lifecyclereportsrouter.controls_fleet: true. - Wait for the lease holder's
/readyto return HTTP 200. - Send load-balancer traffic to that router.
- Check the active router against the last persisted state handoff:
- Its lease epoch is higher than the failed primary's last epoch.
- Its roles match the handoff.
- Its cumulative counters match the handoff.
- Add
--standby-of <active-router>(the active router's URL) to the old primary's launch command. - Restart the old primary.
The recovered standby:
- returns HTTP 503 from
/ready - rejects direct completion requests with HTTP 503
When both routers return HTTP 503 from /ready, read the refusal reasons. If a reason names lease storage or a clock bound, repair the storage or clock with the lease holder and fencing left in place. For any other reason, fix what the reason names.
State handoff is stale or incompatible¶
The standby's /ready returns HTTP 503 with reason no fresh handoff when the previous lease epoch's state handoff expired or the previous router exited before persisting one. A contract version mismatch or a wrong epoch in the state handoff also returns HTTP 503 from the standby's /ready.
- Keep client traffic stopped.
- Restore a compatible router release.
- Restore the deployment set.
If handoff restoration fails:
- Open a maintenance window.
- Set
recovery.resume: falsein the fleet configuration. - Drop
--resumeand--standby-offrom one router's launch command. - Wait until the old router has stopped or fenced itself and its lease has expired.
- Start that router from its opening roles.
- Admit traffic.
Router rollback¶
- Follow Rolling back.
- Run the drills in Validating every release.