Narwhal dev recovery and qualification¶
These procedures recover a Narwhal dev instance after a failed stage, an interrupted Docker command or an interrupted narwhal dev process. Two optional checks qualify timeout and interruption recovery on the reference GPU.
-
Recover from a failed stage.
-
Recover from an interrupted Docker command.
-
Interrupted
narwhal devprocess
Recover from an interrupted
narwhal devprocess. -
Optionally qualify a startup budget expiry on the reference GPU.
-
Reference GPU interruption check
Optionally qualify SIGINT and SIGKILL interruptions on the reference GPU.
Docker command reconciliation¶
Each Docker create or run carries two labels. io.narwhal.launch holds the launch token from docker-owner.json, and io.narwhal.operation holds a unique operation token.
A Docker client timeout or cancellation starts a reconciliation with the daemon. NARWHAL_DOCKER_RECONCILE_SECONDS sets the reconciliation budget, 30 seconds by default, and final cleanup takes up to 15 seconds at default settings.
Reconciliation removes the containers the interrupted operation created and the explicit target of an interrupted start. It keeps the other containers in the launch directory and records their IDs as preserved resources.
Each reconciliation writes a docker-reconcile-*.json report with:
- the operation token
- the targeted, removed, preserved, and surviving IDs
- refusal decisions
- inspection errors
- the observation time
For an inspection_required report result from a stalled daemon:
- Inspect the persisted launch-token label and the recorded IDs before a retry.
-
On the recovered daemon, list the containers that carry the report's operation token:
-
Keep the original launch directory until step 2 completes.
Interrupted narwhal dev process¶
The Linux recovery suite checks twelve barrier and signal combinations for:
- complete committed JSON documents
- released locks
- retained output
- idempotent teardown
- survival of an unrelated process
The twelve combinations:
| Interruption barrier | Signal | Cleanup and fresh down |
Fresh status |
Operator action |
|---|---|---|---|---|
| Child created, before identity capture | SIGINT | A fresh down reports stopped |
||
| Child created, before identity capture | SIGKILL | Recovery uses the last committed process set | Identify the newly created service from its command and log | |
| Process record awaiting atomic replacement | SIGKILL | The previous complete document stays current | Inspect the service described by the pending write | |
| Process record committed | SIGINT | Startup rolls back | ||
| Process record committed | SIGTERM | A fresh down terminates the recorded group |
||
| Service readiness wait | SIGKILL | A fresh down terminates the recorded group |
Reports degraded |
|
| Profiling helper active | SIGINT | Startup rolls back | ||
| Profiling helper active | SIGTERM | Startup rolls back | ||
| Profiling helper active | SIGKILL | A fresh down recovers the helper and service records |
||
| Verification helper active | SIGTERM | A fresh down terminates the retained services |
Reports degraded verification | |
| Verification helper active | SIGKILL | A fresh down terminates the helpers and services |
||
| Recorded service leader exited, delayed worker surviving | SIGKILL | Teardown requests operator inspection | Lists the surviving group | Inspect the worker's identity |
Stage documents record each observed descendant's boot ID and process start ticks.
When the narwhal dev process dies abruptly, status lists matching interrupted helpers in stage_processes, and down terminates the helper supervisor process tree. Both commands retain stdout, stderr, and command evidence.
The stage record reads recovered when the processes stop, and recovery_required with the surviving PIDs when processes survive.
Recover a service left by a SIGKILL before its process or stage record commits:
- Inspect the log, kernel start ticks, and listening ports.
- Stop the confirmed process tree.
- Repeat
down.
If the service leader exited before teardown recorded its workers:
- Wait for
downto reportstopped. - Check the status and teardown documents for a surviving worker group.
Reference GPU timeout check¶
Prerequisites:
- an Ubuntu shell on the reference GPU host
- the pinned template
- an instance you choose
-
a separately identified process that stays alive during the check
-
From a successful process generation, retain the startup logs and
native-start-shared.log.*.stage.json. - Run
narwhal dev down. - Record idle device memory, process IDs, and engine, attestation, and NIXL ports.
- Find the first CUDA allocation time and the startup completion time in the retained logs.
- Set
STARTUP_BUDGET_SECONDSto a budget that expires between the first CUDA allocation and startup completion. -
Run:
-
Record the pre-expiry allocation, command duration, and worker exit.
- Compare the duration with the budget plus termination grace and rollback time.
- Record post-cleanup device memory and port release.
- Compare the device memory with the idle baseline.
- Retain
lifecycle.jsonand the teardown, startup, memory, and stage artifacts. - Confirm the separately identified process survives.
- Start a fresh process generation with the normal budget.
- Verify the fresh process generation.
Reference GPU interruption check¶
Prerequisites:
- an instance you choose on the reference GPU host, with the pinned runtime
- a completed normal
upandverifycycle - its retained process generation, device-memory baseline, process tree, and engine, attestation, and NIXL port assignments
- a separately identified process that stays alive during both checks
Interrupting startup with SIGINT¶
- Start a fresh process generation with
narwhal dev up. - Wait for a committed engine identity and an observed CUDA allocation.
- Send SIGINT to the
narwhal dev upprocess. - Record the process exit, retained startup failure, and worker exits.
- Record device memory and port state.
- From a fresh shell, run
narwhal dev status. - From the same shell, run
narwhal dev down.
Interrupting verification with SIGKILL¶
- Start a fresh process generation.
- Verify the fresh process generation.
- Launch another
narwhal dev verify. - Wait for its preflight stage process record.
- Send SIGKILL to that
narwhal dev verifyprocess. - Record the output of
narwhal dev status. - Run
narwhal dev downtwice. - Compare surviving process identities, device memory, and bound ports against the baseline.
Retaining the evidence¶
- Confirm the separately identified process's identity after teardown.
- Record vLLM, driver, CUDA, and GPU versions with the signal and barrier.
- Retain:
- the
narwhal devoutput andlifecycle.json - the stage and teardown documents
- vLLM logs and GPU process listings
- timestamped memory and port observations
- the
- Investigate the PID of each surviving CUDA allocation or engine port before you start another process generation.