Troubleshooting the console¶
Charts in the standalone console are empty frames¶
Grafana was started before make observe enabled embedding with GF_SECURITY_ALLOW_EMBEDDING.
Run make observe on the router host to recreate Grafana, then reload the console.
Chart markers are missing for actions and load jobs¶
Query the scrape status of the control service:
curl -fsSG http://127.0.0.1:9090/api/v1/query \
--data-urlencode 'query=up{job="fleet-control"}' \
| python3 -m json.tool
- Empty result:
NARWHAL_CONTROL_METRICS_URLwas unset whenmake observelast ran. Set it andNARWHAL_CONTROL_TOKEN, as in Dashboard annotations, and runmake observeagain. - Value
0: open Prometheus/targetsand read thefleet-controlscrape error. For HTTP 401, Prometheus holds an old token. Export the current token and runmake observeagain. For a refused connection,NARWHAL_CONTROL_METRICS_URLnames the wrong address.
Run an engine action to confirm the fix. Its marker appears on the Narwhal Orchestrator Request outcomes chart after the action finishes.
Readmit stays disabled after a drain¶
Readmit shows <engine> must restart before readmission: stop and start it, or restart the engine wave.
The router readmits this engine only after its process restarts.
- Select Stop, then Start. The console enables Readmit once Start succeeds.
- Select Readmit. The engine returns to
in service.
Without the stop and start hooks, restart the engine through its process manager, as in Replacing the process. Under the whole_wave restart policy, restart the engine wave, as in Restarting an engine wave. Then select Readmit.
Start fails and the engine stays ejected¶
Start or Resume fails with one of these errors. The engine's row shows the router's latest lifecycle event in the Readmit tooltip.
| Error | Cause | Fix |
|---|---|---|
<engine> did not return to service within <n>s: the router still ejects <engine> |
The engine process did not start, or it answers health checks too slowly | Read the start hook's output in hooks/<nnn>-engine_start.log in the session directory. Then read the engine's startup log, and fix the cause as in An engine fails to start after a stop |
<engine> did not return to service within <n>s: the router still quarantines <engine> |
Requests to the engine failed, and the router sends it no new requests | Check the engine's hold kind under holds in /narwhal/state. Wait for the hold to end. A timed hold ends after recovery.failure_quarantine_s, or earlier when the engine passes a health check or an inference probe. An inference hold ends once an inference probe passes on every failed path. The engine then returns to service |
<engine> needs fresh profiles before the router readmits it |
The restarted process no longer matches the engine's profiles | Profile the engine and restart the router with the new profiles, as in Activating replacement profiles |
the router blocked readmission of <engine> |
The router's readmission validation failed. The error ends with the router's reason | Fix the reason, then select Readmit |
An engine fails to start after a stop¶
Start fails, and the engine's startup log reports less free GPU memory than the engine requires. The start hook's output is in hooks/<nnn>-engine_start.log in the session directory.
On a host shared with other KV-transfer engines, another engine still holds the stopped engine's KV memory. Restart the engine wave, as in Restarting an engine wave.
To let Start succeed without a wave restart, set UCX_CUDA_IPC_CACHE to n in runtime.environment and use UCX 1.22 or later in the engine image. Peers then release a stopped engine's GPU memory, at the cost of slower transfers and decoding until the engines restart.