GPU telemetry, alerts, and recovery¶
GPU telemetry¶
Run the deployment's AMD or NVIDIA exporter to feed GPU sensor metrics to its hardware dashboard.
Alert evaluation¶
Prometheus publishes the state of the tools/observability/prometheus-alerts.yml rules in the ALERTS series.
Inspect rules on the router host:
Alert rules select the router by job="narwhal-router" and each engine by job="engines" and iid=<engine identity>.
The deployment's alert manager routes severity="page" and severity="warn" alerts in production.
Troubleshooting¶
Find the symptom and apply the check or action beside it:
| Symptom | Check or action |
|---|---|
| Browser connection fails | Check the SSH tunnel and the route to the router host |
make observe reports an occupied listener |
Stop the reported process or socket unit, or move to isolated listeners |
| Startup reports a command deadline | Check Docker daemon health, registry reachability, and the Compose service status |
| Startup reports a container exit or readiness deadline | Inspect the Prometheus and Grafana Compose logs |
Prometheus reports config permission denied |
Follow the mount permission failure steps |
| Grafana returns dashboard 404 | Follow the mount permission failure steps |
| Router target fails | Check NARWHAL_ROUTER_URL, its route from the router host, and Prometheus /targets |
| An engine replica disappears from charts | Check its entry in the router's /narwhal/targets and its iid label |
| Grafana shows an older dashboard | Rerun make observe |
make observe fails the dashboard readiness check |
Read the Grafana provisioning log |
| An alert evaluates against the wrong scope | Inspect target relabelling for job, instance, and iid |
Verify each fix with a make observe rerun that passes the readiness contract.
Inspecting Compose services¶
Every Compose command for tools/observability/compose.yml requires NARWHAL_RENDERER_TOKEN, which make observe stores in runs/observability/renderer-token.
Load the token in the deployed checkout on the router host, then query the services:
export NARWHAL_RENDERER_TOKEN="$(cat runs/observability/renderer-token)"
docker compose -f tools/observability/compose.yml ps
docker compose -f tools/observability/compose.yml logs prometheus grafana
Mount permission failures¶
- Confirm
tools/observability/start/andtools/observability/compose.ymlshare one checkout. - Rerun
make observefrom that deployed checkout. - Read the staged mount permissions and container logs.
- Save the failure output in the private deployment record.
Retaining monitoring captures¶
- Save deployment addresses and captured responses under
runs/. - Attach the verified Prometheus targets and dashboard queries to the load record from the deployment acceptance sequence.