Starting and verifying monitoring¶
Prerequisites¶
- a running Narwhal router
- Linux
- Docker Engine with the Compose plugin
- Python 3.11 or newer
curl- network reachability from the router host to every engine metrics endpoint
Run these commands in the installed router-role shell, with runs/deployment/.env.router loaded and .venv active.
Configuring the monitored deployment¶
NARWHAL_ROUTER_URL is the router origin Prometheus scrapes, direct or through a stable local tunnel. Prometheus reads the engine targets from the router's GET /narwhal/targets every 15 seconds, so the targets follow the fleet the router serves.
Set it in the router-role shell:
Scraping the fleet control service¶
To mark fleet control actions and load jobs on the Narwhal Orchestrator dashboard, Prometheus must scrape the fleet control service. Before make observe, set its address and token:
export NARWHAL_CONTROL_METRICS_URL=http://127.0.0.1:8020
export NARWHAL_CONTROL_TOKEN=<control-token>
Replace <control-token> with the service's bearer token. The address needs an explicit port. If the address is set without the token, make observe stops with an error.
Scraping Narwhal tools¶
The engine table shows which tool keeps an engine busy when Prometheus scrapes narwhal-check and narwhal-profile. Set the address they serve tool metrics on, in the shell that runs make observe and in each shell that runs a tool:
Give the address an explicit port that Prometheus can reach. Run one tool at a time. A second tool finds the port in use, prints tool metrics: cannot listen on, and runs with its engines reading Probing.
Starting Prometheus and Grafana¶
Run:
Readiness contract¶
make observe returns when every component passes its check:
| Component | Check |
|---|---|
| Prometheus | /-/ready answers and the build reports version 3.14.0 |
| Grafana | /api/health reports database ok and version 13.2.1 |
| Grafana datasource | The Prometheus datasource points at the Prometheus listener |
| Dashboard | narwhal-router loads from Grafana's dashboard.grafana.app/v2beta1 API with its router selector defaulting to All (regex .*) |
| Dashboard router queries | Every router-scoped query uses instance=~"$router", with at least one present |
narwhal-router job |
Exactly one healthy target at NARWHAL_ROUTER_URL |
engines job |
One healthy target per engine in the router's /narwhal/targets, labelled with its iid |
| Router readiness | narwhal_router_ready reports 1 |
Each startup stage has a deadline:
| Startup stage | Deadline |
|---|---|
| Listener checks and each Docker inspection command | 10 seconds |
| Initial image pulls and container creation | 5 minutes |
| Prometheus and Grafana HTTP checks, including datasource and dashboard | 60 seconds after the containers start |
Target health and narwhal_router_ready |
60 seconds after the HTTP checks pass |
Verifying Prometheus targets¶
Query the scrape state from the router host:
curl -fsSG http://127.0.0.1:9090/api/v1/query \
--data-urlencode 'query=up{job=~"narwhal-router|engines"}' \
| python3 -m json.tool
The response has one series for the router and one for each configured engine. A value of 1 marks a successful scrape, and 0 marks a failed scrape.
If you set NARWHAL_CONTROL_METRICS_URL, check up{job="fleet-control"} the same way. If you set NARWHAL_TOOL_METRICS_URL, up{job="narwhal-tools"} is 1 while a tool runs preflight, first-token calibration or a live sweep.
Prometheus /targets shows the discovery state and scrape errors for each endpoint.
Staged monitoring files¶
make observe stages its configuration under runs/observability/mounts/:
| Source | Staged path |
|---|---|
tools/observability/prometheus.yml and NARWHAL_ROUTER_URL |
prometheus/prometheus.yml |
tools/observability/prometheus-alerts.yml |
prometheus/prometheus-alerts.yml |
NARWHAL_ROUTER_URL |
prometheus/targets/router.json |
tools/observability/grafana/provisioning/ |
grafana-provisioning/ |
tools/observability/grafana-narwhal.json and NARWHAL_CONTROL_CONSOLE_URL |
grafana-dashboards/narwhal.json |
NARWHAL_CONTROL_METRICS_URL |
prometheus/targets/fleet-control.json, empty if unset |
NARWHAL_CONTROL_TOKEN |
prometheus/fleet-control-token, empty if NARWHAL_CONTROL_METRICS_URL is unset |
NARWHAL_TOOL_METRICS_URL |
prometheus/targets/tools.json, empty if unset |
The staged files and the renderer token have these modes:
| Path | Mode |
|---|---|
runs/observability/mounts/ |
0700 |
| Mounted subdirectories | 0755 |
Mounted files, including prometheus/fleet-control-token |
0644 |
runs/observability/renderer-token |
0600 |
Compose bind-mounts the prometheus, grafana-provisioning, and grafana-dashboards subdirectories read-only.
make observe writes the Grafana-to-renderer token to runs/observability/renderer-token on its first run and reuses that file on later runs.
Each make observe run regenerates the staged files and resets their permissions.