Skip to content

Starting and verifying monitoring

Prerequisites

  • a running Narwhal router
  • Linux
  • Docker Engine with the Compose plugin
  • Python 3.11 or newer
  • curl
  • network reachability from the router host to every engine metrics endpoint

Run these commands in the installed router-role shell, with runs/deployment/.env.router loaded and .venv active.

Configuring the monitored deployment

NARWHAL_ROUTER_URL is the router origin Prometheus scrapes, direct or through a stable local tunnel. Prometheus reads the engine targets from the router's GET /narwhal/targets every 15 seconds, so the targets follow the fleet the router serves.

Set it in the router-role shell:

export NARWHAL_ROUTER_URL=http://127.0.0.1:8000

Scraping the fleet control service

To mark fleet control actions and load jobs on the Narwhal Orchestrator dashboard, Prometheus must scrape the fleet control service. Before make observe, set its address and token:

export NARWHAL_CONTROL_METRICS_URL=http://127.0.0.1:8020
export NARWHAL_CONTROL_TOKEN=<control-token>

Replace <control-token> with the service's bearer token. The address needs an explicit port. If the address is set without the token, make observe stops with an error.

Scraping Narwhal tools

The engine table shows which tool keeps an engine busy when Prometheus scrapes narwhal-check and narwhal-profile. Set the address they serve tool metrics on, in the shell that runs make observe and in each shell that runs a tool:

export NARWHAL_TOOL_METRICS_URL=http://127.0.0.1:18030

Give the address an explicit port that Prometheus can reach. Run one tool at a time. A second tool finds the port in use, prints tool metrics: cannot listen on, and runs with its engines reading Probing.

Starting Prometheus and Grafana

Run:

make observe

Readiness contract

make observe returns when every component passes its check:

Component Check
Prometheus /-/ready answers and the build reports version 3.14.0
Grafana /api/health reports database ok and version 13.2.1
Grafana datasource The Prometheus datasource points at the Prometheus listener
Dashboard narwhal-router loads from Grafana's dashboard.grafana.app/v2beta1 API with its router selector defaulting to All (regex .*)
Dashboard router queries Every router-scoped query uses instance=~"$router", with at least one present
narwhal-router job Exactly one healthy target at NARWHAL_ROUTER_URL
engines job One healthy target per engine in the router's /narwhal/targets, labelled with its iid
Router readiness narwhal_router_ready reports 1

Each startup stage has a deadline:

Startup stage Deadline
Listener checks and each Docker inspection command 10 seconds
Initial image pulls and container creation 5 minutes
Prometheus and Grafana HTTP checks, including datasource and dashboard 60 seconds after the containers start
Target health and narwhal_router_ready 60 seconds after the HTTP checks pass

Verifying Prometheus targets

Query the scrape state from the router host:

curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job=~"narwhal-router|engines"}' \
  | python3 -m json.tool

The response has one series for the router and one for each configured engine. A value of 1 marks a successful scrape, and 0 marks a failed scrape.

If you set NARWHAL_CONTROL_METRICS_URL, check up{job="fleet-control"} the same way. If you set NARWHAL_TOOL_METRICS_URL, up{job="narwhal-tools"} is 1 while a tool runs preflight, first-token calibration or a live sweep.

Prometheus /targets shows the discovery state and scrape errors for each endpoint.

Staged monitoring files

make observe stages its configuration under runs/observability/mounts/:

Source Staged path
tools/observability/prometheus.yml and NARWHAL_ROUTER_URL prometheus/prometheus.yml
tools/observability/prometheus-alerts.yml prometheus/prometheus-alerts.yml
NARWHAL_ROUTER_URL prometheus/targets/router.json
tools/observability/grafana/provisioning/ grafana-provisioning/
tools/observability/grafana-narwhal.json and NARWHAL_CONTROL_CONSOLE_URL grafana-dashboards/narwhal.json
NARWHAL_CONTROL_METRICS_URL prometheus/targets/fleet-control.json, empty if unset
NARWHAL_CONTROL_TOKEN prometheus/fleet-control-token, empty if NARWHAL_CONTROL_METRICS_URL is unset
NARWHAL_TOOL_METRICS_URL prometheus/targets/tools.json, empty if unset

The staged files and the renderer token have these modes:

Path Mode
runs/observability/mounts/ 0700
Mounted subdirectories 0755
Mounted files, including prometheus/fleet-control-token 0644
runs/observability/renderer-token 0600

Compose bind-mounts the prometheus, grafana-provisioning, and grafana-dashboards subdirectories read-only.

make observe writes the Grafana-to-renderer token to runs/observability/renderer-token on its first run and reuses that file on later runs.

Each make observe run regenerates the staged files and resets their permissions.

Access dashboards and isolate listeners