Skip to content

Monitor a WSL2 development fleet

Run Prometheus and Grafana on the homelab and forward the workstation's router and engine metrics from WSL2. Use the canonical Narwhal Orchestrator dashboard and its narwhal-router and engines jobs.

Configure homelab as an SSH destination in WSL2 and Windows and confirm that ssh homelab true succeeds from each. Start the fleet with narwhal dev up and narwhal dev verify. The commands below use the default instance and a homelab checkout at ~/projects/Narwhal.

Connect metrics and dashboard ports

From WSL2, copy the current fleet document to the monitoring checkout:

dev_run=$(python -c 'import json; print(json.load(open("runs/dev/lifecycle.json"))["run"])')
ssh homelab 'mkdir -p ~/projects/Narwhal/runs/workstation'
scp "$dev_run/fleet.json" homelab:projects/Narwhal/runs/workstation/fleet.json

Keep this tunnel running for the fleet's lifetime:

ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
  -R 127.0.0.1:18000:127.0.0.1:18000 \
  -R 127.0.0.1:18101:127.0.0.1:18101 \
  -R 127.0.0.1:18102:127.0.0.1:18102 \
  -R 127.0.0.1:18103:127.0.0.1:18103 \
  -R 127.0.0.1:18104:127.0.0.1:18104 \
  homelab

The reverse forwards let the homelab scrape WSL2's loopback listeners. Adjust the metrics ports to match a custom dev init --port-base. Select free ports when an existing SSH or VS Code forward owns a listener.

From Windows PowerShell, open a separate tunnel for the browser:

ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 `
  -L 127.0.0.1:33000:127.0.0.1:3000 `
  -L 127.0.0.1:39090:127.0.0.1:9090 homelab

The Windows listeners expose Grafana and Prometheus directly to the browser.

Configure the canonical stack

From the homelab checkout, start a new monitoring stack with:

NARWHAL_FLEET=runs/workstation/fleet.json \
NARWHAL_ROUTER_URL=http://127.0.0.1:18000 \
make observe

For an existing stack, set prometheus_targets to its mounted discovery directory and replace the router and engine target files:

prometheus_targets=/path/to/prometheus/targets
.venv/bin/python -m tools.observability.make_targets \
  runs/workstation/fleet.json \
  --router-url http://127.0.0.1:18000 \
  --output-dir "$prometheus_targets"

Prometheus reloads file discovery automatically. Retain the iid labels generated from the fleet so the dashboard joins each engine to its role.

Check scrapes and routed activity

From the homelab:

curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job=~"narwhal-router|engines"}'
curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=narwhal_router_ready{job="narwhal-router"}'

Expect five healthy scrape series and router readiness 1. Open http://127.0.0.1:33000/d/narwhal-router/narwhal-orchestrator, select the current router, and send requests through Narwhal. Check the request and token panels, four engine rows, pool assignments and role history.

Run narwhal dev down to stop the fleet, then close the SSH tunnels. Prometheus and Grafana retain the collected history on the monitoring host.