Skip to content

Monitoring a WSL2 development fleet

Prerequisites

  • monitor is the SSH alias of the Prometheus and Grafana monitoring host.
  • ssh monitor true succeeds from both WSL2 and Windows.
  • A four-engine WSL2 fleet, such as the RTX 5090 reference, started with narwhal dev up passes narwhal dev verify.
  • The monitoring host has a Narwhal checkout at ~/projects/Narwhal.

Connecting metrics and dashboard ports

Every installed template places the router on port 18000 and the engine HTTP ports on 18101 upward, one per engine. With narwhal dev init --port-base P, the router uses port P and the engine HTTP ports start at P+1.

  1. Open the reverse tunnel from WSL2 for the router port and the four engine HTTP ports:

    ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 \
      -R 127.0.0.1:18000:127.0.0.1:18000 \
      -R 127.0.0.1:18101:127.0.0.1:18101 \
      -R 127.0.0.1:18102:127.0.0.1:18102 \
      -R 127.0.0.1:18103:127.0.0.1:18103 \
      -R 127.0.0.1:18104:127.0.0.1:18104 \
      monitor
    
  2. Open the browser tunnel from Windows PowerShell:

    ssh -NT -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 `
      -L 127.0.0.1:33000:127.0.0.1:3000 `
      -L 127.0.0.1:39090:127.0.0.1:9090 monitor
    

Starting or updating the monitoring stack

Start a new stack from the monitoring host's checkout:

NARWHAL_ROUTER_URL=http://127.0.0.1:18000 make observe

Prometheus reads the engine targets from the router through the tunnel and scrapes the tunnelled engine ports.

Checking scrapes and routed activity

Query Prometheus from the monitoring host:

curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=up{job=~"narwhal-router|engines"}'
curl -fsSG http://127.0.0.1:9090/api/v1/query \
  --data-urlencode 'query=narwhal_router_ready{job="narwhal-router"}'

The up query returns five healthy series, one for the router and one for each of the four engines. The narwhal_router_ready query returns 1.

Check routed traffic:

  1. Open http://127.0.0.1:33000/d/narwhal-router/narwhal-orchestrator in a Windows browser.
  2. Select the current router.
  3. Send requests through Narwhal.
  4. Check these panels:
    • Request outcomes and Token throughput
    • The four rows of the engine table
    • Pool assignments
    • Engine role history

Stopping the fleet

  1. Run narwhal dev down in WSL2.
  2. Close both SSH tunnels.