Serve and measure¶
Starting and locally verifying the router¶
Replace <served-model> with the fleet model.
-
In the router shell, start the router:
-
From another router-host shell, check the router:
curl -fsS http://127.0.0.1:8000/health curl -fsS http://127.0.0.1:8000/ready curl -fsS http://127.0.0.1:8000/narwhal/state | python3 -m json.tool curl -fsS http://127.0.0.1:8000/metrics curl -fsS http://127.0.0.1:8000/v1/completions \ -H 'content-type: application/json' \ -d '{"model":"<served-model>","prompt":"Return one sentence about narwhals.","max_tokens":32}' -
Retain these responses before load:
Response Shows /healthLiveness status and instance counts /readyAdmission state /narwhal/stateEngine inventory and role split /metricsRouter metrics /v1/completionsOne successful completion that increments servedin/narwhal/state
Starting the monitoring stack on the router¶
In the router shell, run:
Retain the target-discovery and dashboard-verification output.
Tunnelling router, Prometheus, and Grafana to the workstation¶
- On the workstation, open a terminal in the checkout with
.envloaded. -
Open the tunnel:
-
Keep the tunnel terminal open.
-
From another workstation shell, check through the tunnel:
-
Open Grafana at:
-
Point the workload client at
$NARWHAL_TRIAL_URL. -
Record:
- The router role assignment.
- The workstation hostname.
- The source revision.
- The tunnel mappings.
The tunnel listens on workstation loopback ports and forwards them to 127.0.0.1 on the router. It verifies the router against the recorded SSH host key, authenticates with the router role's configured password, key, or agent, and logs to runs/access-<id>/. Latency through the tunnel includes SSH network and encryption overhead.
Troubleshooting the tunnel¶
If a workstation port is already in use:
- Pick another local port in
--forward. - Match the client URL to it.
If a forwarded request fails after SSH connects, inspect the listener on the router host.
If a service listens on an address other than the router's 127.0.0.1, pass one tunnel per remote address, each with --remote-address.
Running the initial capacity trial¶
The trial requires the capacity-trial prefix-caching setting on every engine.
Run the trial:
- Confirm that each engine's
checked.jsonrecord shows"prefix_caching": false. - Freeze the deployment evidence under a new deployment identifier.
- Attach the passing preflight result to the deployment evidence.
- Retain the monitoring startup output and the router and engine scrape evidence.
- From the workstation, run the synthetic load trial through
$NARWHAL_TRIAL_URL, against its TTFT, TPOT, and attainment thresholds. - Retain the workload definition, the request-level records, and the summaries.
- Capture the client CPU, memory, network, and scheduler behaviour.
- Confirm the drained, idle router state in
state-after.json. - Reconcile every offer against the client and router terminal classes.
- Query the engine, request, token, role, and pool-load series in Grafana.
-
Run the post-load KV ring on the router:
The trial passes when steps 5 to 11 succeed.
Close the trial:
-
Retain with the private deployment record:
- The service locations.
- The approved source revision.
- The fleet configuration.
- The profiles.
- The router journal.
- The monitoring endpoints.
-
Save the client records and the post-load KV ring output.
- Stop the client.
- Press Ctrl+C in the tunnel terminal when private access ends.
- Leave the engines, attestation sidecars, router, and monitoring stack running until a planned drain or shutdown in Operating Narwhal.
- Record the final gate against the evidence and recovery index.