Controlling the fleet¶
The fleet control service runs operator test actions against a running Narwhal fleet:
- engine pause, resume, stop, start, drain and readmit
- configuration overlays
- cold restarts and baseline restores
- AIPerf load jobs
The service records each action, with its time and effect, in a private run record. The console runs on its own page. The Narwhal Orchestrator dashboard links to it and marks each action on its charts.
The service runs on the router host, listens on loopback, and authenticates API requests with a bearer token. Operators reach the console and Grafana from a workstation through the operator tunnel.
Hooks are the commands, defined in the private configuration, that perform each change, such as stopping an engine or restarting the router. Drains and readmits use the router's lifecycle API instead.
Work happens in sessions. Starting a session copies the baseline fleet configuration. Ending it closes the session's record and leaves the fleet as it is. Restore baseline undoes the session's changes.
Walkthrough¶
Pages¶
-
Setting up the control service
Write the private configuration, set the token, and start the service.
-
Reach the console through the tunnel and choose how it authenticates.
-
Using the console from Grafana
Open the console from the Narwhal Orchestrator dashboard and mark actions on its charts.
-
Start and end sessions, read the session strip, and follow the activity log.
-
Pause, stop, drain and readmit engines.
-
Run AIPerf workloads against the router and read the results.
-
Overlays, cold restarts and restores
Change serving settings mid-session, restart the fleet, or return to the baseline.
-
Empty panels, missing chart markers, and engines stuck after a drain or stop.
-
Configuration keys, the hook contract and hook environment.
-
Routes, request fields, status codes and metrics.
-
Session files, the
run.jsonschema and journal extracts.