Narwhal documentation¶
Narwhal is a Python framework for disaggregated LLM inference. It assigns prefill and decode roles across dual-capability engines as demand changes. Read the README for an overview and the product contract.
Run Narwhal¶
-
Install the wheel and run the installed commands.
-
Try Narwhal on one workstation
Set up four llama.cpp engines on a GPU or the CPU, send prompts and watch the roles change.
-
Install the CUDA runtime, start a local fleet, and qualify it.
-
Take a fleet from host discovery through serving and capacity validation.
-
Monitor routers, restart engines, upgrade the deployment, and control the fleet.
-
Profile engines, select targets, and run load and benchmark trials.
-
Export metrics to Prometheus and read the Grafana dashboard.
-
Diagnose overload, engine failures, and router failures.
Reference¶
-
Request flow, topology, role control, failure, and state contracts.
-
Fleet fields, deployment inputs, and validation rules.
-
Commands, options, results, and exit codes.
-
Completion, inspection, live-state, and lifecycle routes.
-
Journal, profile, metric, failure, and compatibility contracts.
Read the contributor guide, code of conduct, and security policy before contributing.