Qualify the transfer fabric¶
Measure each directed host edge against its source cache group's budget while the fleet is idle.
Building the source budget¶
The calculate command sizes the budget from these initial trial parameters:
| Parameter | Value | calculate flag |
|---|---|---|
| Remote KV handoffs per second | 1 | --handoffs-per-s 1 |
| Prompt tokens per handoff | 1,024 | --prompt-tokens 1024 |
| Burst size | 1 | --burst 1 |
| Transfer budget | 1 s | --transfer-budget-s 1 |
| Bandwidth headroom | 25% | --headroom 1.25 |
Each outgoing directed edge from the cache group's roles carries the full handoff rate.
-
In the representative's engine-role shell, calculate the budget. On vLLM, calculate it from the captured cache layout:
python3 "$NARWHAL_FABRIC_BUDGET_TOOL" calculate \ --model-config "$NARWHAL_MODEL_DIR/config.json" \ --launch-config "$NARWHAL_ENGINE_LAUNCH_CONFIG" \ --runtime-layout "$ENGINE_RUN/cache-layout.json" \ --prompt-tokens 1024 --handoffs-per-s 1 --burst 1 \ --transfer-budget-s 1 --headroom 1.25 \ --out "$FABRIC_RUN/budget.json"On SGLang, the representative is a prefill engine. Set
PREFILL_IPto this engine's fabric address andDECODE_URLto the base URL of another idle engine in the group, then measure eight handoffs and calculate the budget from them:export PREFILL_IP='<this engine fabric address>' export DECODE_URL='http://<decode engine address>:<decode engine port>' python3 "$NARWHAL_FABRIC_BUDGET_TOOL" measure-transfer \ --prefill "http://$PREFILL_IP:$NARWHAL_ENGINE_PORT" --decode "$DECODE_URL" \ --bootstrap-host "$PREFILL_IP" --bootstrap-port "$NARWHAL_KV_SIDE_CHANNEL_PORT" \ --model "$NARWHAL_ENGINE_MODEL_NAME" --launch-config "$NARWHAL_ENGINE_LAUNCH_CONFIG" \ --prompt-tokens 1024 --out-dir "$FABRIC_RUN/transfer" python3 "$NARWHAL_FABRIC_BUDGET_TOOL" calculate \ --model-config "$NARWHAL_MODEL_DIR/config.json" \ --launch-config "$NARWHAL_ENGINE_LAUNCH_CONFIG" \ --transfer-metrics "$FABRIC_RUN/transfer/before.prom" "$FABRIC_RUN/transfer/after.prom" \ --prompt-tokens 1024 --handoffs-per-s 1 --burst 1 \ --transfer-budget-s 1 --headroom 1.25 \ --out "$FABRIC_RUN/budget.json" -
On vLLM, verify that every tensor parallel (TP) rank shares one cache layout:
python3 - <<'PY_REPRESENTATIVE_BUDGET' import json import os from pathlib import Path run = Path(os.environ["FABRIC_RUN"]) budget = json.loads((run / "budget.json").read_text()) layout = json.loads((Path(os.environ["ENGINE_RUN"]) / "cache-layout.json").read_text()) names = {rank["kv_cache_layout"] for rank in layout["ranks"]} if len(names) != 1: raise SystemExit("Inspect the differing TP cache layouts before using one group budget.") print(f"required_gbps={budget['required_gbps']}; kv_cache_layout={names.pop()}") PY_REPRESENTATIVE_BUDGET sha256sum "$FABRIC_RUN/budget.json" -
For each representative, retain:
FABRIC_RUNENGINE_RUN- The budget digest
required_gbps- The resolved layout
- The cache-group signature
-
For each role that uses another host's representative budget, create a local comparison budget from the representative's
required_gbpsand budget digest:umask 077 mkdir -p runs export FABRIC_RUN="$(mktemp -d runs/fabric-XXXXXX)" export REQUIRED_GBPS='<required_gbps from representative budget.json>' export REPRESENTATIVE_BUDGET_SHA256='<SHA-256 of representative budget.json>' python3 - <<'PY_EDGE_BUDGET' import json import os from pathlib import Path required = float(os.environ["REQUIRED_GBPS"]) if not 0 < required < float("inf"): raise SystemExit("Use the positive finite rate from the representative budget.") digest = os.environ["REPRESENTATIVE_BUDGET_SHA256"] if len(digest) != 64 or any(char not in "0123456789abcdef" for char in digest): raise SystemExit("Use the representative budget's SHA-256 digest.") path = Path(os.environ["FABRIC_RUN"]) / "budget.json" with path.open("x") as stream: json.dump({"required_gbps": required, "representative_budget_sha256": digest}, stream) stream.write("\n") path.chmod(0o600) PY_EDGE_BUDGET -
Create a new
FABRIC_RUNfor each repeat comparison.
The required link rate, in decimal Gbit/s, is:
8 * payload_bytes *
max(handoffs_per_second, burst_handoffs / transfer_budget_seconds) *
headroom / 1e9
payload_bytes is payload_bytes_per_handoff in budget.json.
Binding each sample to a directed route¶
Record routes with ip route get, ip -4 route get, or ip -6 route get.
-
In both engine-role shells, run the edge variable setup and
TEST_PORTlistener check: -
When
ssshows a listener on either host, pick anotherTEST_PORT. - Allow
TEST_PORTtraffic between the two fabric addresses for the test. -
Record the route on the source:
-
Check the reverse route on the destination:
-
Copy the destination's exact one-line reverse route into
$EDGE_PREFIX.destination-route.txton the source. - Confirm that each route selects
NARWHAL_FABRIC_INTERFACEand the discovered source address.
Measuring ucx_tcp¶
-
Install
iperf3on each host that is missing it: -
Record
iperf3 --versionon both hosts. -
On the destination, start the server:
-
On the source, run the client with one TCP stream per TP rank:
export FABRIC_STREAMS="$(python3 -c 'import json,os; print(json.load(open(os.environ["NARWHAL_ENGINE_LAUNCH_CONFIG"]))["tensor_parallel_size"])')" export EDGE_SAMPLE="$EDGE_PREFIX.json" (set -o noclobber; iperf3 --client "$DEST_IP" --bind "$SOURCE_IP" \ --port "$TEST_PORT" --parallel "$FABRIC_STREAMS" --omit 3 --time 10 \ --json > "$EDGE_SAMPLE") -
On the source, run the link fingerprint and budget check:
python3 "$NARWHAL_FABRIC_BUDGET_TOOL" link \ --source-role "engine-$SOURCE_NODE" --destination-role "engine-$DEST_NODE" \ --source-address "$SOURCE_IP" --destination-address "$DEST_IP" \ --source-interface "$NARWHAL_FABRIC_INTERFACE" \ --destination-interface "$NARWHAL_FABRIC_INTERFACE" \ --source-route "$EDGE_PREFIX.source-route.txt" \ --destination-route "$EDGE_PREFIX.destination-route.txt" \ --transport ucx_tcp --tool-version "$(iperf3 --version | head -n 1)" \ --test-parameters "{\"parallel\":$FABRIC_STREAMS,\"omit_s\":3,\"duration_s\":10,\"port\":$TEST_PORT}" \ --out "$EDGE_PREFIX.link.json" python3 "$NARWHAL_FABRIC_BUDGET_TOOL" record-edge \ --link "$EDGE_PREFIX.link.json" --sample "$EDGE_SAMPLE" \ --budget "$FABRIC_RUN/budget.json" --out "$EDGE_PREFIX.evidence.json" -
Stop the temporary
iperf3server. - Measure the reverse edge against the new source's budget.
record-edge takes the measured ucx_tcp rate from end.sum_received.bits_per_second.
record-edge exit code |
Meaning |
|---|---|
| 0 | The measured rate meets the budget. |
| 1 | The measured rate is below the budget. |
| 2 | An input file is invalid, or the output file already exists. |
Measuring ucx_rdma¶
The test measures one-way RDMA writes between host-memory buffers. On RDMA over Converged Ethernet (RoCE), steps 4 to 7 select a global identifier (GID) index. On native InfiniBand, use the site's active port and GID selection.
-
Install
perfteston each host that is missing it: -
Record
ib_write_bw --version. - On each host, set
HCAandHCA_PORTfromtransfer.net_devices, such asHCA=mlx5_0andHCA_PORT=1formlx5_0:1. -
For RoCE, inspect:
-
Set
GID_INDEXto the index matching the configured fabric interface, address, and RoCE mode. - Retain each host's GID mapping.
- In the source shell, set
DEST_HCA,DEST_HCA_PORT, andDEST_GID_INDEXfrom the destination's inspected mapping. -
Select the address-family flags:
-
On the destination, start the server:
-
On the source, run the client:
-
Set
MEASURED_GBPSto the report'sBW average[Gb/sec]value. -
Run the link fingerprint and budget check:
python3 "$NARWHAL_FABRIC_BUDGET_TOOL" link \ --source-role "engine-$SOURCE_NODE" --destination-role "engine-$DEST_NODE" \ --source-address "$SOURCE_IP" --destination-address "$DEST_IP" \ --source-interface "$NARWHAL_FABRIC_INTERFACE" \ --destination-interface "$NARWHAL_FABRIC_INTERFACE" \ --source-route "$EDGE_PREFIX.source-route.txt" \ --destination-route "$EDGE_PREFIX.destination-route.txt" \ --transport ucx_rdma --tool-version "$(ib_write_bw --version | head -n 1)" \ --test-parameters "{\"source_hca\":\"$HCA\",\"source_port\":$HCA_PORT,\"source_gid\":$GID_INDEX,\"destination_hca\":\"$DEST_HCA\",\"destination_port\":$DEST_HCA_PORT,\"destination_gid\":$DEST_GID_INDEX,\"message_bytes\":1048576,\"duration_s\":10,\"port\":$TEST_PORT}" \ --out "$EDGE_PREFIX.link.json" python3 "$NARWHAL_FABRIC_BUDGET_TOOL" record-edge \ --link "$EDGE_PREFIX.link.json" --sample "$EDGE_SAMPLE" \ --gbps "$MEASURED_GBPS" --budget "$FABRIC_RUN/budget.json" \ --out "$EDGE_PREFIX.evidence.json" -
Measure the reverse edge against the new source's budget.
- For multi-rail deployments, test every selected HCA port.
- Retain every report.
Completing the matrix and matching retained evidence¶
This matrix qualifies KV handoffs between engine hosts. Preflight qualifies handoffs within one host.
For n distinct engine hosts, qualify the n * (n - 1) directed host pairs.
For every edge, retain:
- The source and destination roles.
- The source revision.
- The source and reverse routes.
- The transport and utility version.
- The exact command.
- The budget digest.
- The sample path and exit status.
When the runtime layout changes, recalculate the budget from the new cache-layout.json on vLLM, or from new measured handoffs on SGLang.
Link inputs:
- The host assignment
- Both routes
- The interfaces
- The transport
- The utility version
- The measurement parameters
When any link input changes, collect a new directed sample. When all link inputs are identical, reuse the retained sample:
- Set
CURRENT_EDGE_PREFIXto the current link record prefix. - Set
RETAINED_EDGE_PREFIXto the retained evidence prefix. - Recreate the current link fingerprint with
link. -
Compare the retained sample with the recalculated budget:
For ucx_tcp, --sample takes the retained iperf3 JSON. For ucx_rdma, it takes the retained .txt report.
reuse-edge exit code |
Meaning |
|---|---|
| 0 | The retained sample meets the recalculated budget. |
| 1 | The retained sample is below the recalculated budget. |
| 2 | The current link fingerprint or the retained sample differs from the recorded evidence. |
Troubleshooting an edge below its budget¶
When record-edge or reuse-edge exits 1:
- Check link speed, MTU, retransmissions or RDMA counters, CPU load, and other traffic.
- Fix the cause.
- Resample the directed edge until
record-edgeexits 0.