Installing the Narwhal dev CUDA runtime¶
Requirements:
- An Ubuntu shell, native or under WSL2
- Linux x86-64 for the GGUF plugin wheel
- Python 3.12
- An NVIDIA GPU and driver that support the CUDA runtime of the pinned packages
Install from a checkout on the Linux filesystem:
-
Install Narwhal and the pinned packages:
python3.12 -m venv .venv-dev source .venv-dev/bin/activate python -m pip install . python -m pip install 'vllm==0.30.0' 'torch==2.13.0' \ 'transformers==5.17.0' 'nixl==1.4.1' 'nixl-cu13==1.4.1' python -m pip install \ 'https://github.com/vllm-project/vllm-gguf-plugin/releases/download/v0.0.5/vllm_gguf_plugin-0.0.5-cp310-abi3-manylinux_2_28_x86_64.whl' -
Check the installed requirements:
A consistent environment prints:
-
Overwrite the wheel's Python files with the pinned GGUF plugin sources:
mkdir -p runs git clone https://github.com/vllm-project/vllm-gguf-plugin.git runs/gguf-plugin git -C runs/gguf-plugin checkout d4c1f0d082fc7cd4350da56689109a01c1f29d6c python - <<'PY' from importlib.metadata import distribution from pathlib import Path import shutil source = Path('runs/gguf-plugin/vllm_gguf_plugin') target = Path(distribution('vllm-gguf-plugin').locate_file('vllm_gguf_plugin')) for path in source.rglob('*.py'): destination = target / path.relative_to(source) destination.parent.mkdir(parents=True, exist_ok=True) shutil.copyfile(path, destination) PY -
Download the model, its multimodal projector and the tokenizer into the Hugging Face cache:
Pinned runtime rules:
narwhal dev initrequires the package versions and GGUF plugin hashes of the selected template.- Reapply the pinned plugin sources after every plugin wheel reinstall.
- The GGUF plugin reads the multimodal projector
mmproj-F16.gguffrom the directory of the GGUF file.
For a model outside the standard Hugging Face cache, pass the GGUF file with --model and the tokenizer directory with --model-dir to narwhal dev init.