phoebus_os/docs/adr/0012-nvidia-driver-policy.md
Phoebus 3b1b54ffb9
Some checks are pending
ci / test (push) Waiting to run
docs: reconcile NVIDIA, verification and harness status
Record the reliability pass across STATUS, HANDOFF, CHANGELOG, ADR 0012,
the live README and the boot-verification log: the deterministic live GPU
decision with a real nouveau fallback, the 580 legacy cap and mixed
open/proprietary incompatibility, the compatibility verifier, the
installed lifecycle fix, the repaired GUI harness, the automated but
unrun rollback gate, and the scoped live sudoers. Withdraw the stale
handoff claim that the gui_bridge tests hang (the workspace suite
passes) and record the installed-target missing-sys gap.
2026-09-23 06:28:30 -07:00

8.8 KiB

ADR 0012: NVIDIA accelerated-driver policy and payload

  • Status: Accepted
  • Date: 2026-09-23

Context

Phoebus must run NVIDIA's official accelerated driver stack on both targets:

  • the installed Gentoo system, through Portage's x11-drivers/nvidia-drivers;
  • the Debian-based live ISO, without downloading or compiling a driver at boot.

NVIDIA's Linux driver has three relevant regimes (verified 2026-09 against NVIDIA/Gentoo and the PCI ID database):

  • Turing (GTX 16xx / RTX 20xx) through Hopper can use NVIDIA's open kernel modules (kernel-open). The open modules are recommended from 560 and mandatory from 595.
  • Blackwell (RTX 50xx) requires the open kernel modules; the proprietary kernel modules cannot drive it.
  • Maxwell, Pascal and Volta cannot use the open modules and need the proprietary modules. NVIDIA's 580 series is the last to support them; the next series drops those architectures.
  • Fermi, Kepler and older have no security-supported proprietary branch; the in-tree nouveau driver is the honest choice.

The prior code classified GPUs only by vendor name and set VIDEO_CARDS="nvidia" for any NVIDIA device, with no branch/flavor decision and no live payload. This is unsafe: it can install a driver that cannot drive the GPU, or claim success while the GPU is unserved.

Decision

1. Classify from the PCI device ID, not marketing names

A generated table (phoebus_core::nvidia_devices, produced by scripts/generate-nvidia-devices.py from the public PCI ID database) maps an NVIDIA display device ID to its chip generation via the device's NVIDIA codename. The probe records vendor:device and the PCI address for each VGA/3D/Display controller. Unknown IDs are reported as unknown and never claimed as supported.

2. Deterministic branch/flavor selection

One driver install covers every adapter, so the branch is computed across the whole device set rather than assuming the oldest card's branch serves the newer ones:

Generation Branch Module flavor Portage
Turing / Ampere / Ada / Hopper current open (kernel-open) x11-drivers/nvidia-drivers ~amd64
Blackwell current open required (kernel-open) x11-drivers/nvidia-drivers ~amd64
Maxwell / Pascal / Volta proprietary (580 is the last series) proprietary (-kernel-open) mask >=x11-drivers/nvidia-drivers-581
Fermi / Kepler and older none (nouveau) — no NVIDIA package

A Blackwell GPU (open-only) together with a Maxwell/Pascal/Volta GPU (proprietary-only) has no single compatible branch. The plan reports that incompatibility explicitly and leaves the adapters on nouveau instead of installing one driver that cannot accelerate one of the cards. Pascal plus a Turing-class GPU stays proprietary and supported, because the newer generation accepts either flavor.

Hybrid Intel/AMD GPUs keep their VIDEO_CARDS entries; the NVIDIA decision only adds nvidia, or nouveau when the card is unserved.

3. The AI cannot change it

The driver branch and module flavor are compatibility/security decisions. There is no AI action for them, set_package_use changes touching x11-drivers/nvidia-drivers are refused by the fixed-policy check, and the advisor prompt states the rule. The model may explain the choice, never override it.

4. Installed Gentoo integration

  • Scoped package.use (dist-kernel, kernel-open/-kernel-open), a scoped ~amd64 keyword for the driver atom, and a scoped package.mask for the legacy branch.
  • A modprobe.d policy (nouveau blacklist, nvidia_drm.modeset=1, nvidia-uvm softdep) and a dracut force_drivers fragment, both written before the kernel is built.
  • A dedicated nvidia_drivers install stage runs after the kernel and before boot: it emerges the driver and regenerates the initramfs so the modules and firmware are embedded. Its failure fails the install; the resolved choice is recorded in /etc/phoebus/nvidia and shown in the plan.
  • sys update system runs @module-rebuild and kernel-install add-all on NVIDIA systems, so a kernel update rebuilds the modules and regenerates the initramfs before the next boot.

5. Live ISO payload

  • The signed Debian contrib/non-free/non-free-firmware components supply the driver. Debian 13/testing ships branch 550.163.01, whose proprietary modules cover Maxwell through Hopper. Blackwell is unsupported by that branch; on live it falls back to nouveau, reported, never silently claimed.
  • The live driver choice is deterministic and device-based: phoebus-installer --live-gpu-plan enumerates the PCI display controllers (class 0x03, vendor 0x10de), classifies them against the generated device table, and decides load-nvidia, nouveau-fallback, no-device or disabled. It never matches on a vendor name alone.
  • nouveau is blacklisted so it cannot auto-bind a supported GPU, but an unsupported GPU (or a failed accelerated load) explicitly modprobes nouveau, so the documented fallback is real. The fail-safe GRUB entry sets nomodeset phoebus.nvidia=off; the load script and the unit both stand down on that command line, so recovery intent is authoritative.
  • The live policy file etc/modprobe.d/phoebus-nvidia.conf is created by build.sh (not only on the installed Gentoo target) and replaces Debian's unconditional nouveau configuration.
  • The live kernel is pinned to Debian stable (trixie, 6.12 LTS). Testing's current kernel (7.2) removed the strncpy declaration the 550 driver still uses, so nvidia-kernel-dkms cannot build against it; the rolling kernel metas are filtered out and the exact stable kernel/headers/kbuild packages are resolved from trixie and pinned in an apt setup hook.
  • livefs-hook.sh builds the DKMS modules for that exact kernel and fails if any module is missing. Debian renames them nvidia-current*; the hook and the verifier accept either that or the upstream names.
  • build.sh runs --verify-live-nvidia on every build (including PHOEBUS_REUSE_ROOTFS=1). It is a compatibility gate, not a presence list: it rejects placeholder modules/libraries, requires every module to be registered in the live kernel's modules.dep, normalises the dpkg revision and requires the userspace libnvidia-glcore and firmware-directory versions to match the bundled driver, infers and checks the module flavor, requires a non-empty GSP firmware and the exact etc/modprobe.d/phoebus-nvidia.conf, and keeps every read rooted inside the build root. It records the bundled driver version, module flavor and kernel in /etc/phoebus/nvidia. Any missing or mismatched component fails the build.
  • Nouveau blacklisting and module options come from Debian's own nvidia-kernel-common. Its unconditional modules-load entry is removed in favour of a gated service that loads the driver before greetd/Mango only when an NVIDIA display device is present, so Intel/AMD/virtual GPUs keep their normal boot and no failed unit is recorded. The documented fail-safe entry disables modesetting and blacklists the NVIDIA modules.
  • Debian live packages and configuration never enter the Gentoo target.

A minimal live rootfs was built and its NVIDIA payload verified this session (driver 550.163.01, kernel 6.12.107+deb13-amd64, modules nvidia-current*).

Consequences

  • The live image grows by the NVIDIA driver, firmware and userspace payload.
  • The live branch (550) cannot accelerate Blackwell; the installed Gentoo system can (open modules from the current branch). This asymmetry is documented.
  • The live kernel is held at Debian stable rather than testing. This is deliberate: it is both the project's stability preference and the only way the Debian 550 driver builds. It must be revisited when Debian ships a driver whose kernel support matches its testing kernel.
  • The installed system may compile the driver from source; the dist-kernel flag plus @module-rebuild keep it coherent across kernel updates.
  • The generated device table must be regenerated when NVIDIA releases new hardware; unknown IDs degrade safely to "unknown/nouveau" rather than a wrong claim.
  • No NVIDIA-accelerated boot has been observed: this host has no NVIDIA GPU and QEMU's virtual GPU cannot establish it. A minimal live rootfs with the driver was built and its payload verified, but the full ISO and the live/installed NVIDIA gates are unverified here. This ADR records the design, not a claim of working hardware acceleration.

Non-goals

  • NVIDIA's standalone .run installer (never; packages only).
  • Bundling multiple driver branches in one live rootfs (they conflict).
  • Forcing closed modules onto hardware that needs open modules, or vice versa.
  • A global ~amd64 or a global dist-kernel switch.