This article covers how the NVIDIA Container Toolkit is mandatory, privileged, and almost entirely unmonitored. Most detection content written for it watches binaries that no longer execute. Here is what actually runs on a modern GPU node, and how Stealthium separates legitimate hook execution from abuse.
Between "scheduler picked a node" and "my training job started", NVIDIA Container Toolkit runs as root on the host and wires a GPU into your container. Device nodes. Driver libraries. The ldcache. Symlinks that make libcuda.so.1 resolve to the right place.
This is design, not flaw. Mounting host device nodes into a namespace is a privileged operation by definition. That means on a GPU node, the most privileged (and first) code that runs most often is not your workload, nor your scheduler. It's a small family of hook binaries that fire several times per container start and then exit, in under a second, leaving almost no trace anyone is looking at. So the NVIDIA Container Toolkit is privileged and mandatory. It cannot be removed, hardened away, or swapped out, since every GPU container on the node depends on it.
Unfortunately, it's also now a proven soft spot. Count the disclosures:
- CVE-2024-0132 from September 2024, a TOCTOU container escape rated 9.0 Critical
- CVE-2025-23359 from February 2025, a bypass of the patch for the above
- NVIDIAScape (CVE-2025-23266), a three-line container escape rated 9.0 Critical, alongside CVE-2025-23267, a high-severity link-following flaw, both from July 2025
- CVE-2026-24260 arrived, an 8.5 High, again a TOCTOU race condition, from July 2026
That's five headline vulnerabilities for NVIDIA Container Toolkit in under two years.
However, traditional tooling is unable to differentiate between the Toolkit's legitimate root-level hook execution and a weaponized one. This is the challenge Stealthium set out to solve, and what led to our new approach to detecting NVIDIA Container Toolkit compromise.
How does Stealthium distinguish between attacker-driven and normal hook execution, without generating hundreds of false positives from legitimate executions? Our approach started by observing the Toolkit at work, in our lab.
What Actually Runs
On a current Toolkit, a docker run --gpus all produced six separate root executions of nvidia-cdi-hook on our test host. For a single container start, before the workload's first instruction.
Clearly, this is a privileged path at work, and a potentially noisy one. A busy inference node cycling pods produces hundreds of these executions an hour, every one of them running as UID 0 on the host, all legitimate parts of GPU container startup.
There are two paths in the wild:
The legacy path: nvidia-container-runtime-hook runs as an OCI prestart hook and shells out to nvidia-container-cli configure --pid=<N> ..., which enters the target container's namespace and mounts devices and driver libraries into it.
The CDI path: The Container Device Interface spec declares createContainer hooks: nvidia-cdi-hook create-symlinks, nvidia-cdi-hook update-ldcache, and friends, which the runtime executes inside the container's own cgroup and namespaces, as children of the container's init process, before the entrypoint execs.
On the legacy path the hook runs on the host and reaches in. On the CDI path it runs inside the container's cgroup and namespaces, as a child of container init, and is still root on the host.
A surprising finding: Across four GPU container starts on Toolkit 1.19.1, the legacy binaries never ran at all. nvidia-cdi-hook was the only hook that executed. This has an important follow-on effect: as CDI is the default path now, any detection content written against the older binary names, nvidia-container-runtime-hook, nvidia-container-cli, watches a code path that no longer executes. Outdated detections based on the legacy path will report all clear whilst blind to compromise of the new. Detections thus require an understanding of both privileged paths.
Avoiding False Positives
A detection has to be more specific than the Toolkit running as architected. As above, that fires six times per container start on a healthy node. Detection must key on how and what the hook ran, not whether it ran.
It's the same detection challenge we wrote about with Januscape. The mechanism an attacker is abusing is indistinguishable from the mechanism the platform is for. You can't separate them by watching whether the thing happened. You have to watch how it happened.
Stealthium establishes what normal hook execution looks like: a clean environment, an expected binary path, and an expected invoker (either container runtime or container's init), and we alert on deviations.
Three Approaches To Deviation Detection
A dynamic linker variable in the hook's environment
If a hook execs carrying LD_PRELOAD, LD_AUDIT, or LD_LIBRARY_PATH, whatever it loads runs as root on the host. In our lab captures, real hook executions carry none of these, the environment is empty. Anything present is already anomalous.
With NVIDIAScape, the Toolkit's createContainer hook inherits the container image's environment and working directory, so a malicious image sets LD_PRELOAD=/proc/self/cwd/poc.so and the privileged host-side hook obligingly loads attacker code out of the container's rootfs. Typically, NVIDIAScape detections inspect escaped container processes.
We designed our detection to provide signal a level above. Stealthium inspects the hook itself, which is already host root, and does not require the preload to be working-directory-relative, or to look like any published exploit. Injection into a process that is root on the host is worth alerting on regardless of which technique put it there.
A hook binary running from outside the Toolkit's install directories
Next up, Stealthium hunts for attackers trying to do sneaky swap outs. We search for either nvidia-cdi-hook planted earlier on the runtime's PATH, or a CDI spec rewritten to point somewhere else. The only reason for either of these would be to establish persistence or escalation, as the runtime will faithfully execute that binary as root.
Unusual invokers
Normal hooks are exec'd by runc, crun, containerd-shim, dockerd, podman, or by another Toolkit binary. The CDI hooks are exec'd by the container's own init. Stealthium hunts for anything beyond that: a shell, a compromised service, a payload. Outside the norm is signal of potential abuse of what should be a legitimate, correctly-installed hook.
Here is a Stealthium High severity detection in full. Note the Alert summary as well as the Event Timeline:
You might notice that is the legacy binary, on a node where the legitimate path is CDI. That mismatch is our signal — nothing on that host should be executing nvidia-container-cli at all.
Toolkit Is A Slippery Target
Linux gives you fifteen characters of process name. As such, nvidia-container-runtime-hook and nvidia-container-cli both truncate to nvidia-containe, indistinguishable. Anything keyed on comm can't separate the privileged namespace-mounting path from the rest of the family, and might match things it never meant to. (A small break for operators: nvidia-cdi-hook is exactly fifteen characters, that's about the only convenient thing in this entire surface.)
Given this challenge, Stealthium matches on the executable path, which is unambiguous, and keeps one canonical set of Toolkit binaries that every Toolkit-aware detection reads from. Three consume it today. The hook monitor described above, which decides whether a given execution is normal. The container escape attribution chain, which walks a suspect process's ancestry back to the hook that spawned it, so an escape is traced to the container start that enabled it rather than to whatever the process happens to be doing when it surfaces. And the driver ioctl suppression list, which keeps legitimate Toolkit device setup from burying real driver abuse in noise. One list, three consumers, one place to update when NVIDIA ships the next hook.
A note on this last point: per-detection copies of a fast-moving vendor's binary names are a bug waiting to land. Vendor naming is a dependency, and should be treated as such. If you own GPU and/or container detection, this might be a good time to check which names it keys on, whether it reads comm or the full path, and whether it knows what CDI is.
Attribution Alongside Detection
GPU telemetry can make attribution difficult. Whether it's a CUDA event, kernel launch, device allocation, host-to-device copy or library load, if you ask "which workload did this", you get a process ID (PID) and little else. No container. No cgroup. NVML's per-process utilization samples are similar: a PID, an SM percentage, some bytes. After all, the GPU has no concept of a container, so neither does anything the GPU tells you.
For operators and defenders, we need to know both GPU execution and responsible container. Which tenant is holding the GPU? Which pod copied 40GB off the device? Which image loaded that CUDA library?
Stealthium correlates that host-side. The agent records the cgroup of every process at exec, and inherits it across forks. The server resolves that cgroup to a container and, on Kubernetes, to a namespace, pod, and node.
As a result, Stealthium surfaces every GPU event alongside its container and pod identity. The same join also gives us per-container GPU utilization.
NVML reports SM, memory, encoder, and decoder utilization per PID; Stealthium rolls those up per container and per physical GPU, so we can provide detail like "this pod is at 70% SM on the A100 in slot 3b". Our customers running multi-tenant GPU nodes have needed that number for a long time.
Finally, as it matters here, because hook execs are ordinary process events, the same attribution applies to them. No longer is a Toolkit hook a lonely orphaned PID in a log. It is an execution inside a known container, on a known node, in a known namespace.
Resource details from our alert from earlier.
Catching NVIDIA Container Toolkit Compromise
If you are running Stealthium on GPU infrastructure, the good news is this is already on. There is nothing to enable and no rule to tune. Toolkit hook executions are attributed to their container and pod, the anomalous ones alert, and the normal several-hundred-per-hour correctly stay silent.
The bad news for many is that they might be living with GPU-Blindness, with little understanding of the correlation between GPU runtime and the containers invoking them.
GPU infrastructure is full of privileged, high-frequency, setup paths that make accelerators usable from containers. The Toolkit hooks are one instance. Driver ioctl paths are another. The /dev/nvidia* device surface is a third. An attacker does not need a novel exploit to abuse any of them. They need the security tooling to be looking somewhere else, which, structurally, it is: your EDR sees a short-lived root process called nvidia-cdi-hook and correctly concludes it is part of the container runtime. It is right, and it may have just let compromise run undetected. This blog is just one illustration of an attacker pathway to GPU compromise, where many exist.
Stealthium is built to secure accelerator runtime. Every Toolkit hook execution attributed to its container and pod, the anomalous ones alerting, the several-hundred-per-hour of legitimate ones staying silent.
AI is only as secure as the layer it runs on, and on a GPU node part of that layer runs as root six times per container start. The question isn't whether your stack can catch a weaponized toolkit hook. It's what else is running as root on your most critical compute layer that you can't see.