Error when downloading NVIDIA NGC Containers

Environment

Slurm with Enroot container runtime

NVIDIA NGC containers

Issue

When trying to run NGC containers, you see this error:

$ srun --container-image docker://nvcr.io/nvidia/nvhpc:24.3-devel-cuda12.3-ubuntu22.04
# or
$ srun --container-image nvcr.io/nvidia/nvhpc:24.3-devel-cuda12.3-ubuntu22.04

slurmstepd: error: pyxis: [ERROR] URL https://registry-1.docker.io/v2/nvcr.io/nvidia/nvhpc/manifests/24.3-devel-cuda12.3-ubuntu22.04 returned error code: 401 Unauthorized

Resolution

Replace forward slash (/) with hash (#) in the container path:

Instead of:

--container-image nvcr.io/nvidia/nvhpc:24.3-devel-cuda12.3-ubuntu22.04

Use:

--container-image nvcr.io#nvidia/nvhpc:24.3-devel-cuda12.3-ubuntu22.04

Example working command:

$ srun --account=YOUR_ACCOUNT \
    --nodes=1 \
    --gpus-per-node=1 \
    --container-image nvcr.io#nvidia/nvhpc:24.3-devel-cuda12.3-ubuntu22.04 \
    nvidia-smi

Note

This applies to all NGC containers, not just NVHPC

Root Cause

Pyxis container runtime requires different URL formatting than Docker: - Docker uses: nvcr.io/nvidia/… - Pyxis requires: nvcr.io#nvidia/…

References

NGC container repository