Container Support

Container technologies enable portable, reproducible computational workflows while integrating seamlessly with HPC cluster infrastructure. The container strategy respects “bare metal first” philosophy - containers mirror host environments rather than replacing them.

Host Environment Parity

Containers in HPC contexts serve different purposes than in cloud-native environments:

Host-integrated design

Containers access host-provided software stacks (Spack modules), GPUs, high-speed interconnects, and shared storage. They augment rather than isolate from cluster capabilities.

Bare metal first principle

Most workloads execute directly on nodes using Lmod modules. Containers address specific use cases: software portability, dependency isolation, workflow reproducibility, and complex application packaging.

Environmental transparency

Container configuration mirrors host environment conventions. Module systems, network mounts, and GPU access function identically inside and outside containers.

Note: Environmental transparency requires deliberate configuration—it doesn’t come automatically with container runtime installation. Achieving seamless integration demands careful setup of mount points, device passthrough, UID/GID mapping, and consideration of base image compatibility (glibc versions, kernel requirements). Simply installing Apptainer doesn’t guarantee that arbitrary containers (e.g., CentOS 5 images on Ubuntu 24.04 hosts) will function correctly without additional configuration work.

Container Technologies

Enroot and Pyxis

NVIDIA Enroot provides unprivileged container execution with GPU awareness, integrated with SLURM via Pyxis.

Key capabilities:

  • Unprivileged operation (no root daemon)

  • Native GPU access (full CUDA capabilities)

  • SLURM integration via Pyxis plugin

  • OCI image compatibility

  • Minimal overhead

Use cases:

  • GPU-accelerated workloads requiring specific CUDA versions

  • Deep learning frameworks with complex dependencies

  • Multi-node parallel applications in containers

  • Reproducible computational environments

Apptainer (formerly Singularity)

Apptainer provides OCI-compatible container runtime emphasizing HPC integration:

Key capabilities:

  • Unprivileged execution (no setuid required in recent versions)

  • MPI integration (host MPI can interact with container MPI)

  • GPU support (via --nv flag)

  • Bind mount flexibility

  • HPC-specific design

Use cases:

  • Legacy Singularity workflows

  • MPI applications requiring host fabric integration

  • Software distribution and archiving

  • Multi-tenant environments requiring isolation

Container Configuration

System-Level Settings

User namespace limits:

Increase default limits to support concurrent container launches:

# /etc/sysctl.d/99-containers.conf
user.max_user_namespaces=2048920  # RHEL9 default, adjust as needed

This prevents “no space left on device” errors when many users launch containers simultaneously.

Enroot Configuration

Enable full GPU capabilities:

# /etc/enroot/environ.d/19-nvidia-all-caps.env
NVIDIA_DRIVER_CAPABILITIES=all

This environment variable ensures containers access all NVIDIA driver capabilities (compute, graphics, video, utility).

GPU capability enforcement hook:

# /etc/enroot/hooks.d/98-nvidia.sh
# https://github.com/nvidia/nvidia-container-runtime#nvidia_driver_capabilities

if [ -z "${NVIDIA_DRIVER_CAPABILITIES-}" ]; then
    NVIDIA_DRIVER_CAPABILITIES="utility"
fi

for cap in ${NVIDIA_DRIVER_CAPABILITIES//,/ }; do
    case "${cap}" in
    all)
        cli_args+=("--compute" "--compat32" "--display" "--graphics" "--utility" "--video")
        break
        ;;
    compute | compat32 | display | graphics | utility | video)
        cli_args+=("--${cap}")
        ;;
    *)
        common::err "Unknown NVIDIA driver capability: ${cap}"
        ;;
    esac
done

This hook translates NVIDIA_DRIVER_CAPABILITIES into enroot’s internal flags, ensuring requested GPU features are available.

Auto-mount host directories:

# /etc/enroot/mounts.d/20-mounts.conf
/cm/local        /cm/local        none  x-create=dir,rbind,ro,nosuid,noexec,rslave  0  -1
/cm/shared       /cm/shared       none  x-create=dir,rbind,ro,nosuid,noexec,rslave  0  -1
/opt/shared      /opt/shared      none  x-create=dir,rbind,ro,nosuid,noexec,rslave  0  -1
/data            /data            none  x-create=dir,rbind,rw,nosuid,noexec,rslave  0  -1

# Security: nosuid prevents setuid execution, noexec blocks direct execution
# rslave ensures mount propagation works correctly
Rationale:

  • Researchers access Spack modules from /opt/shared inside containers

  • Shared data directories (/data, /scratch) remain accessible

  • nosuid,noexec prevent security issues with mounted filesystems

  • Read-only mounts for system directories prevent accidental modification

Apptainer Configuration

Bind mount configuration:

# /etc/apptainer/apptainer.conf
bind path = /cm/local
bind path = /cm/shared
bind path = /opt/shared
bind path = /data
bind path = /scratch

User namespace settings:

# Enable unprivileged user namespaces
allow setuid = no        # Modern Apptainer doesn't require setuid
max loop devices = 256   # Support multiple simultaneous containers

Pyxis Integration with SLURM

Pyxis enables native container execution through SLURM job scripts.

Configuration

SLURM integration:

# /etc/slurm/plugstack.conf
required /usr/local/lib/slurm/spank_pyxis.so

Pyxis settings:

# /etc/pyxis/pyxis.conf
{
    "runtime": "enroot",
    "cache_dir": "/tmp/enroot_cache",
    "enroot_path": "/usr/bin/enroot"
}

Conclusion

Container support extends HPC capabilities while maintaining host environment integration. Enroot/Pyxis and Apptainer provide complementary technologies serving different use cases. Configuration emphasizing host parity ensures containers augment rather than complicate researcher workflows. Proper implementation balances portability, reproducibility, performance, and security.