HPC Administration Guide

Contents

• System Config
  • BIOS and Firmware
    • AMD CPU specific
    • Intel CPU specific
  • Drivers
    • Kernel Built-in
    • Mellanox Driver
    • Nvidia GPU Driver
  • Kernel
    • Kernel Selection
    • Time Synchronization
    • Entropy Generation
    • Swap
    • CGroup
    • IRQ Handling
    • CPU Power Management
    • General Tunables
    • Huge Page Management
  • TCP/IP Stack
    • TCP/IP
    • RoCEv2
  • File Systems
    • NFS Client Tuning
    • Lustre Client Tuning
  • User Environment
    • Environment Defaults
    • Resource Limits and Quotas
  • SLURM
    • SLURM Environment
    • Resource Control
    • Restrict SSH Access
    • OOM Handling
    • Module System
  • Protective Measures
    • Temperature Protection
    • Kernel Lockup Recovery
• HPC Software Ecosystem
  • Design Philosophy
    • Software Provision Layers
    • Rationale: “Software-Libraries-as-a-Service”
    • Why Testing Matters
  • Testing Strategy
  • Subtopics
    • Base OS Software Stack
    • Scientific Software Stack (Spack/Lmod)
    • Container Support
  • Further References
• Kubernetes in HPC Environments
  • The Problem: Managing HPC Software Lifecycle
  • Our Approach: K8s as a Low-Risk Learning Platform
    • Philosophy: “Do No Harm”
    • The Architecture Boundary: K8s vs. Production
    • The “Disposable K8s” Philosophy
  • What We Learned: Key Insights
    • Streamlining Operations
    • Introducing GitOps and CI/CD to HPC Operations
    • Developing K8s Operational Expertise
  • Is This Approach Right for You?
    • This approach is well-suited for:
    • This approach may NOT be suitable if:
  • Real-World Applications
    • Case Study: Kubernetes-Based OS Image Testing
    • Case Study: Kubernetes-Based Spack Stack Builds
  • Getting Started
    • Getting Started: K8s Deployment from Bare Metal
  • Further Reading
• HPC Subsystems
  • Linux NFS Client
    • Performance Optimization Methodology
    • Benchmarking Tools
    • General Optimization Dimensions
    • Other Considerations
    • Troubleshooting
  • AMD 9004 CPU
    • BIOS Config
    • Kernel Parameters
    • AMD-specific Kernel Modules
    • References
  • Networking
    • Benchmarks
• Testing and Validation
  • Validation Philosophy
  • Tool Categories
    • External Validation
    • Internal Baseline & Regression Detection
    • Monitoring
    • Logging
    • Tracing
  • Tool Selection Strategy
    • Coverage Matrix Approach
    • Runtime Considerations
    • Diagnostic Workflow
    • Setup Complexity Considerations
  • Tool Documentation
    • Atop
    • Intel MPI Benchmarks (IMB-MPI1)
    • nvidia-smi
    • SPEC HPC 2021 Benchmark
  • References and Resources
    • Validation Methodology
    • Tool Categories
• Upstream Software Issues and Workarounds
  • RHEL 9: Binutils Incompatibility with Modern Instruction Sets
  • RHEL 9: DMA Driver Unavailable for AMD Zen4
  • AOCC 5.0: Fortran Compiler Configuration Regression
• NUMA Performance Optimizations
  • NUMA Topology Verification
  • NUMA-Aware Memory Allocation
    • Non-MPI Applications
    • MPI Applications
  • RoCEv2 CPU Core Reservation for Network Processing
  • GPU-Aware NUMA Affinity