Case Study: Kubernetes-Based OS Image Testing

This case study demonstrates using Kubernetes to automate software-layer validation for PXE boot images. Container-based testing executes rapidly at scale, validating aspects functionally equivalent to bare-metal execution while deferring deployment-specific validation to dedicated test infrastructure.

Automation Scope:

  • Software presence and basic functionality (compilers, libraries, utilities)

  • Compilation and linking behavior (development packages, toolchains)

  • Module system configuration (Lmod, environment setup)

  • Single-node MPI operation (initialization, intra-process communication)

Deferred to Bare-Metal Testing:

  • Boot process validation (PXE, GRUB, initramfs)

  • Hardware driver operation (GPU, network fabric)

  • SLURM scheduler integration

  • Multi-node communication

  • Production filesystem access

Value Proposition: Kubernetes enables rapid, repeatable software validation, identifying configuration errors within minutes rather than hours. This trades computing resources for manual administrative effort, deferring expensive bare-metal validation until software integrity is confirmed.

Problem Context

Manual Image Validation Challenges

Traditional PXE image builds require extensive manual validation before deployment:

Manual validation workflow:

  1. Build modified image on dedicated build system

  2. Deploy test image to PXE infrastructure

  3. Boot test node from network

  4. SSH to test node and manually verify:

    • System boot successful

    • Kernel modules load correctly

    • Compilers function properly

    • Module system operates as expected

    • GPU drivers initialize

    • MPI implementations execute

    • Network connectivity established

  5. Identify issues, rebuild image, repeat validation

Operational challenges:

  • Time investment: Manual validation requires 2-3 hours per iteration

  • Limited coverage: Validation depth depends on administrator memory and available time

  • Regression risk: No systematic verification that previously resolved issues remain fixed

  • Undocumented procedures: Validation knowledge resides in administrator experience rather than documented processes

  • Update hesitancy: Manual validation burden discourages OS updates and security patches

Representative failure modes encountered:

  • Kernel modules missing after kernel updates

  • Module system reverting to environment-modules after package updates

  • Compiler environment variables incorrectly configured

  • MPI failing initialization due to missing system libraries

  • GPU drivers incompatible with updated kernel versions

Solution Architecture

Automated testing executes in Kubernetes using Argo Workflows, validating image integrity before manual deployment validation.

Workflow Overview

Git Commit → Build Image → Container Tests → Test Report
                                  ↓
                          (tests pass)
                                  ↓
                  Manual Deployment to Test Nodes
                                  ↓
                  Bare-Metal Validation (SLURM, MPI, etc.)
                                  ↓
                      Progressive Production Rollout

Design rationale: Container-based tests validate software functionality rapidly, deferring expensive manual validation until software integrity is established. This prevents wasted effort validating images with fundamental software defects.

Automated Test Suite

Test Organization

Tests validate base OS software stack components as documented in Base OS Software Stack:

image-tests/
├── fixtures/                     # Test program source code
│   ├── hello.c
│   ├── hello.cpp
│   ├── hello.f90
│   ├── mpi_hello.c
│   ├── test_curl.c
│   ├── test_fontconfig.c
│   ├── test_munge.c
│   └── test_pmix.c
├── run-test-commandline-utils.sh
├── run-test-os-gcc.sh
├── run-test-os-gxx.sh
├── run-test-os-gfortran.sh
├── run-test-curl-devel.sh
├── run-test-fontconfig-devel.sh
├── run-test-munge-devel.sh
├── run-test-pmix-devel.sh
├── run-test-mlnx-openmpi.sh
├── run-test-lmod.sh
├── run-test-qt5.sh
└── run-test-env.sh

Test Categories

Command-line utilities

Validates presence and basic operation of essential system tools (bash, make, cmake, git, tmux, rsync).

System compilers

Verifies OS-provided GCC, G++, and GFortran can compile and execute test programs.

Development libraries

Compiles test programs linking against system libraries (curl, fontconfig, pmix, munge), validating both header availability and dynamic linking.

MPI functionality

Tests system-provided MPI (Mellanox OpenMPI) compilation, linking, and basic execution including communication primitives.

Module system

Verifies Lmod installation and correct alternative selection (Lmod versus environment-modules).

GUI libraries

Validates Qt5 installation paths and pkg-config integration.

Environment configuration

Tests system-wide environment variables and shell initialization.

Test Implementation Example

Representative test structure (system compiler validation):

#!/bin/bash
# run-test-os-gcc.sh
set -euo pipefail

echo "=== Testing OS-provided GCC ==="

# Verify compiler exists
command -v gcc || exit 1

# Test compilation
cat > test.c << 'EOF'
#include <stdio.h>
int main() {
    printf("Compiler validation successful\n");
    return 0;
}
EOF

gcc test.c -o test_c
./test_c

echo "✓ OS GCC validated"

See Base OS Software Stack for comprehensive test examples and rationale.

Container Test Limitations

Scope of Container-Based Validation

Container testing validates software functionality but cannot verify deployment-specific aspects requiring bare-metal hardware:

Cannot validate in containers:

  • Bootability: PXE boot process, GRUB configuration, initramfs integrity

  • SLURM integration: Job scheduler communication, process management via PMIx

  • Cross-node MPI: Multi-node communication across physical interconnect

  • Network fabric drivers: InfiniBand/RoCE hardware operation (containers use host networking)

  • GPU hardware detection: Physical GPU recognition and initialization

  • Filesystem mounting: NFS client behavior with production storage systems

  • Hardware-specific kernel modules: Drivers for specialized HPC hardware

Container testing validates:

  • Software package presence and basic functionality

  • Compilation toolchain operation

  • Library linking and dynamic loading

  • Module system configuration

  • Single-node MPI initialization and intra-process communication

Important

Container tests serve as unit tests validating software layer integrity. Successful container tests indicate the image warrants deployment validation, but cannot verify bootability, hardware driver operation, or production integration. See Base OS Software Stack for comprehensive validation methodology including bare-metal testing and progressive rollout procedures.

Workflow Implementation

Argo Workflow Structure

Simplified workflow template demonstrating test orchestration:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: pxe-image-build-test
spec:
  entrypoint: main

  templates:
  - name: main
    steps:
    - - name: build-image
        template: build-pxe-image

    - - name: test-image
        template: run-test-suite
        arguments:
          artifacts:
          - name: image
            from: "{{steps.build-image.outputs.artifacts.image}}"

    - - name: generate-report
        template: create-test-report
        arguments:
          parameters:
          - name: results
            value: "{{steps.test-image.outputs.parameters.results}}"

  - name: build-pxe-image
    container:
      image: rhel9-build-tools:latest
      command: ["/scripts/build-pxe-image.sh"]
      volumeMounts:
      - name: shared-storage
        mountPath: /output
    outputs:
      artifacts:
      - name: image
        path: /output/pxe-image.img

  - name: run-test-suite
    inputs:
      artifacts:
      - name: image
    dag:
      tasks:
      - name: test-compilers
        template: test-compiler-suite
      - name: test-libraries
        template: test-library-suite
      - name: test-mpi
        template: test-mpi-suite
      - name: test-modules
        template: test-module-system

  - name: test-compiler-suite
    script:
      image: "{{workflow.parameters.test-image}}"
      command: [bash]
      source: |
        #!/bin/bash
        set -e
        /tests/run-test-os-gcc.sh
        /tests/run-test-os-gxx.sh
        /tests/run-test-os-gfortran.sh

Test execution occurs automatically on image builds, generating reports documenting validation results.

Why Kubernetes for This Testing

Kubernetes Advantages

Rapid execution:

Container-based tests complete in minutes. Traditional approach requiring PXE deployment and manual SSH validation takes hours.

Parallel test execution:

Kubernetes orchestrates multiple test suites concurrently. DAG-based workflows (Argo) efficiently manage test dependencies.

Reproducible environment:

Container isolation ensures consistent test conditions. Eliminates “works on my test node” variability.

Infrastructure as code:

Workflow definitions in Git provide version-controlled, auditable test procedures. Changes trigger automatic validation.

Declarative test specifications:

YAML-defined workflows serve as executable documentation. New team members reference workflow definitions rather than tribal knowledge.

Automation Scope and Limitations

What k8s testing validates effectively:

Software components exhibiting identical behavior in containers and bare-metal:

  • Package installation and file presence

  • Compiler toolchain functionality

  • Library linking and symbol resolution

  • Script execution and path resolution

  • Module system operation (Lmod configuration)

  • Environment variable configuration

What requires bare-metal validation:

Deployment aspects dependent on physical hardware or system integration:

  • Boot process (firmware, bootloader, initramfs)

  • Hardware driver initialization (GPU, InfiniBand)

  • Kernel module loading in production kernel

  • Scheduler integration (SLURM communication)

  • Multi-node fabric communication

  • Production storage system access

Design principle: Automate what’s equivalent; defer what’s not. Container tests catch software configuration errors rapidly, reserving expensive bare-metal validation for hardware-dependent verification.

Operational Impact

Time savings:

Software-layer validation completes in 5-10 minutes (previously 2-3 hours manual testing).

Documented procedures:

Test scripts explicitly document validation requirements. Reduces institutional knowledge dependency.

Regression prevention:

Previously encountered issues become automated test cases, preventing recurrence.

Resource trade-off:

Exchanges computing resources (Kubernetes cluster cycles) for human time. Kubernetes infrastructure cost justified by reduced manual effort.

Current Limitations and Future Considerations

Known Constraints

Test coverage limitations:

  • Test suite validates common failure modes encountered operationally

  • Coverage grows organically as new issues are discovered

  • Some edge cases remain undetected until manual validation

Maintenance requirements:

  • Tests require updates when OS versions change (RHEL 8 to 9 transitions)

  • New software additions necessitate corresponding test development

  • Hardware changes (new GPU models, network adapters) require test modifications

Infrastructure dependencies:

  • Kubernetes cluster availability required for test execution

  • Container runtime limitations affect test fidelity

  • Storage system performance impacts test execution duration

Future Development

Potential enhancements under consideration:

  • Automated deployment to dedicated test nodes post-container validation

  • Integration with production monitoring systems for deployment verification

  • Historical test result tracking for trend analysis

  • Expanded test coverage for specialized hardware configurations

Implementation of these enhancements depends on operational priorities and available resources.

Conclusion

Automated PXE image testing establishes documented validation procedures while reducing manual testing burden. Container-based tests provide rapid software validation, deferring expensive bare-metal validation until software integrity is confirmed. This approach balances automation benefits with practical recognition of container testing limitations, maintaining manual validation for deployment-critical verification.

The test suite grows organically through operational experience, documenting institutional knowledge as executable specifications. While automation does not eliminate manual validation requirements, it enables more efficient resource allocation by deferring human effort until automated validation confirms basic software functionality.