Testing and Validation

This documentation describes operational validation methodology for HPC infrastructure. Testing serves multiple purposes across system lifecycle: establishing performance expectations, detecting regressions, monitoring operational health, and diagnosing issues.

Validation Philosophy

Testing follows data-driven operational principles (see HPC Administration Guide):

Establish baselines:

Initial validation creates archived golden standards for later comparison. Without baseline measurements, “is this normal?” questions become unanswerable.

Detect regressions:

Automated comparison against baselines identifies performance degradation after system changes.

Diagnose root causes:

Problem-specific tools with known limitations and use cases enable efficient troubleshooting.

Every validation result should be archived: performance figures, network topology snapshots, configuration states. Regression detection requires historical data - current measurements derive meaning only through comparison with established baselines.

Tool Categories

Validation tools organize into five operational categories. Effective HPC operations require coverage across all categories - not exhaustive tools in one category. Each category serves distinct diagnostic purposes and complements others in the validation workflow.

External Validation

External validation compares system performance against published results from other HPC centers using standardized benchmarks. This approach answers: “Is our performance competitive with similar systems?”

Characteristics:

• Standardized benchmarks with published results (HPL, SPEC HPC, MLPerf)

• Vendor datasheets and research papers provide comparison points

• Runtime varies from minutes (HPL quick check) to hours (SPEC HPC comprehensive)

Subsystem coverage:

• Compute: HPL (CPU), SPEC HPC CPU benchmarks

• Network: SPEC HPC multi-node scaling, NCCL collective operations

• Storage: SPEC HPC I/O benchmarks, IOR

• Memory: STREAM, HPCG

First-line tools: HPL (CPU compute), HPL-MxP + NCCL (GPU compute), STREAM (memory bandwidth)

Internal Baseline & Regression Detection

Internal baseline tracking monitors system performance over time, detecting degradation after changes. This approach answers: “Did system performance degrade after this change?”

Characteristics:

• Any reproducible benchmark works - archive baseline, compare after changes

• Can reuse external validation benchmarks or domain-specific applications

• Tolerance guidelines: ±5% acceptable variation, >10% investigate, >20% halt deployment

Subsystem coverage:

• Compute: Any CPU/GPU benchmark with reproducible results

• Network: IMB-MPI1, NCCL, point-to-point bandwidth tests

• Storage: FIO, user I/O workloads

• Topology: ibnetdiscover diff, lstopo comparison

First-line tools: HPL (fast execution), SPEC HPC (comprehensive coverage), user applications (domain relevance)

Execution cadence: After OS/kernel/driver updates, firmware changes, hardware maintenance, monthly drift detection

Monitoring

Monitoring provides real-time observability of system health and resource utilization. This approach answers: “What’s happening right now?”

Characteristics:

• Always-on with low overhead

• Real-time dashboards and alerting

• Time-series data with retention (typically 30 days)

Subsystem coverage:

• Compute: CPU/GPU utilization, temperature, frequency scaling

• Network: Bandwidth utilization, packet drops, error rates

• Storage: I/O throughput, latency, queue depth

• Memory: Utilization, swap activity, NUMA balance

First-line tools: Prometheus + Grafana (time-series metrics), node exporters, GPU metrics

Use cases: Capacity planning, real-time alerts, performance trending

Logging

Logging maintains persistent records enabling post-mortem analysis of past events. This approach answers: “What happened yesterday?” or “Why did this fail overnight?”

Characteristics:

• Persistent storage surviving crashes

• Historical data with retention (typically 90 days)

• Subsystem-specific granularity (10-minute intervals to per-event)

Subsystem coverage:

• System-wide resources: Atop (10-minute intervals, all resources)

• Kernel events: journalctl (hardware faults, driver errors, OOM)

• Application execution: SLURM logs (job failures, scheduler issues)

• Network fabric: Switch logs (link flaps, congestion events)

First-line tools: Atop (resource backtracking), journalctl (system events), SLURM logs (job context)

Critical requirement: Configure persistent logging at system deployment - logs must survive crashes to fulfill their diagnostic purpose.

Tracing

Tracing provides deep, code-level diagnosis of specific issues through targeted investigation tools. This approach answers: “Why exactly is this broken?” or “Which line of code caused this?”

Characteristics:

• Not first-line tools - use after logs localize subsystem

• High overhead - targeted use only, not always-on

• Requires instrumentation (debug symbols, profiling enabled)

Subsystem coverage:

• Application crashes: gdb + core dumps (stack traces)

• Application slowness: strace (system calls), perf (CPU profiling)

• Network issues: IMB-MPI1 (MPI patterns), RDMA tools (physical layer)

• GPU bottlenecks: Nsight Systems (kernel profiling), nvidia-smi debug

First-line tools: gdb (crashes), strace (I/O slowness), perf (CPU slowness), IMB-MPI1 (network diagnosis)

Usage pattern: After monitoring/logging identified subsystem, use tracing for precise root cause identification.

Tool Selection Strategy

Coverage Matrix Approach

Effective validation requires at least one tool in each category covering each subsystem. Tool selection depends on available infrastructure, operational needs, and resource constraints.

Example coverage matrix:

Subsystem	External	Baseline	Monitoring	Logging	Tracing
CPU	HPL	HPL	node_exporter	atop	perf
GPU	HPL-MxP	HPL-MxP	dcgm_exporter	atop	nsight
Memory	STREAM	STREAM	node_exporter	atop	valgrind
Network	NCCL	NCCL	IB counters	fabric logs	ibdiagnet
Storage	IOR	FIO	node_exporter	journalctl	strace

This matrix represents one possible configuration. Actual tool selection varies based on hardware capabilities (InfiniBand vs Ethernet, NVIDIA vs AMD GPUs), workload characteristics, and operational priorities. Tracing tools are problem-specific and invoked as needed rather than providing always-on coverage.

Runtime Considerations

Tool selection often depends on available runtime:

Quick checks (< 5 minutes):

• HPL: 1-minute configuration for boot validation

• STREAM: Memory bandwidth check

• FIO: Single-node storage validation

• Topology diff: ibnetdiscover comparison

Quick checks serve post-maintenance validation and rapid regression detection.

Comprehensive validation (> 1 hour):

• SPEC HPC 2021: Multi-workload suite, subsystem isolation

• MLPerf Training: AI infrastructure validation

• User applications: Domain-specific validation

Comprehensive validation suits major system changes, quarterly validation cycles, and external performance comparison.

Diagnostic Workflow

Typical diagnostic progression follows increasing depth:

1. External validation (initial deployment)

Establish baseline performance against published results. Identifies gross misconfigurations or hardware issues before production deployment.

2. Regression detection (after changes)

Compare current performance against archived baselines. Detects degradation from updates, configuration changes, or hardware drift.

3. Monitoring (real-time issues)

Real-time dashboards identify ongoing resource saturation or anomalous behavior. Provides immediate visibility into current system state.

4. Logging (post-mortem analysis)

Historical logs enable backtracking: “What consumed resources during yesterday’s job failure?” Localizes issues to specific subsystems and timeframes.

5. Tracing (targeted diagnosis)

Deep instrumentation reveals code-level root causes after logging identified problematic subsystem. Provides function-level detail for precise diagnosis.

Key principle: Start broad (monitoring/logs), narrow to specific (traces). Tracing without log context wastes investigation time wandering through irrelevant code paths.

Setup Complexity Considerations

Low setup complexity (hours):

• HPL, STREAM, FIO: Pre-built binaries, minimal configuration

• Monitoring: Deploy exporters, configure retention

• Logging: Enable atop, configure journalctl persistence

Low-complexity tools provide immediate diagnostic capability.

Medium setup complexity (days):

• IMB-MPI1: Requires MPI runtime configuration

• NCCL: Requires GPU/network fabric tuning

• User dashboards: Grafana dashboard development

Medium-complexity tools require deeper system integration.

High setup complexity (weeks):

• SPEC HPC 2021: Multiple application builds, environment tuning, validation

• MLPerf: Dataset preparation, framework configuration

• User applications: Domain expertise, representative datasets

High-complexity tools provide comprehensive validation at cost of extended deployment time.

Strategy: Start with low-complexity coverage (HPL + monitoring + atop), add diagnostic depth as operational needs emerge.

Tool Documentation

Detailed documentation for specific validation tools:

• Atop
• Intel MPI Benchmarks (IMB-MPI1)
• nvidia-smi
• SPEC HPC 2021 Benchmark

Each tool document follows consistent structure: Overview, Basic Usage, Use Cases, Best Practices, References.

References and Resources

Validation Methodology

• Baseline establishment: HPC Administration Guide - Data-driven operational principles

• Regression detection: Compare current measurements against archived baselines

• Diagnostic workflows: Progress from broad monitoring to targeted tracing

Tool Categories

• External validation tools: SPEC HPC, HPL, MLPerf benchmarks

• Internal baseline tools: User applications, reproducible benchmarks

• Monitoring tools: Prometheus, Grafana, node exporters

• Logging tools: Atop, journalctl, SLURM logs

• Tracing tools: gdb, strace, perf, Nsight Systems