When using spack generated module, error while loading shared libraries

Environment

HPC4

Spack (All versions)

Issue

When using Spack and generated module following these steps

Using Spack to compile prerequisite .so dynamically linked libraries, such as netcdf / hdf5

module load libraries and compilers

compile custom source code that uses the libraries

Generated executable cannot execute, with an error cannot open shared object file: No such file or directory

Resolution

Both fixes are required to ensure the shared libraries are locatable by ldconfig at runtime.

System-wide fix

In site-wise spack configs, add these lines to modules.yaml

modules:
  prefix_inspections:
    ./lib:
      - LD_LIBRARY_PATH
    ./lib64:
      - LD_LIBRARY_PATH

Refresh lmod modulefiles with environment deactivated

spack env deactivate
spack module lmod refresh --all --delete-tree

User-specific fix

Applicable if affected module is from user’s Spack instance.

Refresh lmod modulefiles with environment deactivated

spack env deactivate
spack module lmod refresh --all --delete-tree

Root Cause

Search paths to corresponding shared libraries must be present in LD_LIBRARY_PATH for ldconfig to locate and load the libraries at runtime.

As of spack version 0.24.0, the modulefiles generated by default config still does not update LD_LIBRARY_PATH, only PATH, MANPATH, ACLOCAL_PATH, PKG_CONFIG_PATH and CMAKE_PREFIX_PATH are updated.

Diagnosis

Executable failed with an error message

./a.out: error while loading shared libraries: libnetcdf.so.19: cannot open shared object file: No such file or directory

ldd command shows some shared libraries as not found

$ ldd ./a.out

linux-vdso.so.1 (0x00007fffbf3f8000)
libnetcdf.so.19 => not found
libhdf5.so.103 => not found
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7b1b1b0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7b1b1a8000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7b1b04a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7b1b028000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7b1ae37000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7b1b1d0000)

module list shows the corresponding module is loaded

$ module list

Currently Loaded Modules:
  1)  netcdf-c/4.9.2-77j3okj
  2)  netcdf-fortran/4.6.1-p5bkdkc

  Where:
  H:  Hidden Module

module show for the corresponding modules does not contains LD_LIBRARY_PATH

$ module show netcdf-c/4.9.2-77j3okj

--------------------------------------------------------------------------------------
  /opt/shared/spack/lmod/mpich/4.2.1-begklur/Core/netcdf-c/4.9.2-77j3okj.lua:
--------------------------------------------------------------------------------------
whatis("Name : netcdf-c")
# ...
prepend_path("PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/bin")
prepend_path("MANPATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/share/man")
prepend_path("PKG_CONFIG_PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/lib64/pkgconfig")
prepend_path("CMAKE_PREFIX_PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/.")
append_path("HDF5_PLUGIN_PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/plugins")
append_path("MANPATH","")

Shared library can be located in lib/ or lib64 relative to the installation path of the module

$ find /opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h -name "libnetcdf.so.19"
/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/lib64/libnetcdf.so.19