When using spack generated module, error while loading shared libraries
=======================================================================

.. meta::
    :description: When using Spack and generated module, executable cannot find shared libraries at runtime
    :keywords: spack, module, shared libraries, compiler
    :author: kftse <kftse@ust.hk>

.. rst-class:: header

    | Last updated: 2024-11-25
    | Solution verified: 2024-11-25

Environment
-----------

    - HPC4
    - Spack (All versions)

Issue
-----

    When using Spack and generated module following these steps

        1. Using Spack to compile prerequisite ``.so`` dynamically linked libraries, such as netcdf / hdf5
        2. ``module load`` libraries and compilers
        3. compile custom source code that uses the libraries

    Generated executable cannot execute, with an error ``cannot open shared object file: No such file or directory``

Resolution
----------

Both fixes are required to ensure the shared libraries are locatable by ``ldconfig`` at runtime.

System-wide fix
~~~~~~~~~~~~~~~

- In site-wise spack configs, add these lines to ``modules.yaml``

  .. code-block:: yaml

      modules:
        prefix_inspections:
          ./lib:
            - LD_LIBRARY_PATH
          ./lib64:
            - LD_LIBRARY_PATH

- Refresh lmod modulefiles with environment deactivated

  .. code-block:: bash

      spack env deactivate
      spack module lmod refresh --all --delete-tree

User-specific fix
~~~~~~~~~~~~~~~~~

Applicable if affected module is from user's Spack instance.

- Refresh lmod modulefiles with environment deactivated

  .. code-block:: bash

      spack env deactivate
      spack module lmod refresh --all --delete-tree

Root Cause
----------

Search paths to corresponding shared libraries must be present in ``LD_LIBRARY_PATH`` for ``ldconfig`` to locate and
load the libraries at runtime.

As of spack version ``0.24.0``, the modulefiles generated by default config still does not update ``LD_LIBRARY_PATH``,
only ``PATH``, ``MANPATH``, ``ACLOCAL_PATH``, ``PKG_CONFIG_PATH`` and ``CMAKE_PREFIX_PATH`` are updated.

Diagnosis
---------

- Executable failed with an error message

  .. code-block:: text

      ./a.out: error while loading shared libraries: libnetcdf.so.19: cannot open shared object file: No such file or directory

- ``ldd`` command shows some shared libraries as ``not found``

  .. code-block:: shell

      $ ldd ./a.out

      linux-vdso.so.1 (0x00007fffbf3f8000)
      libnetcdf.so.19 => not found
      libhdf5.so.103 => not found
      libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7b1b1b0000)
      libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7b1b1a8000)
      libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7b1b04a000)
      libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7b1b028000)
      libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7b1ae37000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f7b1b1d0000)

- ``module list`` shows the corresponding module is loaded

  .. code-block:: shell

      $ module list

      Currently Loaded Modules:
        1)  netcdf-c/4.9.2-77j3okj
        2)  netcdf-fortran/4.6.1-p5bkdkc

        Where:
        H:  Hidden Module

- ``module show`` for the corresponding modules does not contains ``LD_LIBRARY_PATH``

  .. code-block:: shell

      $ module show netcdf-c/4.9.2-77j3okj

      --------------------------------------------------------------------------------------
        /opt/shared/spack/lmod/mpich/4.2.1-begklur/Core/netcdf-c/4.9.2-77j3okj.lua:
      --------------------------------------------------------------------------------------
      whatis("Name : netcdf-c")
      # ...
      prepend_path("PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/bin")
      prepend_path("MANPATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/share/man")
      prepend_path("PKG_CONFIG_PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/lib64/pkgconfig")
      prepend_path("CMAKE_PREFIX_PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/.")
      append_path("HDF5_PLUGIN_PATH","/opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/plugins")
      append_path("MANPATH","")

- Shared library can be located in ``lib/`` or ``lib64`` relative to the installation path of the module

  .. code-block:: shell

      $ find /opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h -name "libnetcdf.so.19"
      /opt/shared/spack/local/linux-rocky9-x86_64_v4/gcc-11.4.1/netcdf-c-4.9.2-77j3okj3qeoh6cowismbrrvlduneu53h/lib64/libnetcdf.so.19

.. rst-class:: footer

    **HPC Support Team**
      | ITSO, HKUST
      | Email: cchelp@ust.hk
      | Web: https://itso.hkust.edu.hk/

    **Article Info**
      | Issued: 2024-11-25
      | Issued by: kftse (at) ust.hk