How to speed up Spack package installation
Environment
Spack
Linux
Network filesystem (NFS) storage
Issue
Spack package build is slow
Spack does not parallelize configure phase
Spack install independent packages in serial
NFS is slow during Spack installation
Resolution
Cluster spack instance by default uses -j $(nproc) each time spack install is invoked, and use /dev/shm as
temporary directory for build artifacts to speed up builds.
You may find these optimizations useful to further improve Spack build speed when installing packages with large number of dependencies.
Parallel Package Installation
Install multiple packages simultaneously using background processes
# spack env activate ...
spack install &
spack install &
spack install &
Warning
We do no recommend having > 4 concurrent installations to avoid deadlock or system overload.
Local Lock File
Replace NFS lock with a lock on local filesystem to reduce I/O bottleneck
Note
The local lock file needs to be recreated after system reboot
Create local lock directory
mkdir -p /run/$(id -u)/spack-app
Create local lock file
touch /run/$(id -u)/spack-app/prefix_lock
Link local lock to Spack directory
rm -f $HOME/.spack/apps/.spack-db/prefix_lock ln -s /run/$(id -u)/spack-app/prefix_lock $HOME/.spack/apps/.spack-db/prefix_lock
Root Cause
Spack’s default behavior has several performance limitations:
Package installations are in serial, parallelization only occurs during the build phase of each package.
Configure scripts run serially and are not parallelizable.
NFS lock operations can become a bottleneck when handling many concurrent range locks