How to efficiently list all files recursively on Linux

Environment

Linux HPC Cluster

fd command (available as fd-find on Ubuntu-based systems)

Issue

Need to list all files recursively in large directory structures

Traditional ls -R is slow and produces cluttered output

find command has complex syntax and can be slow on large filesystems

Need to quickly search for files by name, extension, or other attributes in HPC environments

Resolution

Basic recursive file listing

The fd command provides a fast, user-friendly alternative to find with simpler syntax and parallel execution.

List all files recursively:

$ # List all files in current directory and subdirectories
$ fd
scripts/process_data.py
scripts/analyze.R
data/input.csv
data/results/output.txt
logs/job_12345.log

$ # List all files with full paths
$ fd --absolute-path
/home/username/project/scripts/process_data.py
/home/username/project/data/input.csv

Note

By default, fd automatically excludes hidden files and respects .gitignore patterns, making output cleaner than find.

Search for files by name

Simple pattern matching:

$ # Find files containing "data" in the name
$ fd data
scripts/process_data.py
data/input.csv
backup/old_data.tar.gz

$ # Case-insensitive search
$ fd -i DATA
scripts/process_data.py
analysis/DATA_2024.xlsx

$ # Exact filename match
$ fd --glob "results.txt"
output/results.txt
analysis/final/results.txt

Search by file extension:

$ # Find all Python files
$ fd -e py
scripts/process_data.py
scripts/analyze_results.py
tools/helper.py

$ # Find all CSV and TSV data files
$ fd -e csv -e tsv
data/input.csv
data/sample.tsv
results/output.csv

$ # Find all log files
$ fd -e log
logs/slurm-12345.log
logs/application.log

Search only files or only directories

Filter by file type:

$ # List only regular files
$ fd -t f
scripts/process_data.py
data/input.csv

$ # List only directories
$ fd -t d
scripts/
data/
results/
logs/

$ # Find directories named "output"
$ fd -t d output
analysis/output/
results/output/
backup/2024/output/

Search with depth control

Limit search depth for better performance:

$ # List files only in current directory (depth 1)
$ fd -d 1
README.md
setup.sh

$ # Search up to 2 levels deep
$ fd -d 2 -e py
scripts/main.py
tools/helper.py

$ # Find all data files within 3 directory levels
$ fd -d 3 -e csv -e dat
data/input.csv
data/2024/sample.csv
analysis/raw/measurements.dat

Include hidden files and ignore patterns

Control what files to include:

$ # Include hidden files (starting with .)
$ fd -H config
.config/settings.ini
scripts/.config.json

$ # Show all files including ignored ones
$ fd -I -e tmp
cache/temp.tmp
build/output.tmp

$ # Exclude specific directories
$ fd -E node_modules -E __pycache__ -e py
scripts/main.py
analysis/process.py

Finding files by modification time

Search based on when files were modified:

$ # Files changed in the last 24 hours
$ fd --changed-within 24h
results/latest_output.txt
logs/job_today.log

$ # Files changed in the last week
$ fd --changed-within 7d -e py
scripts/new_analysis.py

$ # Files older than 30 days
$ fd --changed-before 30d -e log
logs/old_job.log
archive/2024-11/results.log

Note

Time units: s (seconds), m (minutes), h (hours), d (days), w (weeks)

Executing commands on found files

Perform operations on search results:

$ # Count lines in all Python files
$ fd -e py -x wc -l
234 scripts/process_data.py
156 scripts/analyze.py
89 tools/helper.py

$ # Copy all CSV files to backup directory
$ fd -e csv -x cp {} backup/

$ # Compress all log files older than 7 days
$ fd -e log --changed-before 7d -x gzip {}

$ # Show detailed info for recently modified files
$ fd --changed-within 1d -x ls -lh
-rw-r--r-- 1 username users 2.3M Dec 20 14:30 results/output.csv
-rw-r--r-- 1 username users  156K Dec 20 15:45 logs/job.log

Practical HPC examples

Common use cases on HPC clusters:

$ # Find all SLURM output files
$ fd 'slurm-*.out'
jobs/slurm-12345.out
jobs/slurm-12346.out

$ # Find all checkpoint files
$ fd -e ckpt -e chk
checkpoints/model_epoch10.ckpt
results/simulation.chk

$ # Find large output files (>100MB) modified this week
$ fd --changed-within 7d --size +100m
results/simulation_output.dat
data/processed/large_dataset.hdf5

$ # List all job scripts in home directory
$ fd -e sh -e pbs -e slurm ~
/home/username/jobs/run_analysis.sh
/home/username/scripts/submit.slurm

$ # Find all files owned by you in scratch directory
$ fd . /scratch/username -t f

$ # Clean up temporary files older than 30 days
$ fd -e tmp -e temp --changed-before 30d -x rm {}

Performance comparison

Compare fd with traditional commands:

$ # Traditional approach (slower)
$ time find . -name "*.py"
real    0m2.450s

$ # Using fd (faster with parallel execution)
$ time fd -e py
real    0m0.124s

Note

fd is significantly faster than find on large directory trees due to parallel traversal and optimized algorithms. This is especially noticeable on HPC filesystems with millions of files.

Root Cause

Traditional file listing commands have limitations:

ls -R produces verbose output and is slow on large directories
find has complex syntax that is hard to remember (e.g., -name, -type, -mtime)
find performs sequential directory traversal, which is slow on large filesystems
No built-in filtering for version control files or common ignore patterns

The fd command addresses these issues by:

Using simpler, more intuitive syntax with sensible defaults
Executing parallel directory traversal for better performance
Automatically respecting .gitignore patterns
Providing colored output for better readability
Supporting human-readable time specifications
Offering regex support by default (no need for complex glob patterns)

References

fd official repository: https://github.com/sharkdp/fd
fd user guide: https://github.com/sharkdp/fd#tutorial
Command help: fd --help or man fd