Data and Storage Guide

Last updated: 2026-06-04

This page helps new users decide where to place files on HPC4 or SuperPOD and how to transfer files to and from the system.

Environment

• Users who have already obtained HPC4 or SuperPOD access

• Windows 10/11, macOS, or Linux

• SSH client or file transfer tool with SSH support

Directory Structure

Both HPC4 and SuperPOD provide three main storage tiers. The directory layout is similar, but quotas and retention policies differ.

Path	HPC4	SuperPOD
/home/<username>	200 GB quota · backed up (14-day window)	persistent · see welcome email
/scratch/<username>	500 GB quota · auto-purged after 60 days of inactivity	/scratch/<groupname> · 5 TB group quota · auto-purged after 30 days
/project/<groupname>	10 TB per research group · not backed up	shared group storage · see welcome email

Note

/scratch is temporary high-speed storage. Do not rely on it for long-term retention. Files without read or write access for the retention period are automatically deleted.

Recommended Placement

• ``/home/<username>`` — login scripts, dotfiles, small source trees, personal Python/Conda environments, container images.

• ``/scratch/<username>`` (HPC4) or ``/scratch/<groupname>`` (SuperPOD) — large temporary datasets, intermediate results, short-term job output.

• ``/project/<groupname>`` — shared datasets, shared source code, shared software stacks for your research group.

Check Disk Usage

Use df -h to inspect available space:

df -h /home /scratch /project

File Transfer

rsync is the recommended tool for most file transfers. It supports directory sync, resume on failure, and delta transfers (only changed files).

Tip

Use rsync for all transfers. It is more reliable than scp for directories and large files, and it can resume interrupted transfers.

Use the same hostname and username format as your SSH login:

• Use your HKUST username only (do not append @ust.hk).

• If you are off campus, connect through VPN before starting file transfer.

Rsync Example (Recommended)

Upload a local directory to your home directory on the cluster:

rsync -av local-folder/ <username>@hpc4.ust.hk:~/rsync-test/

The -a flag preserves permissions and timestamps; -v shows progress. Add --progress for per-file transfer details.

Download a directory from the cluster:

rsync -av <username>@hpc4.ust.hk:~/rsync-test/ ./local-folder/

Preview what would be transferred without copying:

rsync -avn local-folder/ <username>@hpc4.ust.hk:~/rsync-test/

For real project use:

• Upload small personal scripts to /home/<username>

• Upload large temporary job input and output under /scratch/<username> (HPC4) or /scratch/<groupname> (SuperPOD)

• Upload team-shared files under /project/<groupname>

SCP Example

Upload a single file:

scp local-file.txt <username>@hpc4.ust.hk:~/

Download a single file:

scp <username>@hpc4.ust.hk:~/local-file.txt ./downloaded-file.txt

SFTP Example

Open an interactive SFTP session:

sftp <username>@hpc4.ust.hk

Example SFTP commands:

pwd
ls
put local-file.txt
get local-file.txt downloaded-via-sftp.txt
bye

Verify the Transfer

After uploading or syncing files, verify that they exist on the cluster.

For rsync verification:

ssh <username>@hpc4.ust.hk 'ls -R ~/rsync-test'
ssh <username>@hpc4.ust.hk 'cat ~/rsync-test/test.txt'

For SCP verification:

ssh <username>@hpc4.ust.hk 'ls -l ~/local-file.txt'

Transfer Checklist

• Confirm the target directory before uploading.

• Use rsync instead of repeated scp when updating many files.

• Verify that the transfer completed successfully before starting a job.

• Remember that /scratch is automatically purged after inactivity — it is not a backup location.

References

• Official login guide: https://itso.hkust.edu.hk/services/academic-teaching-support/high-performance-computing/hpc4/login

• Official VPN setup page: https://itso.hkust.edu.hk/services/cyber-security/vpn

• Official storage types page: https://itso.hkust.edu.hk/services/academic-teaching-support/high-performance-computing/hpc4/fileandstorage