How to Submit Overlapping SLURM Jobs Using Scrontab
Environment
SLURM workload manager with scrontab support
Any HPC partition (e.g., amd, intel)
Issue
Users may need to submit jobs at regular intervals regardless of whether previously submitted jobs have completed. This creates a scenario where:
Jobs need to run periodically (e.g., every minute, every hour)
Multiple instances of the same job should be allowed to run simultaneously
Previous job runs should continue executing without being cancelled
The scheduling should not wait for previous jobs to complete before submitting new ones
Traditional cron-like scheduling might wait for job completion, but some workflows require overlapping execution for parallel processing or continuous monitoring tasks.
Resolution
Use scrontab to submit SLURM jobs at scheduled intervals. The scrontab entry itself is a minimal SLURM job that runs the sbatch command to submit your actual workload.
Configure the scrontab entry
Create a scrontab entry that submits your job script at the desired interval. The scrontab itself should be a lightweight job:
$ scrontab -l
#SCRON --job-name=periodic_submit
#SCRON --account=username
#SCRON --partition=amd
#SCRON --nodes=1
#SCRON --ntasks-per-node=1
#SCRON --cpus-per-task=1
#SCRON --time=0:0:5
* * * * * sbatch /home/username/path/to/job.sbatch
The cron schedule * * * * * means the job will be submitted every minute. You can adjust this to your needs:
*/5 * * * *- every 5 minutes0 * * * *- every hour0 0 * * *- daily at midnight
Please refer to How to Run SLURM Jobs Periodically Using scrontab for basic SLURM cronjob creation.
Create your actual job script
The job script referenced in the scrontab entry contains your actual workload:
#!/bin/bash
#SBATCH --account=username
#SBATCH --partition=amd
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=256
#SBATCH --time=1:0:0
echo "Hello, World! The time is $(date)"
# Your actual processing logic here
Edit your scrontab
To add or modify scrontab entries:
$ scrontab -e
This opens your default editor to modify the scrontab configuration.
Verify overlapping execution
After setup, check the job queue to verify multiple instances are running:
$ squeue -u username
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
212355 amd periodic username PD 0:00 1 (BeginTime)
212455 amd job username R 0:19 1 cpu43
212452 amd job username R 1:19 1 cpu42
212447 amd job username R 2:19 1 cpu41
212437 amd job username R 3:24 1 cpu33
212433 amd job username R 4:21 1 cpu39
212429 amd job username R 5:19 1 cpu38
212424 amd job username R 6:23 1 cpu37
212420 amd job username R 7:16 1 cpu36
212419 amd job username R 8:20 1 cpu58
212417 amd job username R 9:20 1 cpu44
In this output, you can see multiple instances of the same job running simultaneously, with the periodic_submit job scheduled to run again (PD status with BeginTime reason).
Note
Be mindful of resource usage when allowing overlapping jobs. Ensure your account has sufficient allocation and the cluster has enough resources to accommodate multiple concurrent instances.
Warning
Monitor your job queue regularly to prevent excessive accumulation of jobs if they run longer than expected. Consider implementing job duration limits or dependency chains if jobs should not overlap indefinitely.
Root Cause
The scrontab mechanism works by submitting a lightweight SLURM job at the scheduled interval. This job executes sbatch, which submits another independent job to the queue. Since each submission creates a new, separate job, they can overlap freely without interfering with each other.
This approach differs from traditional job dependencies (--dependency=afterok:jobid) which would wait for completion. The scrontab method provides true periodic submission regardless of previous job states, making it ideal for:
Continuous monitoring or polling tasks
Parallel processing of time-windowed data
Periodic backup or checkpointing operations
Concurrent analysis pipelines