FAQ: Batch Jobs — Magus02 HPC

Latest revision as of 06:46, 20 April 2026

📍 Magus02 Knowledge Base → HPC User → FAQ: Batch Jobs

This page covers the most common problems and misunderstandings when submitting jobs to Slurm on Magus02. All examples use Slurm syntax.

Why do I have to specify resource requirements?

The Slurm scheduler needs to know at least three things about every job to decide where and when to run it:

Runtime — how long the job will run
CPU cores — how many processing units are needed
Memory — how much RAM is required

Without these, Slurm cannot place your job correctly. For example, if you omit --mem, a job requiring 500 GB of RAM might be scheduled to a node with only 64 GB, causing it to crash immediately.

Think of Slurm as playing multidimensional Tetris — it needs to know the shape of your job (its resource footprint) to fit it efficiently into the available cluster capacity.

✅ Minimum required directives for every job:
#SBATCH --partition= (which queue)
#SBATCH --time= (wall clock limit)
#SBATCH --mem= or --mem-per-cpu= (memory)
#SBATCH --ntasks= or --nodes= (compute resources)

My job starts but exits immediately with no output — what's wrong?

Most likely cause: The output/error directory does not exist.

Slurm writes your stdout and stderr to files before your commands run. If the target directory does not exist, Slurm cannot open the file and immediately kills the job.

Check your directives:

#SBATCH --output=/path/to/output/directory/%j.out
#SBATCH --error=/path/to/error/directory/%j.err

The directory /path/to/output/directory/ must exist before you submit the job. You cannot create it inside the jobscript with mkdir — the directory must exist when Slurm tries to open the output file, which happens before any of your script commands run.

Fix:

# Create the directory before submitting:
$ mkdir -p /path/to/output/directory/
$ sbatch myjobscript.sh

My job sometimes runs fine, sometimes fails for no apparent reason — why?

Most likely cause: You are relying on modules loaded in your interactive session rather than in the jobscript.

When you load modules on the login node before submitting, those modules may or may not be inherited by the batch job — this is unreliable and system-dependent. Your job's success should never depend on what you happened to have loaded in your terminal.

Fix: Begin every jobscript with:

module purge
module load <every module your job needs>

This ensures your job always loads exactly the right software, regardless of your interactive session state. It also makes your jobs reproducible — you can re-submit them months later and get the same result.

I get "srun: Job step creation temporarily disabled" and my job times out doing nothing — why?

Cause: Nested calls to srun or mpirun inside the same job.

This happens when your jobscript calls srun ./myprogram, but myprogram is itself a script that calls srun or mpirun again internally. The inner call tries to allocate resources that the outer srun has already taken — causing a deadlock.

# Problematic jobscript:
#!/bin/bash
#SBATCH ...
srun ./run_simulation.sh    # if run_simulation.sh calls mpirun, this deadlocks!

Fix: Check whether your program or wrapper script already invokes mpirun/srun internally:

If it does: remove the outer srun and call the script directly:

./run_simulation.sh

If your program is a true MPI binary (not a wrapper): keep srun or mpirun:

mpirun -n $SLURM_NTASKS ./my_mpi_binary

My job is reported as COMPLETED but my scientific program actually failed — why?

Cause: Slurm determines job success or failure from the exit code of your jobscript, not from the exit code of programs called within it.

The exit code of the whole script is the exit code of its last command. If you have commands after your main program (e.g. a file copy), their exit code overwrites your program's.

Bad example

#!/bin/bash
#SBATCH ...
./my_scientific_program input.dat    # fails with exit code 1
cp results.dat $HOME/archive/         # succeeds with exit code 0
# Slurm sees exit code 0 → marks job as COMPLETED, even though the program failed!

Correct approach: save and re-use the exit code

#!/bin/bash
#SBATCH ...

./my_scientific_program input.dat
EXITCODE=$?                           # capture exit code immediately

# Post-processing and cleanup:
cp results.dat $HOME/archive/
echo "Job finished at $(date)"

exit $EXITCODE                        # report actual program success/failure to Slurm

Now Slurm will correctly show FAILED if your scientific program failed, even if all post-processing steps succeeded.

My job is stuck in PD (Pending) — when will it run?

A job stays Pending for several possible reasons:

Reason Code (squeue -u $USER)	Meaning	Action
`Resources`	Waiting for enough nodes/cores to be free	Wait — will run when resources are available
`Priority`	Higher-priority jobs ahead in the queue	Wait — or consider using a lower-priority queue like `low/long`
`QOSMaxJobsPerUser`	You have hit the maximum number of running jobs for your queue	Wait for one of your running jobs to finish
`QOSMaxCpuPerUser`	You have hit the core limit for your user in this partition	Request fewer cores, or wait
`InvalidQOS` or `PartitionNodeLimit`	You requested more resources than the partition allows	Reduce `--nodes`, `--ntasks`, or `--mem`; check partition limits

Check why your job is pending:

$ squeue -u $USER -o "%.10i %.12j %.8T %.12R"
# %R shows the reason code

How do I check how much memory my job actually used?

After your job completes, use sacct:

# Show memory usage and elapsed time for a completed job:
$ sacct -j JOB_ID --format=JobID,Elapsed,MaxRSS,MaxVMSize,State,ExitCode

# MaxRSS = peak real memory used (in kilobytes)
# MaxVMSize = peak virtual memory used

Use this information to tune your --mem requests for future jobs — requesting too much wastes allocation; requesting too little risks OOM (Out Of Memory) kills.

References

↑ Back to Knowledge Base · Maintained by HPC Administration · wiki.snu.edu.in