Magus02/FAQ Batch Jobs - Revision history

Deepak: Created page with "{{DISPLAYTITLE:FAQ: Batch Jobs — Magus02 HPC}}

2026-04-20T06:46:26Z

Created page with "{{DISPLAYTITLE:FAQ: Batch Jobs — Magus02 HPC}} <div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;..."

New page
{{DISPLAYTITLE:FAQ: Batch Jobs — Magus02 HPC}}
<div style="background:#0d1b2a;border:1px solid #1e3a5a;border-radius:8px;padding:12px 18px;margin-bottom:20px;font-size:13px;font-family:monospace;color:#5ba3d9;">
📍 [[Magus02/Knowledge_Base|Magus02 Knowledge Base]] → HPC User → FAQ: Batch Jobs
</div>

This page covers the most common problems and misunderstandings when submitting jobs to Slurm on Magus02. All examples use Slurm syntax.

TOC

== Why do I have to specify resource requirements? ==

The Slurm scheduler needs to know at least three things about every job to decide where and when to run it:

* '''Runtime''' — how long the job will run
* '''CPU cores''' — how many processing units are needed
* '''Memory''' — how much RAM is required

Without these, Slurm cannot place your job correctly. For example, if you omit <code>--mem</code>, a job requiring 500 GB of RAM might be scheduled to a node with only 64 GB, causing it to crash immediately.

Think of Slurm as playing '''multidimensional Tetris''' — it needs to know the shape of your job (its resource footprint) to fit it efficiently into the available cluster capacity.

<div style="background:#0d1a0d;border-left:4px solid #39d353;border-radius:6px;padding:14px 18px;margin:16px 0;font-size:14px;color:#80b880;">
✅ <strong>Minimum required directives for every job:</strong><br>
<code>#SBATCH --partition=</code> (which queue)<br>
<code>#SBATCH --time=</code> (wall clock limit)<br>
<code>#SBATCH --mem=</code> or <code>--mem-per-cpu=</code> (memory)<br>
<code>#SBATCH --ntasks=</code> or <code>--nodes=</code> (compute resources)
</div>

----

== My job starts but exits immediately with no output — what's wrong? ==

'''Most likely cause:''' The output/error directory does not exist.

Slurm writes your <code>stdout</code> and <code>stderr</code> to files before your commands run. If the target directory does not exist, Slurm cannot open the file and immediately kills the job.

Check your directives:

<pre>
#SBATCH --output=/path/to/output/directory/%j.out
#SBATCH --error=/path/to/error/directory/%j.err
</pre>

The directory <code>/path/to/output/directory/</code> must exist before you submit the job. You '''cannot''' create it inside the jobscript with <code>mkdir</code> — the directory must exist when Slurm tries to open the output file, which happens before any of your script commands run.

'''Fix:'''

<pre>
# Create the directory before submitting:
$ mkdir -p /path/to/output/directory/
$ sbatch myjobscript.sh
</pre>

----

== My job sometimes runs fine, sometimes fails for no apparent reason — why? ==

'''Most likely cause:''' You are relying on modules loaded in your interactive session rather than in the jobscript.

When you load modules on the login node before submitting, those modules may or may not be inherited by the batch job — this is unreliable and system-dependent. Your job's success should never depend on what you happened to have loaded in your terminal.

'''Fix:''' Begin every jobscript with:

<pre>
module purge
module load <every module your job needs>
</pre>

This ensures your job always loads exactly the right software, regardless of your interactive session state. It also makes your jobs '''reproducible''' — you can re-submit them months later and get the same result.

----

== I get "srun: Job step creation temporarily disabled" and my job times out doing nothing — why? ==

'''Cause:''' Nested calls to <code>srun</code> or <code>mpirun</code> inside the same job.

This happens when your jobscript calls <code>srun ./myprogram</code>, but <code>myprogram</code> is itself a script that calls <code>srun</code> or <code>mpirun</code> again internally. The inner call tries to allocate resources that the outer <code>srun</code> has already taken — causing a deadlock.

<pre>
# Problematic jobscript:
#!/bin/bash
#SBATCH ...
srun ./run_simulation.sh # if run_simulation.sh calls mpirun, this deadlocks!
</pre>

'''Fix:''' Check whether your program or wrapper script already invokes <code>mpirun</code>/<code>srun</code> internally:

* If it does: remove the outer <code>srun</code> and call the script directly:
<pre>./run_simulation.sh</pre>

* If your program is a true MPI binary (not a wrapper): keep <code>srun</code> or <code>mpirun</code>:
<pre>mpirun -n $SLURM_NTASKS ./my_mpi_binary</pre>

----

== My job is reported as COMPLETED but my scientific program actually failed — why? ==

'''Cause:''' Slurm determines job success or failure from the '''exit code of your jobscript''', not from the exit code of programs called within it.

The exit code of the whole script is the exit code of its '''last command'''. If you have commands after your main program (e.g. a file copy), their exit code overwrites your program's.

=== Bad example ===

<pre>
#!/bin/bash
#SBATCH ...
./my_scientific_program input.dat # fails with exit code 1
cp results.dat $HOME/archive/ # succeeds with exit code 0
# Slurm sees exit code 0 → marks job as COMPLETED, even though the program failed!
</pre>

=== Correct approach: save and re-use the exit code ===

<pre>
#!/bin/bash
#SBATCH ...

./my_scientific_program input.dat
EXITCODE=$? # capture exit code immediately

# Post-processing and cleanup:
cp results.dat $HOME/archive/
echo "Job finished at $(date)"

exit $EXITCODE # report actual program success/failure to Slurm
</pre>

Now Slurm will correctly show FAILED if your scientific program failed, even if all post-processing steps succeeded.

----

== My job is stuck in PD (Pending) — when will it run? ==

A job stays Pending for several possible reasons:

{| class="wikitable" style="width:100%;"
|-
! style="background:#0d1b2a;color:#94a8be;" | Reason Code (squeue -u $USER)
! style="background:#0d1b2a;color:#94a8be;" | Meaning
! style="background:#0d1b2a;color:#94a8be;" | Action
|-
| <code>Resources</code> || Waiting for enough nodes/cores to be free || Wait — will run when resources are available
|-
| <code>Priority</code> || Higher-priority jobs ahead in the queue || Wait — or consider using a lower-priority queue like <code>low/long</code>
|-
| <code>QOSMaxJobsPerUser</code> || You have hit the maximum number of running jobs for your queue || Wait for one of your running jobs to finish
|-
| <code>QOSMaxCpuPerUser</code> || You have hit the core limit for your user in this partition || Request fewer cores, or wait
|-
| <code>InvalidQOS</code> or <code>PartitionNodeLimit</code> || You requested more resources than the partition allows || Reduce <code>--nodes</code>, <code>--ntasks</code>, or <code>--mem</code>; check partition limits
|}

Check why your job is pending:

<pre>
$ squeue -u $USER -o "%.10i %.12j %.8T %.12R"
# %R shows the reason code
</pre>

----

== How do I check how much memory my job actually used? ==

After your job completes, use <code>sacct</code>:

<pre>
# Show memory usage and elapsed time for a completed job:
$ sacct -j JOB_ID --format=JobID,Elapsed,MaxRSS,MaxVMSize,State,ExitCode

# MaxRSS = peak real memory used (in kilobytes)
# MaxVMSize = peak virtual memory used
</pre>

Use this information to tune your <code>--mem</code> requests for future jobs — requesting too much wastes allocation; requesting too little risks OOM (Out Of Memory) kills.

----

== References ==

* [[Magus02/Batch_Scheduler|Batch Scheduler — Job Submission Guide]]
* [[Magus02/Scheduling_Basics|Scheduling Basics]]
* [[Magus02/Modules|Environment Modules]]
* [[Magus02|Magus02 Main Page — Queue Structure]]

----
<div style="font-family:monospace;font-size:11px;color:#484f58;text-align:right;">
[[Magus02/Knowledge_Base|↑ Back to Knowledge Base]] · Maintained by HPC Administration · wiki.snu.edu.in
</div>