File Transfer β€” Magus02 HPC

πŸ“ Magus02 Knowledge Base β†’ Basics β†’ File Transfer

To run jobs on Magus02, your input data and scripts must reside on the cluster's filesystem. This page covers all methods for moving files between your local machine and Magus02.

Choosing a Transfer Method

Method Best For Platform
scp Quick transfer of one or a few files Linux / macOS / Windows 10+
rsync Syncing large directories efficiently Linux / macOS
WinSCP Graphical file manager for Windows Windows
FileZilla Cross-platform graphical FTP/SFTP client Windows / macOS / Linux

πŸ’‘ Note: scp and rsync build on SSH. Ensure your SSH access is working first (see SSH Access). If you load modules in your .bashrc that produce output, scp and sftp may fail β€” see the troubleshooting note below.

scp β€” Secure Copy

scp is the simplest method for transferring individual files. It works wherever SSH works.

# Upload a local file to your home directory on Magus02:
$ scp myinput.dat [email protected]:~/

# Upload to a specific remote path:
$ scp myinput.dat [email protected]:/scratch/myuser/project/

# Download a file from Magus02 to your local machine:
$ scp [email protected]:~/results/output.log ./

# Upload an entire directory (recursive):
$ scp -r myproject/ [email protected]:~/

Troubleshooting scp / sftp failures

If scp fails silently or graphical clients like FileZilla show no files, the cause is often output generated during login (e.g. from module load commands in your .bashrc). Fix it by adding this guard to your ~/.bashrc:

# Only run interactive-session commands when shell is interactive:
[ -z "$PS1" ] && return
# --- put all your 'module load ...' commands AFTER this line ---

rsync β€” Directory Synchronisation

rsync is more powerful than scp: it only transfers the differences between source and destination, making it far more efficient for large directories or repeated transfers.

# General syntax:
$ rsync [options] [source] [destination]

# Sync a local directory to Magus02 (compressed, verbose, human-readable):
$ rsync -azvh -e ssh myproject/ [email protected]:/scratch/myuser/project/

# Sync results back from Magus02 to your local machine:
$ rsync -azvh -e ssh [email protected]:/scratch/myuser/project/results/ ./results/

Common options:

Option Effect
-a Archive mode: preserves timestamps, permissions, symlinks, and recurses into subdirectories
-z Compress data during transfer
-v Verbose output β€” show files being transferred
-h Human-readable sizes
-n Dry run β€” show what would be transferred without actually doing it
--delete Delete files at the destination that no longer exist at the source

WinSCP β€” Graphical Client (Windows)

WinSCP provides a drag-and-drop file manager for Windows users.

Step 1 β€” Open WinSCP and configure the session
Set File protocol to SFTP, Host name to magus02.snu.edu.in, Port to 22, and enter your username and password. Click Login.
Step 2 β€” Accept the host key
On first connection, click Yes to trust the host fingerprint.
Step 3 β€” Transfer files
Your local files appear on the left panel, Magus02 files on the right. Drag and drop, or right-click to upload/download.

Download WinSCP: winscp.net

FileZilla β€” Cross-Platform Graphical Client

FileZilla works on Windows, macOS, and Linux. Connect using SFTP:

Your local files appear in the left panel; remote cluster files appear in the right panel.

Download FileZilla: filezilla-project.org

References


↑ Back to Knowledge Base Β· Maintained by HPC Administration Β· wiki.snu.edu.in

No categories assignedEdit