Difference between revisions of "HPC Backup"

Latest revision as of 11:24, 20 March 2026

HPC Cluster Upgrade: Data Backup Guide (March 2026)

Maintenance Window	Starting 27th March 2026
Backup Location	`/backup/<username>`
Storage Limit	10 GB per user
Recommended Data	Source code, scripts, configuration files, and small datasets.

Overview

During the extensive upgrade planned from 27th March, we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from /data/home/<username> to the temporary backup partition.

Step 1: Verify Current Usage

Check the size of your home directory to ensure you stay within the 10 GB limit for the backup partition.

du -sh /data/home/$USER

Step 2: Compress Your Critical Files

To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts.

# Create a compressed backup of a specific project
tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder

-c: Create a new archive.
-z: Compress using gzip.
-v: Verbosely list files processed (optional).
-f: Use archive file.

Step 3: Synchronize to Backup Directory

Use rsync to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer.

Option A: Copying a Compressed Archive (Recommended)

rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/

-a: Archive mode (preserves permissions and symbolic links).
-h: Human-readable output.
-P: Shows a progress bar during the transfer.

Option B: Copying a Directory Directly

rsync -ahP /data/home/$USER/important_folder /backup/$USER/

Step 4: Verification

Confirm your files are present and check the total space used in the backup directory:

ls -lh /backup/$USER/
du -sh /backup/$USER/

Handling Large Data (>10 GB)

If you have a genuine need to back up more than 10 GB:

Local Download: Download the data to your local machine using SCP or SFTP.
Contact IT/HPC Admic: If local download is not feasible, contact the IT team for alternative arrangements. Email: [email protected]

@@ Line 1: / Line 1: @@
-== HPC Cluster Upgrade: Data Backup Guide (March 2026) == {| class="wikitable" style="width: 100%;" |- ! scope="col" style="background:#f2f2f2;" | <nowiki>'''</nowiki>Maintenance Window<nowiki>'''</nowiki> | Starting 27th March 2026 |- ! scope="col" style="background:#f2f2f2;" | <nowiki>'''</nowiki>Backup Location<nowiki>'''</nowiki> | <nowiki><code>/backup/&</nowiki>lt;username<nowiki>&</nowiki>gt;<nowiki></code></nowiki> |- ! scope="col" style="background:#f2f2f2;" | <nowiki>'''</nowiki>Storage Limit<nowiki>'''</nowiki> | 10 GB per user |- ! scope="col" style="background:#f2f2f2;" | <nowiki>'''</nowiki>Recommended Data<nowiki>'''</nowiki> | Source code, scripts, configuration files, and small datasets. |} === Overview === During the extensive upgrade planned from <nowiki>'''</nowiki>27th March<nowiki>'''</nowiki>, we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from <nowiki><code>/data/home/&</nowiki>lt;username<nowiki>&</nowiki>gt;<nowiki></code></nowiki> to the temporary backup partition. === Step 1: Verify Current Usage === Check the size of your home directory to ensure you stay within the <nowiki>'''</nowiki>10 GB<nowiki>'''</nowiki> limit for the backup partition. <nowiki><syntaxhighlight lang="bash"> du -sh /data/home/$USER </syntaxhighlight></nowiki> === Step 2: Compress Your Critical Files === To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts. <nowiki><syntaxhighlight lang="bash"> # Create a compressed backup of a specific project tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder </syntaxhighlight></nowiki> === Step 3: Synchronize to Backup Directory === Use <nowiki><code>rsync</code></nowiki> to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer. ==== Option A: Copying a Compressed Archive (Recommended) ==== <nowiki><syntaxhighlight lang="bash"> rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/ </syntaxhighlight></nowiki> ==== Option B: Copying a Directory Directly ==== <nowiki><syntaxhighlight lang="bash"> rsync -ahP /data/home/$USER/important_folder /backup/$USER/ </syntaxhighlight></nowiki> === Step 4: Verification === Confirm your files are present and check the total space used in the backup directory: <nowiki><syntaxhighlight lang="bash"> ls -lh /backup/$USER/ du -sh /backup/$USER/ </syntaxhighlight></nowiki> === Handling Large Data (>10 GB) === If you have a genuine need to back up more than 10 GB: # <nowiki>'''</nowiki>Local Download:<nowiki>'''</nowiki> Download the data to your local machine using SCP or SFTP. # <nowiki>'''</nowiki>Contact IT:<nowiki>'''</nowiki> If local download is not feasible, contact the IT team for alternative arrangements. <nowiki>[[Category:HPC User Guides]]</nowiki> <nowiki>[[Category:Maintenance]]</nowiki>
+== HPC Cluster Upgrade: Data Backup Guide (March 2026) ==
+{| class="wikitable" style="width: 100%;"
+|-
+! scope="col" style="background:#f2f2f2;" | '''Maintenance Window'''
+| Starting 27th March 2026
+|-
+! scope="col" style="background:#f2f2f2;" | '''Backup Location'''
+| <code>/backup/&lt;username&gt;</code>
+|-
+! scope="col" style="background:#f2f2f2;" | '''Storage Limit'''
+| 10 GB per user
+|-
+! scope="col" style="background:#f2f2f2;" | '''Recommended Data'''
+| Source code, scripts, configuration files, and small datasets.
+|}
+=== Overview ===
+During the extensive upgrade planned from '''27th March''', we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from <code>/data/home/&lt;username&gt;</code> to the temporary backup partition.
+=== Step 1: Verify Current Usage ===
+Check the size of your home directory to ensure you stay within the '''10 GB''' limit for the backup partition.
+<syntaxhighlight lang="bash">
+du -sh /data/home/$USER
+</syntaxhighlight>
+=== Step 2: Compress Your Critical Files ===
+To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts.
+<syntaxhighlight lang="bash">
+# Create a compressed backup of a specific project
+tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder
+</syntaxhighlight>
+* <code data-path-to-node="10,0,0" data-index-in-node="0">-c</code>: Create a new archive.
+* <code data-path-to-node="10,1,0" data-index-in-node="0">-z</code>: Compress using gzip.
+* <code data-path-to-node="10,2,0" data-index-in-node="0">-v</code>: Verbosely list files processed (optional).
+* <code data-path-to-node="10,3,0" data-index-in-node="0">-f</code>: Use archive file.
+=== Step 3: Synchronize to Backup Directory ===
+Use <code>rsync</code> to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer.
+==== Option A: Copying a Compressed Archive (Recommended) ====
+<syntaxhighlight lang="bash">
+rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/
+</syntaxhighlight>
+* <code data-path-to-node="18,0,0" data-index-in-node="0">-a</code>: Archive mode (preserves permissions and symbolic links).
+* <code data-path-to-node="18,1,0" data-index-in-node="0">-h</code>: Human-readable output.
+* <code data-path-to-node="18,2,0" data-index-in-node="0">-P</code>: Shows a progress bar during the transfer.
+==== Option B: Copying a Directory Directly ====
+<syntaxhighlight lang="bash">
+rsync -ahP /data/home/$USER/important_folder /backup/$USER/
+</syntaxhighlight>
+=== Step 4: Verification ===
+Confirm your files are present and check the total space used in the backup directory:
+<syntaxhighlight lang="bash">
+ls -lh /backup/$USER/
+du -sh /backup/$USER/
+</syntaxhighlight>
+=== Handling Large Data (>10 GB) ===
+If you have a genuine need to back up more than 10 GB:
+# '''Local Download:''' Download the data to your local machine using SCP or SFTP.
+# '''Contact IT/HPC Admic:''' If local download is not feasible, contact the IT team for alternative arrangements. Email: [email protected]
+[[Category:HPC User Guides]]
+[[Category:Maintenance]]