Difference between revisions of "HPC Backup"

Latest revision as of 11:24, 20 March 2026

HPC Cluster Upgrade: Data Backup Guide (March 2026)

Maintenance Window	Starting 27th March 2026
Backup Location	`/backup/<username>`
Storage Limit	10 GB per user
Recommended Data	Source code, scripts, configuration files, and small datasets.

Overview

During the extensive upgrade planned from 27th March, we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from /data/home/<username> to the temporary backup partition.

Step 1: Verify Current Usage

Check the size of your home directory to ensure you stay within the 10 GB limit for the backup partition.

du -sh /data/home/$USER

Step 2: Compress Your Critical Files

To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts.

# Create a compressed backup of a specific project
tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder

-c: Create a new archive.
-z: Compress using gzip.
-v: Verbosely list files processed (optional).
-f: Use archive file.

Step 3: Synchronize to Backup Directory

Use rsync to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer.

Option A: Copying a Compressed Archive (Recommended)

rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/

-a: Archive mode (preserves permissions and symbolic links).
-h: Human-readable output.
-P: Shows a progress bar during the transfer.

Option B: Copying a Directory Directly

rsync -ahP /data/home/$USER/important_folder /backup/$USER/

Step 4: Verification

Confirm your files are present and check the total space used in the backup directory:

ls -lh /backup/$USER/
du -sh /backup/$USER/

Handling Large Data (>10 GB)

If you have a genuine need to back up more than 10 GB:

Local Download: Download the data to your local machine using SCP or SFTP.
Contact IT/HPC Admic: If local download is not feasible, contact the IT team for alternative arrangements. Email: [email protected]

@@ Line 1: / Line 1: @@
-== '''HPC Data Backup Technical Guide''' ==
+== HPC Cluster Upgrade: Data Backup Guide (March 2026) ==
-=== '''Step 1: Identify Your Critical Data''' ===
+{| class="wikitable" style="width: 100%;"
-Before starting, check your current usage to ensure you stay within the '''10 GB''' limit in the <code data-path-to-node="4" data-index-in-node="91">/backup</code> directory.
+|-
-<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-55 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-55 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkAE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-55 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-55">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-55 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-55 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-55"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-55">
+! scope="col" style="background:#f2f2f2;" | '''Maintenance Window'''
- <code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-55">du -sh /data/home/<span class="hljs-variable">$USER</span>
+| Starting 27th March 2026
- </code>
+|-
-</div></div></div></code-block></response-element>
+! scope="col" style="background:#f2f2f2;" | '''Backup Location'''
+| <code>/backup/&lt;username&gt;</code>
+|-
+! scope="col" style="background:#f2f2f2;" | '''Storage Limit'''
+| 10 GB per user
+|-
+! scope="col" style="background:#f2f2f2;" | '''Recommended Data'''
+| Source code, scripts, configuration files, and small datasets.
+|}
-=== '''Step 2: Compress Your Files (Recommended)''' ===
+=== Overview ===
-Compressing your files into a single archive (tarball) makes the transfer faster and keeps the backup directory organized.
+During the extensive upgrade planned from '''27th March''', we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from <code>/data/home/&lt;username&gt;</code> to the temporary backup partition.
-'''To create a compressed archive of a specific project folder:'''
+=== Step 1: Verify Current Usage ===
-<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-56 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-56 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkQE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-56 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-56">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-56 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-56 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-56"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-56">
+Check the size of your home directory to ensure you stay within the '''10 GB''' limit for the backup partition.
- <code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-56">tar -czvf ~/my_backup_project.tar.gz /data/home/<span class="hljs-variable">$USER</span>/path_to_your_project
+<syntaxhighlight lang="bash">
- </code>
+du -sh /data/home/$USER
-</div></div></div></code-block></response-element>
+</syntaxhighlight>
+=== Step 2: Compress Your Critical Files ===
+To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts.
+<syntaxhighlight lang="bash">
+# Create a compressed backup of a specific project
+tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder
+</syntaxhighlight>
 * <code data-path-to-node="10,0,0" data-index-in-node="0">-c</code>: Create a new archive.
@@ Line 22: / Line 37: @@
 * <code data-path-to-node="10,3,0" data-index-in-node="0">-f</code>: Use archive file.
-----
+=== Step 3: Synchronize to Backup Directory ===
+Use <code>rsync</code> to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer.
-=== '''Step 3: Synchronize to the Backup Directory''' ===
+==== Option A: Copying a Compressed Archive (Recommended) ====
-Use <code data-path-to-node="13" data-index-in-node="4">rsync</code> to move your compressed file or raw directories to the designated backup path. This tool is efficient as it verifies the integrity of the data during the copy.
+<syntaxhighlight lang="bash">
+rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/
-'''Option A: Copying a compressed tarball (Recommended)'''
+</syntaxhighlight>
-<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-57 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-57 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkgE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-57 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-57">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-57 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-57 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-57"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-57">
- <code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-57">rsync -ahP ~/my_backup_project.tar.gz /backup/<span class="hljs-variable">$USER</span>/
- </code>
-</div></div></div></code-block></response-element>
-'''Option B: Copying a directory directly (if under 10 GB)'''
-<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-58 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-58 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkwE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-58 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-58">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-58 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-58 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-58"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-58">
- <code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-58">rsync -ahP /data/home/<span class="hljs-variable">$USER</span>/my_folder /backup/<span class="hljs-variable">$USER</span>/
- </code>
-</div></div></div></code-block></response-element>
 * <code data-path-to-node="18,0,0" data-index-in-node="0">-a</code>: Archive mode (preserves permissions and symbolic links).
@@ Line 43: / Line 49: @@
 * <code data-path-to-node="18,2,0" data-index-in-node="0">-P</code>: Shows a progress bar during the transfer.
-----
+==== Option B: Copying a Directory Directly ====
+<syntaxhighlight lang="bash">
+rsync -ahP /data/home/$USER/important_folder /backup/$USER/
+</syntaxhighlight>
+=== Step 4: Verification ===
+Confirm your files are present and check the total space used in the backup directory:
+<syntaxhighlight lang="bash">
+ls -lh /backup/$USER/
+du -sh /backup/$USER/
+</syntaxhighlight>
+=== Handling Large Data (>10 GB) ===
+If you have a genuine need to back up more than 10 GB:
+# '''Local Download:''' Download the data to your local machine using SCP or SFTP.
+# '''Contact IT/HPC Admic:''' If local download is not feasible, contact the IT team for alternative arrangements. Email: [email protected]
-=== '''Step 4: Verify the Backup''' ===
+[[Category:HPC User Guides]]
-Once the transfer is complete, confirm your files are present and check the final size:
+[[Category:Maintenance]]
-<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-59 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-59 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQlAE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-59 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-59">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-59 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-59 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[[&quot;r_272d4d52df4c55c4&quot;,&quot;c_370a422751401628&quot;,null,&quot;rc_94a40eca3057ca6d&quot;,null,null,&quot;en&quot;,null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-59"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-59">
- <code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-59">ls -lh /backup/<span class="hljs-variable">$USER</span>/
- du -sh /backup/<span class="hljs-variable">$USER</span></code>
-</div></div></div></code-block></response-element>