HPC Backup - Revision history

Deepak at 11:24, 20 March 2026

2026-03-20T11:24:34Z

Deepak: email

2026-03-20T11:17:12Z

Deepak: page

2026-03-20T11:16:05Z

page

Deepak: new page

2026-03-20T11:15:23Z

new page

Show changes

Deepak: new page

2026-03-20T11:14:10Z

new page

New page

== '''HPC Data Backup Technical Guide''' ==

=== '''Step 1: Identify Your Critical Data''' ===
Before starting, check your current usage to ensure you stay within the '''10 GB''' limit in the <code data-path-to-node="4" data-index-in-node="91">/backup</code> directory.
<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-55 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-55 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkAE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-55 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-55">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-55 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-55 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-55"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-55">
<code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-55">du -sh /data/home/<span class="hljs-variable">$USER</span>
</code>
</div></div></div></code-block></response-element>

=== '''Step 2: Compress Your Files (Recommended)''' ===
Compressing your files into a single archive (tarball) makes the transfer faster and keeps the backup directory organized.

'''To create a compressed archive of a specific project folder:'''
<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-56 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-56 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkQE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-56 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-56">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-56 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-56 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-56"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-56">
<code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-56">tar -czvf ~/my_backup_project.tar.gz /data/home/<span class="hljs-variable">$USER</span>/path_to_your_project
</code>
</div></div></div></code-block></response-element>

* <code data-path-to-node="10,0,0" data-index-in-node="0">-c</code>: Create a new archive.
* <code data-path-to-node="10,1,0" data-index-in-node="0">-z</code>: Compress using gzip.
* <code data-path-to-node="10,2,0" data-index-in-node="0">-v</code>: Verbosely list files processed (optional).
* <code data-path-to-node="10,3,0" data-index-in-node="0">-f</code>: Use archive file.

----

=== '''Step 3: Synchronize to the Backup Directory''' ===
Use <code data-path-to-node="13" data-index-in-node="4">rsync</code> to move your compressed file or raw directories to the designated backup path. This tool is efficient as it verifies the integrity of the data during the copy.

'''Option A: Copying a compressed tarball (Recommended)'''
<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-57 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-57 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkgE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-57 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-57">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-57 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-57 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-57"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-57">
<code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-57">rsync -ahP ~/my_backup_project.tar.gz /backup/<span class="hljs-variable">$USER</span>/
</code>
</div></div></div></code-block></response-element>
'''Option B: Copying a directory directly (if under 10 GB)'''

<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-58 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-58 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQkwE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-58 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-58">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-58 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-58 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-58"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-58">
<code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-58">rsync -ahP /data/home/<span class="hljs-variable">$USER</span>/my_folder /backup/<span class="hljs-variable">$USER</span>/
</code>
</div></div></div></code-block></response-element>

* <code data-path-to-node="18,0,0" data-index-in-node="0">-a</code>: Archive mode (preserves permissions and symbolic links).
* <code data-path-to-node="18,1,0" data-index-in-node="0">-h</code>: Human-readable output.
* <code data-path-to-node="18,2,0" data-index-in-node="0">-P</code>: Shows a progress bar during the transfer.

----

=== '''Step 4: Verify the Backup''' ===
Once the transfer is complete, confirm your files are present and check the final size:
<response-element class="" ng-version="0.0.0-PLACEHOLDER"><code-block _nghost-ng-c1964922315="" class="ng-tns-c1964922315-59 ng-star-inserted" style=""><div _ngcontent-ng-c1964922315="" class="code-block ng-tns-c1964922315-59 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation" jslog="223238;track:impression,attention;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]]" data-hveid="0" decode-data-ved="1" data-ved="0CAAQhtANahgKEwj_yZXd7q2TAxUAAAAAHQAAAAAQlAE" style="display: block;"><div _ngcontent-ng-c1964922315="" class="code-block-decoration header-formatted gds-title-s ng-tns-c1964922315-59 ng-star-inserted" style=""><span _ngcontent-ng-c1964922315="" class="ng-tns-c1964922315-59">Bash</span><div _ngcontent-ng-c1964922315="" class="buttons ng-tns-c1964922315-59 ng-star-inserted"><button _ngcontent-ng-c1964922315="" aria-label="Copy code" mat-icon-button="" mattooltip="Copy code" class="mdc-icon-button mat-mdc-icon-button mat-mdc-button-base mat-mdc-tooltip-trigger copy-button ng-tns-c1964922315-59 mat-unthemed ng-star-inserted" mat-ripple-loader-uninitialized="" mat-ripple-loader-class-name="mat-mdc-button-ripple" mat-ripple-loader-centered="" jslog="179062;track:generic_click,impression;BardVeMetadataKey:[["r_272d4d52df4c55c4","c_370a422751401628",null,"rc_94a40eca3057ca6d",null,null,"en",null,1,null,null,1,0]];mutable:true"><span class="mat-mdc-button-persistent-ripple mdc-icon-button__ripple"></span><mat-icon _ngcontent-ng-c1964922315="" role="img" fonticon="content_copy" class="mat-icon notranslate gds-icon-s google-symbols mat-ligature-font mat-icon-no-color" aria-hidden="true" data-mat-icon-type="font" data-mat-icon-name="content_copy"></mat-icon><span class="mat-focus-indicator"></span><span class="mat-mdc-button-touch-target"></span></button></div></div><div _ngcontent-ng-c1964922315="" class="formatted-code-block-internal-container ng-tns-c1964922315-59"><div _ngcontent-ng-c1964922315="" class="animated-opacity ng-tns-c1964922315-59">
<code _ngcontent-ng-c1964922315="" role="text" data-test-id="code-content" class="code-container formatted ng-tns-c1964922315-59">ls -lh /backup/<span class="hljs-variable">$USER</span>/
du -sh /backup/<span class="hljs-variable">$USER</span></code>
</div></div></div></code-block></response-element>

@@ Line 31: / Line 31: @@
 tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder
 </syntaxhighlight>
 === Step 3: Synchronize to Backup Directory ===
@@ Line 39: / Line 44: @@
 rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/
 </syntaxhighlight>
 ==== Option B: Copying a Directory Directly ====

@@ Line 55: / Line 55: @@
 If you have a genuine need to back up more than 10 GB:
 # '''Local Download:''' Download the data to your local machine using SCP or SFTP.
-# '''Contact IT:''' If local download is not feasible, contact the IT team for alternative arrangements.
+# '''Contact IT/HPC Admic:''' If local download is not feasible, contact the IT team for alternative arrangements. Email: hpc.admin@snu.edu.in
 [[Category:HPC User Guides]]
 [[Category:Maintenance]]

← Older revision		Revision as of 11:16, 20 March 2026
Line 1:		Line 1:
−	== HPC Cluster Upgrade: Data Backup Guide (March 2026) == {\| class="wikitable" style="width: 100%;" \|- ! scope="col" style="background:#f2f2f2;" \| ~~<nowiki>~~'''~~</nowiki>~~Maintenance Window~~<nowiki>~~'''~~</nowiki>~~ \| Starting 27th March 2026 \|- ! scope="col" style="background:#f2f2f2;" \| ~~<nowiki>~~'''~~</nowiki>~~Backup Location~~<nowiki>~~'''~~</nowiki>~~ \| ~~<nowiki>~~<code>/backup/&~~</nowiki>~~lt;username~~<nowiki>~~&~~</nowiki>~~gt;~~<nowiki>~~</code>~~</nowiki>~~ \|- ! scope="col" style="background:#f2f2f2;" \| ~~<nowiki>~~'''~~</nowiki>~~Storage Limit~~<nowiki>~~'''~~</nowiki>~~ \| 10 GB per user \|- ! scope="col" style="background:#f2f2f2;" \| ~~<nowiki>~~'''~~</nowiki>~~Recommended Data~~<nowiki>~~'''~~</nowiki>~~ \| Source code, scripts, configuration files, and small datasets. \|} === Overview === During the extensive upgrade planned from ~~<nowiki>~~'''~~</nowiki>~~27th March~~<nowiki>~~'''~~</nowiki>~~, we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from ~~<nowiki>~~<code>/data/home/&~~</nowiki>~~lt;username~~<nowiki>~~&~~</nowiki>~~gt;~~<nowiki>~~</code~~></nowiki~~> to the temporary backup partition. === Step 1: Verify Current Usage === Check the size of your home directory to ensure you stay within the ~~<nowiki>~~'''~~</nowiki>~~10 GB~~<nowiki>~~'''~~</nowiki>~~ limit for the backup partition. ~~<nowiki>~~<syntaxhighlight lang="bash"> du -sh /data/home/$USER </syntaxhighlight>~~</nowiki>~~ === Step 2: Compress Your Critical Files === To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts. ~~<nowiki>~~<syntaxhighlight lang="bash"> # Create a compressed backup of a specific project tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder </syntaxhighlight>~~</nowiki>~~ === Step 3: Synchronize to Backup Directory === Use ~~<nowiki>~~<code>rsync</code~~></nowiki~~> to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer. ==== Option A: Copying a Compressed Archive (Recommended) ==== ~~<nowiki>~~<syntaxhighlight lang="bash"> rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/ </syntaxhighlight>~~</nowiki>~~ ==== Option B: Copying a Directory Directly ==== ~~<nowiki>~~<syntaxhighlight lang="bash"> rsync -ahP /data/home/$USER/important_folder /backup/$USER/ </syntaxhighlight>~~</nowiki>~~ === Step 4: Verification === Confirm your files are present and check the total space used in the backup directory: ~~<nowiki>~~<syntaxhighlight lang="bash"> ls -lh /backup/$USER/ du -sh /backup/$USER/ </syntaxhighlight>~~</nowiki>~~ === Handling Large Data (>10 GB) === If you have a genuine need to back up more than 10 GB: # ~~<nowiki>~~'''~~</nowiki>~~Local Download:~~<nowiki>~~'''~~</nowiki>~~ Download the data to your local machine using SCP or SFTP. # ~~<nowiki>~~'''~~</nowiki>~~Contact IT:~~<nowiki>~~'''~~</nowiki>~~ If local download is not feasible, contact the IT team for alternative arrangements. ~~<nowiki>~~[[Category:HPC User Guides]]~~</nowiki> <nowiki>~~[[Category:Maintenance]]~~</nowiki>~~	+	== HPC Cluster Upgrade: Data Backup Guide (March 2026) ==
		+
		+	{\| class="wikitable" style="width: 100%;"
		+	\|-
		+	! scope="col" style="background:#f2f2f2;" \| '''Maintenance Window'''
		+	\| Starting 27th March 2026
		+	\|-
		+	! scope="col" style="background:#f2f2f2;" \| '''Backup Location'''
		+	\| <code>/backup/<username></code>
		+	\|-
		+	! scope="col" style="background:#f2f2f2;" \| '''Storage Limit'''
		+	\| 10 GB per user
		+	\|-
		+	! scope="col" style="background:#f2f2f2;" \| '''Recommended Data'''
		+	\| Source code, scripts, configuration files, and small datasets.
		+	\|}
		+
		+	=== Overview ===
		+	During the extensive upgrade planned from '''27th March''', we aim to keep all data intact. However, as a safety measure against any untoward incidents, all users are requested to back up their critical work from <code>/data/home/<username></code> to the temporary backup partition.
		+
		+	=== Step 1: Verify Current Usage ===
		+	Check the size of your home directory to ensure you stay within the '''10 GB''' limit for the backup partition.
		+	<syntaxhighlight lang="bash">
		+	du -sh /data/home/$USER
		+	</syntaxhighlight>
		+
		+	=== Step 2: Compress Your Critical Files ===
		+	To save space and speed up the transfer, we recommend creating a compressed archive (tarball) of your code and scripts.
		+	<syntaxhighlight lang="bash">
		+	# Create a compressed backup of a specific project
		+	tar -czvf ~/my_code_backup_$(date +%F).tar.gz /data/home/$USER/my_project_folder
		+	</syntaxhighlight>
		+
		+	=== Step 3: Synchronize to Backup Directory ===
		+	Use <code>rsync</code> to copy your files to the dedicated backup path. This tool ensures file integrity during the transfer.
		+
		+	==== Option A: Copying a Compressed Archive (Recommended) ====
		+	<syntaxhighlight lang="bash">
		+	rsync -ahP ~/my_code_backup_*.tar.gz /backup/$USER/
		+	</syntaxhighlight>
		+
		+	==== Option B: Copying a Directory Directly ====
		+	<syntaxhighlight lang="bash">
		+	rsync -ahP /data/home/$USER/important_folder /backup/$USER/
		+	</syntaxhighlight>
		+
		+	=== Step 4: Verification ===
		+	Confirm your files are present and check the total space used in the backup directory:
		+	<syntaxhighlight lang="bash">
		+	ls -lh /backup/$USER/
		+	du -sh /backup/$USER/
		+	</syntaxhighlight>
		+
		+	=== Handling Large Data (>10 GB) ===
		+	If you have a genuine need to back up more than 10 GB:
		+	# '''Local Download:''' Download the data to your local machine using SCP or SFTP.
		+	# '''Contact IT:''' If local download is not feasible, contact the IT team for alternative arrangements.
		+
		+	[[Category:HPC User Guides]]
		+	[[Category:Maintenance]]