abstract:farber:transfer

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
abstract:farber:transfer [2018-03-20 14:09] – sraskarabstract:farber:transfer [2021-12-01 11:10] (current) – anita
Line 1: Line 1:
-====== Transferring files ======+====== Transferring files to/from Farber ======
  
-//This section uses the wiki's [[:#documentation-conventions|documentation conventions]].//+//The following sections use the wiki's [[:#documentation-conventions|documentation conventions]].//
  
 <note important>Be careful about modifications you make to your startup files (e.g. ''.bash*''). Commands that produce output such as VALET commands may cause your file transfer command or application to not work. Log into the cluster with ''ssh'' to check what happens during login, and modify your startup files accordingly to remove any commands which are producing output and try again.</note> <note important>Be careful about modifications you make to your startup files (e.g. ''.bash*''). Commands that produce output such as VALET commands may cause your file transfer command or application to not work. Log into the cluster with ''ssh'' to check what happens during login, and modify your startup files accordingly to remove any commands which are producing output and try again.</note>
Line 14: Line 14:
 | **hpn-scp**| Recommended for batch script use with at least 1Gbps network connection, and high speed disk drives at both source and destination. ((**hpn-** command-line variants are based on [[http://www.psc.edu/index.php/hpn-ssh|High Performance SSH/SCP - HPN-SSH]], OpenSSH 6.1 with hpn13v14 patch.)) | | **hpn-scp**| Recommended for batch script use with at least 1Gbps network connection, and high speed disk drives at both source and destination. ((**hpn-** command-line variants are based on [[http://www.psc.edu/index.php/hpn-ssh|High Performance SSH/SCP - HPN-SSH]], OpenSSH 6.1 with hpn13v14 patch.)) |
 | **rsync**| Most appropriate for synchronizing the file directories of two systems when only a small fraction \\ of the files have been changed since the last synchronization. | | **rsync**| Most appropriate for synchronizing the file directories of two systems when only a small fraction \\ of the files have been changed since the last synchronization. |
 +| **Rclone**| [[https://rclone.org/|Rclone]] is a command line program to sync files and directories to and from popular cloud storage services. |
 +
 +
 ^ Graphical-user-interface clients include: || ^ Graphical-user-interface clients include: ||
 | [[http://udeploy.udel.edu/software/winscp/|winscp]]| Windows only | | [[http://udeploy.udel.edu/software/winscp/|winscp]]| Windows only |
Line 30: Line 33:
   * **rsync -e 'hpn-ssh -c arcfour'**   * **rsync -e 'hpn-ssh -c arcfour'**
  
-This option is **not** available for the [[general/connect/putty-command-line|PuTTY command line mode]] unless you use a saved session with your encryption cipher selection policy set to Arcfour (SSH-2 only) at the top of the list.+This option is **not** available for the [[abstract:farber:system_access:windows|PuTTY command line mode]] unless you use a saved session with your encryption cipher selection policy set to Arcfour (SSH-2 only) at the top of the list.
 </note> </note>
  
-<note warning>For **Windows** clients editing files on Windows desktops and then transferring them back to the cluster, you may find that your file becomes "corrupt" during file transfer process.  The symptoms are very subtle because the file appears to be okay, but in fact contains ''CRLF'' line terminators.  This causes problems when reading the file on a Unix cluster and generates very strange errors. Some examples might be a file used for submitting a batch job such as ''submit.qs'' and one you have used before and know is correct, will no longer work. Or an input file used for ABAQUS like ''tissue.inp'' which has worked many times before produces an error like ''Abaqus Error: Command line option "input" must have a value.''. +<note warning>For **Windows** clients editing files on Windows desktops and then transferring them back to the cluster, you may find that your file becomes "corrupt" during file transfer process.  The symptoms are very subtle because the file appears to be okay, but in fact contains ''CRLF'' line terminators.  This causes problems when reading the file on a Linux cluster and generates very strange errors. Some examples might be a file used for submitting a batch job such as ''submit.qs'' and one you have used before and know is correct, will no longer work. Or an input file used for ABAQUS like ''tissue.inp'' which has worked many times before produces an error like ''Abaqus Error: Command line option "input" must have a value.''. 
  
 Use the utility ''file'' to check for ''CRLF'' line terminators and ''dos2unix'' to fix it, like this below Use the utility ''file'' to check for ''CRLF'' line terminators and ''dos2unix'' to fix it, like this below
  
 <code> <code>
-[traine@mills ABAQUS]$ file tissue.inp+[traine@farber ABAQUS]$ file tissue.inp
 tissue.inp: ASCII text, with CRLF line terminators tissue.inp: ASCII text, with CRLF line terminators
-[traine@mills ABAQUS]$ dos2unix tissue.inp+[traine@farber ABAQUS]$ dos2unix tissue.inp
 dos2unix: converting file tissue.inp to UNIX format ... dos2unix: converting file tissue.inp to UNIX format ...
-[traine@mills ABAQUS]$ file tissue.inp+[traine@farber ABAQUS]$ file tissue.inp
 tissue.inp: ASCII text tissue.inp: ASCII text
 </code> </code>
Line 59: Line 62:
  
 <code bash> <code bash>
-  scp -c arcfour -r fuelcell traine@mills.hpc.udel.edu:/archive/it_css/projects+  scp -c arcfour -r fuelcell traine@farber.hpc.udel.edu:/home/work/it_css/projects
 </code> </code>
  
-copies the ''fuelcell'' directory in your local current working directory into the ''/archive/it_css/projects'' directory on the Mills cluster. The ''/archive/it_css/projects'' directory on the Mills cluster must exist, and ''traine'' must have write access to it.+copies the ''fuelcell'' directory in your local current working directory into the ''/home/work/it_css/projects'' directory on the Farber cluster. The ''/home/work/it_css/projects'' directory on the Farber cluster must exist, and ''traine'' must have write access to it.
  
 ===== Copying files from the cluster ===== ===== Copying files from the cluster =====
Line 78: Line 81:
  
 <code bash> <code bash>
-  scp -c arcfour -r traine@mills.hpc.udel.edu:/archive/it_css/project/fuelcell  .+  scp -c arcfour -r traine@farber.hpc.udel.edu:/home/work/it_css/project/fuelcell  .
 </code> </code>
  
-will copy the directory ''fuelcell'' on the Mills cluster into a new ''fuelcell'' directory in your local system's current working directory. (Note the final period in the command.) +will copy the directory ''fuelcell'' on the Farber cluster into a new ''fuelcell'' directory in your local system's current working directory. (Note the final period in the command.)
 ===== Copying files between clusters ===== ===== Copying files between clusters =====
  
Line 108: Line 110:
  
 will copy the directory ''fuelcell'' from Mills to a new ''fuelcell'' directory on Farber. will copy the directory ''fuelcell'' from Mills to a new ''fuelcell'' directory on Farber.
- 
  
  • abstract/farber/transfer.txt
  • Last modified: 2021-12-01 11:10
  • by anita