Differences

This shows you the differences between two versions of the page.

--- abstract:darwin:transfer [2021-04-28 13:06] – [Copying files between clusters] anita
+++ abstract:darwin:transfer [2024-04-08 17:01] (current) – [Common clients for file transfer] anita
@@ Line 10: / Line 10: @@
 ^ Command-line clients include: ^^
 | **sftp**| Recommended for interactive, command-line use. |
-| **hpn-sftp**| Recommended for interactive, command-line use with at least 1Gbps network connection, and high speed disk drives at both source and destination.((**hpn-** command-line variants are based on [[http://www.psc.edu/index.php/hpn-ssh|High Performance SSH/SCP - HPN-SSH]], OpenSSH 6.1 with hpn13v14 patch.)) |
 | **scp**| Recommended for batch script use. |
-| **hpn-scp**| Recommended for batch script use with at least 1Gbps network connection, and high speed disk drives at both source and destination. ((**hpn-** command-line variants are based on [[http://www.psc.edu/index.php/hpn-ssh|High Performance SSH/SCP - HPN-SSH]], OpenSSH 6.1 with hpn13v14 patch.)) |
 | **rsync**| Most appropriate for synchronizing the file directories of two systems when only a small fraction \\ of the files have been changed since the last synchronization. |
+| **Globus**| [[https://www.globusonline.org/|Globus]] web browser based and recommended for 'fire and forget' high-performance data transfers between systems within and across organizations.\\ See [[software:globus:globus|Globus]] for more details.|
 | **Rclone**| [[https://rclone.org/|Rclone]] is a command line program to sync files and directories to and from popular cloud storage services. |
@@ Line 22: / Line 21: @@
 | [[http://filezilla-project.org/download.php?type=client|filezilla]] | Windows, Mac OS X, UNIX, Linux |
 | [[http://cyberduck.ch/|cyberduck]] | Windows, Mac OS X (command line version for Linux) |
+| [[https://www.globusonline.org/|Globus]] | Web browser. See [[software:globus:globus|Globus]] for more details.|
-<note tip>
+<note warning>For **Windows** clients editing files on Windows desktops and then transferring them back to the cluster, you may find that your file becomes "corrupt" during file transfer process.  The symptoms are very subtle because the file appears to be okay, but in fact contains ''CRLF'' line terminators.  This causes problems when reading the file on a Linux cluster and generates very strange errors. Some examples might be a file used for submitting a batch job such as ''submit.qs'' and one you have used before and know is correct, will no longer work. Or an input file used for ABAQUS like ''tissue.inp'' which has worked many times before produces an error like ''Abaqus Error: Command line option "input" must have a value.''.
-Specifying the //-c arcfour// cipher option for command-line clients should provide the best transfer speeds.
-  * **sftp -c arcfour**
-  * **hpn-sftp -c arcfour**
-  * **scp -c arcfour**
-  * **hpn-scp -c arcfour**
-  * **rsync -e 'ssh -c arcfour'**
-  * **rsync -e 'hpn-ssh -c arcfour'**
-This option is **not** available for the [[abstract:darwin:system_access:windows|PuTTY command line mode]] unless you use a saved session with your encryption cipher selection policy set to Arcfour (SSH-2 only) at the top of the list.
-</note>
-<note warning>For **Windows** clients editing files on Windows desktops and then transferring them back to the cluster, you may find that your file becomes "corrupt" during file transfer process.  The symptoms are very subtle because the file appears to be okay, but in fact contains ''CRLF'' line terminators.  This causes problems when reading the file on a Unix cluster and generates very strange errors. Some examples might be a file used for submitting a batch job such as ''submit.qs'' and one you have used before and know is correct, will no longer work. Or an input file used for ABAQUS like ''tissue.inp'' which has worked many times before produces an error like ''Abaqus Error: Command line option "input" must have a value.''.
 Use the utility ''file'' to check for ''CRLF'' line terminators and ''dos2unix'' to fix it, like this below
@@ Line 62: / Line 49: @@
 <code bash>
-  scp -r fuelcell traine@darwin.hpc.udel.edu:/home/work/it_css/projects
+  scp -r fuelcell traine@darwin.hpc.udel.edu:/lustre/it_css/users/1201/projects
 </code>
-copies the ''fuelcell'' directory in your local current working directory into the ''/home/work/it_css/projects'' directory on the DARWIN cluster. The ''/home/work/it_css/projects'' directory on the DARWIN cluster must exist, and ''traine'' must have write access to it.
+copies the ''fuelcell'' directory in your local current working directory into the ''/lustre/it_css/users/1201/projects'' directory on the DARWIN cluster. The ''/lustre/it_css/users/1201/projects'' directory on the DARWIN cluster must exist, and ''traine'' must have write access to it.
 ===== Copying files from the cluster =====
@@ Line 81: / Line 68: @@
 <code bash>
-  scp -r traine@darwin.hpc.udel.edu:/home/work/it_css/project/fuelcell  .
+  scp -r traine@darwin.hpc.udel.edu:/lustre/it_css/users/1201/projects/fuelcell  .
 </code>
@@ Line 106: / Line 93: @@
   ssh -A traine@caviness.hpc.udel.edu
   cd archive/it_css/project
-  scp -r fuelcell traine@darwin.hpc.udel.edu:/home/work/it_css/project/fuelcell
+  scp -r fuelcell traine@darwin.hpc.udel.edu:/lustre/it_css/users/1201/projects/fuelcell
 </code>
 will copy the directory ''fuelcell'' from Farber to a new ''fuelcell'' directory on DARWIN.