abstract:caviness:transfer:rclone

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
abstract:caviness:transfer:rclone [2025-05-21 16:10] – [Testing new Rclone remote configuration] thuachenabstract:caviness:transfer:rclone [2025-05-21 16:10] (current) – [Configuration Steps] thuachen
Line 1: Line 1:
 +====== Rclone on Caviness======
 +
 +This document describes how to manually configure (set up) Rclone using University of Delaware email account for Google Drive on Caviness.
 +
 +<note important>Due to firewall rules, the standard documentation for setting up [[https://rclone.org/|Rclone]] on Caviness might not work for all cloud storage providers. The following directions will show you how to manually set up Rclone and avoid issues with the firewall. </note>
 +
 +===== Rclone manual configuration of Google Drive =====
 +==== Loading Rclone ====
 +First, use VALET to load Rclone into your environment; you must load the latest version ''rclone/1.69.2'' as the old versions will not work on Caviness.
 +<code>
 +[traine@login00.caviness ~]$ vpkg_versions rclone
 +
 +Available versions in package (* = default version):
 +
 +[/opt/shared/valet/2.1/etc/rclone.vpkg_yaml]
 +rclone    Rclone is a command line program to sync files and directories to and from cloud services
 +  1.45    Pre-built executable for AMD64/Linux
 +  1.50    Pre-built executable for AMD64/Linux
 +* 1.69.2  official release 1.69.2
 +[traine@login00.caviness ~]$ vpkg_require rclone/1.69.2
 +Adding package `rclone/1.69.2` to your environment
 +</code>
 +
 +==== Configuration Steps ====
 +Create a new Rclone remote configuration by typing ''rclone config''. In this example, we are going to call this new configuration ''cavinessGoogleDrive''. You don't have to use this name, you can call your configuration whatever you would like. 
 +
 +<code>
 +[traine@login00 ~]$ rclone config
 +2025/05/15 15:49:07 NOTICE: Config file "/home/1200/.config/rclone/rclone.conf" not found - using defaults
 +No remotes found - make a new one
 +n) New remote
 +s) Set configuration password
 +q) Quit config
 +n/s/q> n
 +name> cavinessGoogleDrive
 +</code>
 +Next, find the "Google Drive" option from the given list of cloud services. Once it is located, enter its value. For this example, it is ''drive''. These options can change so please make sure that you read and choose the right option.
 +
 +<code>
 +Option Storage.
 +Type of storage to configure.
 +Choose a number from below, or type in your own value.
 + 1 / 1Fichier
 +   \ (fichier)
 + 2 / Akamai NetStorage
 +   \ (netstorage)
 + 3 / Alias for an existing remote
 +   \ (alias)
 +
 +... Skipping lines...
 +   
 +18 / Gofile
 +   \ (gofile)
 +19 / Google Cloud Storage (this is not Google Drive)
 +   \ (google cloud storage)
 +20 / Google Drive
 +   \ (drive)
 +21 / Google Photos
 +   \ (google photos)
 +22 / HTTP
 +   \ (http)
 +
 +... Skipping more lines ...
 +
 +59 / iCloud Drive
 +   \ (iclouddrive)
 +60 / premiumize.me
 +   \ (premiumizeme)
 +61 / seafile
 +   \ (seafile)
 +Storage> drive
 +</code>
 +
 +The next couple of steps will be ask about adding a ''client_id'' and a ''client_secret''. Unless you know you will need to use these, the default blank option is all that needs to be entered.
 +
 +<code>
 +Option client_id.
 +Google Application Client Id
 +Setting your own is recommended.
 +See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
 +If you leave this blank, it will use an internal key which is low performance.
 +Enter a value. Press Enter to leave empty.
 +client_id>
 +
 +Option client_secret.
 +OAuth Client Secret.
 +Leave blank normally.
 +Enter a value. Press Enter to leave empty.
 +client_secret>
 +</code>
 +The next step will give you options to which you will grant Rclone permission to access your files and directories in your Google Drive. We are going to use the default value of ''full access'' but you can read the different options and choose the best option for you. 
 +
 +<code>
 +Option scope.
 +Comma separated list of scopes that rclone should use when requesting access from drive.
 +Choose a number from below, or type in your own value.
 +Press Enter to leave empty.
 + 1 / Full access all files, excluding Application Data Folder.
 +   \ (drive)
 + 2 / Read-only access to file metadata and file contents.
 +   \ (drive.readonly)
 +   / Access to files created by rclone only.
 + 3 | These are visible in the drive website.
 +   | File authorization is revoked when the user deauthorizes the app.
 +   \ (drive.file)
 +   / Allows read and write access to the Application Data folder.
 + 4 | This is not visible in the drive website.
 +   \ (drive.appfolder)
 +   / Allows read-only access to file metadata but
 + 5 | does not allow any access to read or download file content.
 +   \ (drive.metadata.readonly)
 +scope> 1
 +</code>
 +For the next question the default blank choices should be all that is needed.
 +<code>
 +Option service_account_file.
 +Service Account Credentials JSON file path.
 +Leave blank normally.
 +Needed only if you want use SA instead of interactive login.
 +Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
 +Enter a value. Press Enter to leave empty.
 +service_account_file>
 +</code>
 +For the next question ''advance config'' is generally not required so ''n'' for the no option is typically the correct option. 
 +<code>
 +Edit advanced config? (y/n)
 +y) Yes
 +n) No (default)
 +y/n> n
 +</code>
 +
 +So far we have been answering the configuration questions mostly with the default options. However, the next question is where we will differ from the typical setup process. Since firewall permissions will not allow the ''auto config'' to work correctly, we will say ''n''. This option will allow us to safely work around the firewall permission issue by allowing us to request a verification code from our local machine. Then we can manually copy and paste the verification code to Caviness. 
 +<code>
 +Use web browser to automatically authenticate rclone with remote?
 + * Say Y if the machine running rclone has a web browser you can use
 + * Say N if running rclone on a (remote) machine without web browser access
 +If not sure try Y. If Y failed, try N.
 +
 +y) Yes (default)
 +n) No
 +y/n> n
 +</code>
 +After you enter ''n,'' you will be instructed to run ''rclone authorize'' on your local computer. The next step is to [[https://rclone.org/downloads/|Download the Rclone release v1.69.2]] on your local computer and run ''rclone authorize''.
 +<code>
 +Option config_token.
 +For this to work, you will need rclone available on a machine that has a web browser available.
 +For more help and alternate methods see: https://rclone.org/remote_setup/
 +Execute the following on the machine with the web browser (same rclone
 +version recommended):
 +        rclone authorize "drive" "base64_json_blob"
 +Then paste the result.
 +Enter a value.
 +config_token>
 +</code>
 +<note important>
 +The following command used to download Rclone and authorize will need to be run on your local computer, not on Caviness. The original terminal will need to remain open since we want to paste the code in a future step. The name of the drive (e.g., "drive") and a base64 encoded JSON blob obtained from a previous Rclone config session (e.g., "qwGDY34wZSI6ImRzdNMlIt8") may be different in your case. For more details about installing Rclone, please see [[https://rclone.org/install/|Rclone install]].
 +</note>
 +
 +For Linux OS, open a new terminal and run the following command on your local computer: 
 +<code>
 +curl -O https://downloads.rclone.org/v1.69.2/rclone-v1.69.2-linux-amd64.zip
 +unzip rclone-v1.69.2-linux-amd64.zip
 +cd rclone-v1.69.2-linux-amd64
 +./rclone authorize "drive" "base64_json_blob"
 +</code> 
 +
 +For macOS, open a new terminal and run the following command. Download the [[https://downloads.rclone.org/v1.69.2/rclone-v1.69.2-osx-arm64.zip|Rclone for ARM]] if your machine is equipped with ARM chips. You may also need to allow running Rclone in the security settings. 
 +<code>
 +cd && curl -O https://downloads.rclone.org/v1.69.2/rclone-v1.69.2-osx-amd64.zip
 +unzip -a rclone-v1.69.2-osx-amd64.zip && cd rclone-v1.69.2-osx-amd64
 +./rclone authorize "drive" "base64_json_blob"
 +</code>
 +
 +For Windows, first download [[https://downloads.rclone.org/v1.69.2/rclone-v1.69.2-windows-amd64.zip|Rclone v1.69.2 for Windows]], then extract all files. Open ''Windows PowerShell'', and then change to the folder where you put the ''rclone.exe'', run the following command:
 +<code>
 +.\rclone.exe authorize "drive" "base64_json_blob"
 +</code>  
 +After running the command, you will be given a URL(e.g., http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx) that will allow you to sign into your university Google email account and request a verification code for Rclone to use. Copy this URL and paste it into a web browser on your local computer. Sign in and afterwards you will be given your verification code. Copy and paste this verification code into the Caviness shell session.  It will look something like this below:
 +<code>
 +025/05/14 15:01:53 NOTICE: Config file "/home/traine/.config/rclone/rclone.conf" not found - using defaults
 +2025/05/14 15:01:53 NOTICE: Make sure your Redirect URL is set to "http://127.0.0.1:53682/" in your custom config.
 +2025/05/14 15:01:53 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=xxxxxxxxxxxx
 +2025/05/14 15:01:53 NOTICE: Log in and authorize rclone for access
 +2025/05/14 15:01:53 NOTICE: Waiting for code...
 +2025/05/14 15:02:19 NOTICE: Got code
 +Paste the following into your remote machine --->
 +***YOUR UNIQUE VERIFICATION CODE***
 +<---End paste
 +</code>
 +<note important>
 +The following options will be displayed in the original terminal where you configure Rclone on a Caviness shell session.
 +</note>
 +
 +After you paste the verification code, the last step will ask you if you want to configure this as a "Shared Drive(Team Drive)". For this example we will say no.
 +<code>
 +Configure this as a Shared Drive (Team Drive)?
 +y) Yes
 +n) No (default)
 +y/n> n
 +</code>
 +
 +Now you can review and your configuration and make edits if needed. If everything is okay, select ''y''.
 +<code>
 +Configuration complete.
 +Options:
 +- type: drive
 +- scope: drive
 +- token: {"access_token":"**The rest of this is private and unique information for each user.**"}
 +- team_drive:
 +Keep this "cavinessGoogleDrive" remote?
 +y) Yes this is OK (default)
 +e) Edit this remote
 +d) Delete this remote
 +y/e/d> y
 +</code>
 +
 +Once everything is configured correctly, Rclone will show you a list of your remote configurations, and give you options to modify, create a new one, or simply exit the Rclone configuration.  At this point, assuming all steps were followed as described, we will simple ''Quit Config'' with the ''q'' option, and press ''Enter''. 
 +
 +<code>
 +Name                   Type
 +====                   ====
 +cavinessGoogleDrive    drive
 +
 +e) Edit existing remote
 +n) New remote
 +d) Delete remote
 +r) Rename remote
 +c) Copy remote
 +s) Set configuration password
 +q) Quit config
 +e/n/d/r/c/s/q> q
 +</code>
 +==== Testing new Rclone remote configuration ====
 +
 +To make sure the new remote configuration is working correctly, we will use the Rclone ''lsd'' command to list out the root level directories of the remote configuration, ''cavinessGoogleDrive'', for our University of Delaware Google Drive on Caviness.
 +
 +<code bash>
 +[traine@login00 ~]$ rclone lsd cavinessGoogleDrive:
 +          -1 2020-11-16 15:43:38        -1 Introduction to Unix Shell on Caviness GDoc
 +          -1 2020-05-05 16:46:11        -1 wifi-map
 +</code>
 +
 +For additional information on Rclone and more options please visit [[https://rclone.org/|Rclone]].
 +
 +
  
  • abstract/caviness/transfer/rclone.txt
  • Last modified: 2025-05-21 16:10
  • by thuachen