Setting up Rclone with Google Drive
Introduction – What is Rclone?
Rclone is a fantastic and free command-line tool to manage files in cloud storage. It has the capability to work with over 40 cloud storage products and is much more feature rich and and powerful than the traditional web interfaces that most cloud storage providers offer. In this article, I will be focusing specifically on its integration with Google Drive and how you can utilize it to copy and then sync files with Google Drive.
The major feature that draws me to Google Drive is the fact that you can make use of unlimited* storage with a GSuite Business/Workspace account. Google says that they limit usage to 1TB for accounts with less than five users but has never enforced this. Should they ever end up enforcing it, Google always handles the downgrading of storage for accounts in a non-destructive manner – meaning your files will continue to exist on Google Drive in a worst-case scenario. Regardless, the future of Google continuing to offer unlimited storage in the manner they are now is uncertain. I would personally not rely on this method of storage for anything mission critical.
Getting Started
Creating a Google Drive API Key
To preface, while you're able to use Rclone's public Google Drive API key for all of this, I highly encourage you to create your own. Since each API key has a global rate limit on the number of queries per second that it can do, and with RClone's API key being shared among everyone else using it, it is likely you could run into some issues getting rate limited by using Rclone's key. To not worry about having to deal with any of this in the future, you can create your own. It's very simple if you follow these instructions:
- Visit the Google API Console here
- Select a project. If you don't already have one created, create one.
- Click "Enable APIs and Services" and then select Google Drive
- You will then need to click on the "Credentials" tab on the left and then "Create Credentials at the top of the screen.
- Once here, you'll want to create an OAuth Client ID
- If you haven't already, the site will prompt you to create a consent screen.
- You will be able to choose between "Internal" and "External". For our purposes, you'll want to select "External"
- Once on the consent screen configuration page, give your application a name and domain to link it to. Much of the other fields here are optional but I encourage you to fill out as much as we can here.
- Once done, simply hit "Save"
- Next, you'll want to select an Application Type. Here, you'll want to select the type of device that most directly pertains to your use case.
- Finally, you'll want to grab your Client ID and Secret and keep them handy for later on in our Rclone configuration.
Installing Rclone
To install Rclone, simply run the command found here on Rclone's site:
curl https://rclone.org/install.sh | sudo bash
I encourage you to use this command, versus simply installing it via your distro's package manager to avoid any issues in running an old version of the program.
Configuring Rclone
Once installed, you're able to begin configuring the application by running rclone config
Here, we'll be provided the following options:
No remotes found - make a new one
n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit configSince you have no existing remotes, start by entering n to begin.
After having named your remote, you will be shown a list of various cloud storage options that Rclone is compatible with. Locate "Google Drive" and enter the number associated with it.
XX / Google Drive
\ "drive"
Storage> driveNow you will supply the Client ID and Client Secret generated in one of the earlier steps:
Google Application Client Id
client_id> enter_client_id_here
Google Application Client Secret
client_secret> enter_client_secret_here
Next, you will be able to chose between 4 different scope options
Scope that rclone should use when requesting access from drive.
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"
scope>
For our purposes, we'll stick with option 1.
You will now be given the option to restrict your rclone remote to a specific folder or Team Drive. To do so, simply set "root_folder_id" to the folder ID. This can be easily found by looking at the URL of your folder/Team Drive.
For example, navigating to a team drive, I might see the URL as: https://drive.google.com/drive/u/1/folders/jk23jkfsd3
I'll want to copy and paste the jk23jkfsd3 part of the URL as my root_folder_id.
ID of the root folder - leave blank normally. Fill in to access "Computers" folders. (see docs).
root_folder_id> jk23jkfsd3
For your next step, you will be given the option to configure a Service Account but for our purposes, we'll skip over this option. We'll also skip over the "advanced config" option following this step.
Next, you'll be able to give rclone permission to your Google Drive. If you are on a desktop, enter "y". If you're on a headless machine, enter "n". For my setup, I'll be entering "n".
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n
Please go to the following link: https://accounts.google.com/o/oauth2/auth?access_type=offline&client_id=dsagfresfdg&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%32324A%32%Fwww.googleapis.com%2Fauth%2Fdrive&state=XJbsd3#fl@!0fs1jg
Log in and authorize rclone for access
Enter verification code> XXXXXXXXXXXXXXXXXXXXXXXXXX
Depending on which selection you make, you'll either have your browser open and provide you a link to Google's OAuth page, which you can then grant access and be given a verification code. If you're on a headless install, you'll simply be given a link to paste in your browser to do the same process. After having obtained the verification code, simply paste it into your terminal where rclone should now be requesting it.
In the next step, you'll be asked whether you want to set this remote up with a Team Drive or not. This is up to you but I suggest using a Team Drive purely to keep things a bit more organized.
Configure this as a team drive?
y) Yes
n) No
y/n> y
Finally, you will be prompted to review your input and ensure that it's configured correctly. After confirming, you are now able to mount the drive but in our current setup, this will not happen automatically so let's look at setting rclone up with a systemd service.
[Gdrive_Mount]
client_id = XXXXXXXXXXXXXXXXXXXX
client_secret = XXXXXXXXXXXXXXXXXXXX
scope = drive
root_folder_id = jk23jkfsd3
service_account_file =
token = {"access_token":"XXXXXXXXXXXXXXXXXXXX","token_type":"Bearer","refresh_token":"XXXXXXXXXXXXXXXXXXXX","expiry":"XXXXXXXXXXXXXXXXXXXX"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> ySetting up Crypt (Optional)
This step is optional and is for users who would like to have their files encrypted before sending them up to Google Drive. Personally, I do this for everything I throw up there, even if it's at the expense of a small amount of convenience.
Now that you've become a bit more familiar with Rclone, and the process involved in setting up an rclone remote, let's go ahead and dive into ~/.config/rclone/rclone.conf. Use the text editor of your choice
[gdrive_mount]
type = drive
client_id = XXXXXXXXXXXXXXXXXXXX
client_secret = XXXXXXXXXXXXXXXXXXXX
scope = drive
root_folder_id = jk23jkfsd3
token = token = {"access_token":"XXXXXXXXXXXXXXXXXXXX","token_type":"Bearer","refresh_token":"XXXXXXXXXXXXXXXXXXXX","expiry":"XXXXXXXXXXXXXXXXXXXX"}
team_drive = XXXXXXXXXXXXXXXXXXXXYour ~/.config/rclone/rclone.conf file should look something like what is shown above. You will need to edit the file to include your cache and crypt.
[gdrive_mount]
type = drive
client_id = XXXXXXXXXXXXXXXXXXXX
client_secret = XXXXXXXXXXXXXXXXXXXX
scope = drive
root_folder_id = jk23jkfsd3
token = token = {"access_token":"XXXXXXXXXXXXXXXXXXXX","token_type":"Bearer","refresh_token":"XXXXXXXXXXXXXXXXXXXX","expiry":"XXXXXXXXXXXXXXXXXXXX"}
team_drive = XXXXXXXXXXXXXXXXXXXX
[gdrive_mount_crypt]
type = crypt
remote = gdrive_mount:
filename_encryption = standard
directory_name_encryption = trueIn the end, you will want to emulate this setup in your rclone.conf. For your crypt, remote = will need to point to your mount name. As you can see, I have mine set to remote = gdrive_mount:
Setting Encryption Password and Salt
It is now time to launch rclone config once more to add the encryption password and salt.
Current remotes:
Name Type
==== ====
gdrive_mount drive
gdrive_mount_crypt crypt
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> e
Choose a number from below, or type in an existing value
1 > gdrive_mount
2 > gdrive_mount_crypt
remote> 3
--------------------
[gdrive_mount_crypt]
type = crypt
remote = gdrive_mount:
filename_encryption = standard
directory_name_encryption = true
--------------------
Edit remote
** See help for crypt backend at: https://rclone.org/crypt/ **
Value "filename_encryption" = "standard"
Edit? (y/n)>
y) Yes
n) No (default)
y/n> n
Value "directory_name_encryption" = "true"
Edit? (y/n)>
y) Yes
n) No (default)
y/n> n
Value "password" = ""
Edit? (y/n)>
y) Yes
n) No (default)
y/n> y
Password or pass phrase for encryption.
y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:
Value "password2" = ""
Edit? (y/n)>
y) Yes
n) No (default)
y/n> y
Password or passphrase for salt. Optional but recommended.
Should be different to the previous password.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:
--------------------
[gdrive_mount_crypt]
type = crypt
remote = gdrive_mount:
filename_encryption = standard
directory_name_encryption = true
password = *** ENCRYPTED ***
password2 = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> yMounting the Remote
In just a bit, I will show you how to configure rclone to mount upon system startup using systemd but for now, let's test to ensure functionality of the remote that you just created.
To start, create a directory in /mnt/ on the machine you'd like to use with rclone.
mkdir /mnt/gdriveNow, to mount rclone to your newly created /mnt/ point, you will need to run the following command:
rclone mount gdrive_mount_crypt: /mnt/gdrive --allow-other --cache-db-purge --fast-list --poll-interval 10mNote that I specified gdrive_mount_crypt:, rather than gdrive_mount. This is because I'm wanting data to pass through the crypt, encrypting it before it goes on to be stored in Google Drive.
Configuring Systemd
As I briefly spoke on previously, manually running the rclone mount command will work but does not persist between reboots or run automatically, which I prefer to have in place.
To make this process happen, navigate to /etc/systemd/system. Here, you will want to make a new .service file that will let us automatically have rclone mount when the machine boots up.
Create a file named gdrive.service and edit it with a text editor of your choosing. I use VIM, which allows me to do this all in one command.
vim gdrive.service
[Unit]
Description=rclone for gdrive_mount
AssertPathIsDirectory=/mnt/gdrive
After=networking.service
[Service]
Type=simple
ExecStart=rclone mount --config=/home/tcude/.config/rclone/rclone.conf gdrive_mount_crypt: /mnt/gdrive --allow-other --cache-db-purge --fast-list --poll-interval 10m
ExecStop=/bin/fusermount -u /mnt/gdrive
Restart=always
RestartSec=10
[Install]
WantedBy=default.targetPopulate the file with what I have displayed above and tweak the necessary parts of it, such as gdrive_mount to what corresponds with the name of your mount. You'll also need to ensure you replace tcude with the name of your user.
After saving the new gdrive.service file, you can ensure it's used upon startup by running the command:
systemctl enable gdrive.serviceAt this point, the service will not actually be running. To enable it, use the command:
systemctl start gdrive.serviceUsage
I encourage you to visit rclone's documentation for Google Drive here to understand how to fully utilize your new rclone mount.
For my purposes, I've found that the following command is what I use most:
rclone sync /path/to/local/dir gdrive_crypt: --bwlimit=8.5M --progress
Breaking things down a bit:
rclone syncwill sync the contents of the directory you specify with the remote you're sending them to. This means, if you add a file to the local directory and run the command, it will then show up on your gdrive remote. The same goes for deleting a file in your local directory--bwlimit=8.5Mlimits the bandwidth rate to 8.5M, keeping you from hitting your daily maximum upload imposed by Google--progressdoes exactly what it sounds and provides output showing which files are currently being cloned/synced to gdrive, how much time is left for the clone/sync to finish, and how fast the clone/sync is being transferred
Conclusion
While the setup process in getting rclone working with Google Drive is a bit of a long one, I have found it to be very handy in backing up files with ease.
Should you want to improve upon things even more, look into creating a simple script to run the rclone sync command, add a cronjob, and you now have an automated offsite backup for a directory of your choosing!
I hope you found this guide helpful. If you did, consider adding my site to your RSS reader using the following URL:
Next up:
Check out this post, covering the process of enabling Rclone's web GUI!