Rclone

Rclone is a program that syncs local directories to cloud services. The Rclone project is at: http://rclone.org

HashBackup can use Rclone as a file transfer agent via HB’s "shell destination" to access cloud storage systems that are not natively supported in HashBackup. If there is native support for a destination, it should always be preferred over a shell destination.

rclone.py is an HB shell destination, written in Python, that runs Rclone to get, send, and delete files on any storage provider supported by Rclone. For example, Rclone supports Google Drive, which is not natively supported in HashBackup. Using the rclone.py shell destination, HashBackup gains the ability to use Google Drive via Rclone.

Steps to use Rclone with HashBackup:

  1. download and install Rclone from http://rclone.org

  2. run rclone config to create a .rclone.conf file

  3. download the rclone.py shell destination script to your backup directory. If you will have multiple backups using this script, you might want to copy it to a single directory all backups can access.

  4. read the Destination Setup page and Shell Destination page to see how destinations work in general, and how shell destinations work. Then read this page to see how the Rclone shell destination works.

  5. create a dest.conf file in your HB backup directory. Below are example destinations using Rclone.

  6. Before your first backup with any shell destination, do:

$ hb dest -c backupdir setid <shell destname>

If you skip this step, HB will display errors about DESTID on your first backup and will not start the destination.

IMPORTANT NOTES

  • the destname keyword in dest.conf and --destname are HB values and should match

  • the Rclone remote name following --clonedir `must match the name used with `rclone config.

  • the run line is all one line, though it may not display that way here

Example

destname gdrive
type shell
debug 1
run python2 /home/jim/hb/rclone.py --debug --args "-v " --destname gd --backupdir /home/jim/hb --clonedir gdrive:test --command

The debug keyword (shown beneath type shell) is optional. It causes the HB shell destination to display debug info. Specifically, it displays the command line used to run the rclone.py script.

Shell destinations support the workers keyword to control how many threads are used.

The run keyword controls the rclone.py script. It must be all on one line and has several keywords. You can use run python2.7 …​ to use a different version of python to run this script

rclone.py runs the rclone command without a pathname, so the rclone executable must be in your current path. For cron jobs, add PATH=…​ before your crontab entry to set the path cron uses to find rclone.

The options below apply only to the rclone.py shell destination. Every shell script has its own set of options. These options are all used on the run line.

--debug is optional and will cause rclone.py to display more about what it is doing and what is happening. It displays the complete Rclone command line before it is executed.

--args is optional. Anything in the next quoted string will be added to the Rclone command line. In the examples above, -v is being added to the Rclone command so that Rclone displays detailed output about its operations. See the Rclone option list for examples of what you can put here.

IMPORTANT GOTCHA! With --args there must be a space in the quoted string or Python will complain.

--destname is required. It tells the rclone.py script what your destination name is, usually to display error messages.

--backupdir is required. It is your HB -c backup directory.

--clonedir is your Rclone remote destination directory. Different types of storage may require different things with --clonedir.

--command is required and must be at the end of the run line.

When trying to debug problems, you will likely want to use debug 1, add --debug to the run `line, and add `--args "-v " to get as much output as possible. It’s also a good idea to use workers 1 when debugging, to avoid mixing multiple workers' output. Debug output has a process ID in square brackets so you can follow the messages for a specific worker.

Some of the destinations Rclone supports can be quite slow, and many use throttling, limits and quotas. You may need patience!