Rclone
Rclone is a program that syncs local directories to cloud services. The Rclone project is at: http://rclone.org
HashBackup can use Rclone as a file transfer agent via HB’s "shell destination" to access cloud storage systems that are not natively supported in HashBackup. If there is native support for a destination, it should always be preferred over a shell destination.
rclone.py
is an HB shell destination, written in Python, that runs
Rclone to get, send, and delete files on any storage provider
supported by Rclone. For example, Rclone supports Google Drive, which
is not natively supported in HashBackup. Using the rclone.py
shell
destination, HashBackup gains the ability to use Google Drive via
Rclone.
Steps to use Rclone with HashBackup:
-
download and install Rclone from http://rclone.org
-
run
rclone config
to create a.rclone.conf
file -
download the
rclone.py
shell destination script to your backup directory. If you will have multiple backups using this script, you might want to copy it to a single directory all backups can access. -
read the Destination Setup page and Shell Destination page to see how destinations work in general, and how shell destinations work. Then read this page to see how the Rclone shell destination works.
-
create a
dest.conf
file in your HB backup directory. Below are example destinations using Rclone. -
Before your first backup with any shell destination, do:
$ hb dest -c backupdir setid <shell destname>
If you skip this step, HB will display errors about DESTID
on your
first backup and will not start the destination.
IMPORTANT NOTES
-
the
destname
keyword indest.conf
and--destname
are HB values and should match -
the Rclone remote name following
--clonedir `must match the name used with `rclone config
. -
the
run
line is all one line, though it may not display that way here
Example
destname gdrive
type shell
debug 1
run python2 /home/jim/hb/rclone.py --debug --args "-v " --destname gd --backupdir /home/jim/hb --clonedir gdrive:test --command
The debug
keyword (shown beneath type shell
) is optional. It
causes the HB shell destination to display debug info. Specifically,
it displays the command line used to run the rclone.py script.
Shell destinations support the workers
keyword to control how many
threads are used.
The run
keyword controls the rclone.py script. It must be all on
one line and has several keywords. You can use run python2.7 …
to
use a different version of python to run this script
rclone.py
runs the rclone command without a pathname, so the rclone
executable must be in your current path. For cron jobs, add
PATH=…
before your crontab
entry to set the path cron uses to
find rclone.
The options below apply only to the rclone.py shell destination.
Every shell script has its own set of options. These options are all
used on the run
line.
--debug
is optional and will cause rclone.py to display more about
what it is doing and what is happening. It displays the complete
Rclone command line before it is executed.
--args
is optional. Anything in the next quoted string will be
added to the Rclone command line. In the examples above, -v
is
being added to the Rclone command so that Rclone displays detailed
output about its operations. See the Rclone option list for examples
of what you can put here.
IMPORTANT GOTCHA! With --args
there must be a space in the quoted
string or Python will complain.
--destname
is required. It tells the rclone.py script what your destination name is, usually to display error messages.
--backupdir
is required. It is your HB -c
backup directory.
--clonedir
is your Rclone remote destination directory. Different types of storage may require different things with --clonedir
.
--command
is required and must be at the end of the run
line.
When trying to debug problems, you will likely want to use debug 1
,
add --debug
to the run `line, and add `--args "-v "
to get as much
output as possible. It’s also a good idea to use workers 1
when
debugging, to avoid mixing multiple workers' output. Debug output has
a process ID in square brackets so you can follow the messages for a
specific worker.
Some of the destinations Rclone supports can be quite slow, and many use throttling, limits and quotas. You may need patience!