Recover

Recovers the backup directory from a remote destination in a disaster situation. This command is used when the disk containing your local backup directory fails or the whole system fails or is unavailable for some reason such as a fire, theft, or lighting strike.

The recover command does not restore individual files like /etc/hosts. To do that, use the Get command.

Options

$ hb recover -c recoverdir [destname] [-a] [-n] [-o]
                                      [--force] [--check [N]]

The recover command downloads backup files from the specified remote destination, copies them to a new local backup directory, and re-generates the backup database. The destination is optional if only one is in dest.conf.

To do a recover, create an empty recover directory, copy the original backup’s key.conf and dest.conf files there, then run the recover command with -c and the new recover directory name. Do not run the init command before recover as this creates a new key.conf file.

The --force option starts recovery without a confirmation prompt.

The -n option skips downloading of archive files. Normally archive files are recovered if the cache-size-limit config option is -1 (the default), but this may require downloading a lot of data. In a disaster recovery situation, only some critical user files may need to be restored immediately. With -n, the local backup directory can be recovered quickly, the hb get command can be used to restore the critical user files, downloading only the required archive data, then the recover command without -n can be used later to recover all archive files. Another way to download arc files is to set the cache-size-limit config option to -1 and run selftest with either the -v3 or -v4 options. This can be combined with --inc to download arc files over time.

The -a option (archives) forces downloading of archive files even if the cache-size-limit config option is >= 0. This is useful if you previously didn’t want all archive files stored locally, but now are switching to a local NAS for example and do want all archive files stored locally. HB will set the cache-size-limit config option to -1 after the recover.

The -o option (overwrite) downloads files even if they already exist in the local backup directory. This is useful if the local copies are suspect for some reason (or you can delete everything but key.conf and dest.conf and start over).

The --check option recovers the main database in a slower way that can recover historical versions if necessary. This is only used if a normal recover cannot recover the latest database for some reason. The resulting database may not be the latest version. To recover the latest N versions of the database, use --check N`. This will recover databases from oldest to newest, keeping the latest N versions, and testing them from newest to oldest until one passes an integrity test. That becomes the backup database hb.db.

To recover your backup, you must have a copy of the dest.conf and key.conf files from your backup directory. These files are never copied to remote destinations. The dest.conf file is required to connect to a destination containing a copy of your backup, and the key.conf file is required to decode your encrypted backup data.

You MUST copy your dest.conf and key.conf files to one or more USB flash drives, or print these files, and store the copies in several locations separate from the backup data. If your backup is passphrase protected, you must also know the passphrase.

Test Recover

Some customers do test recovers to verify their backup, which is a great idea. To prevent accidentally overwriting remote data following a test recover, a file testonly.conf is created in the newly-recovered backup directory. While this file is present, downloads work but destinations cannot be modified (no upload or remove). This is necessary because after a test recover, both the real and test backup directories match the unique ID HashBackup uses to prevent accidental overwrites of remote data. An operation in the test directory that modifies the backup could unintentionally modify the remote data, potentially causing problems for the production backup. The testonly.conf file prevents that and allows tests to be run without allowing remote data to be changed.

For real recovers of the local backup directory, delete the testonly.conf file after performing a selftest or other commands to verify the recover was successful.

db-history-days

The main database, hb.db, is critical to operation of HashBackup. It stores file system metadata shown by ls -l such as file size, file names, etc., as well as recipes for how to restore individual files and directories. To ensure its safety and integrity, the HB database is backed up offsite using hb.db.N files, where N is a sequence number, along with arc files containing user data. These hb.db.N files can be thought of as incremental backups for the HashBackup database.

HB uses database software to ensure that power failures and other interruptions do not cause failures. But nothing is perfect, and it’s possible to disrupt database operations by copying database files while they are in use, or by memory parity errors on systems without ECC RAM (error correcting). In these situations, a database error may go unnoticed and not cause immediate problems, so as extra insurance, it is useful to have older versions of the database as a last resort. This is setup with the db-history-days config option.

When db-history-days is set to 15 (the default), recover --check can recover the last 15 daily versions of the main database, assuming there have been daily backups. If HB is run weekly, there will still be 15 earlier database backups, but they will be weekly versions. If db-history-days is set lower, there will be fewer versions saved and they will usually take less space on the remote storage service. Setting db-history-days to 1 uses the least remote storage, but also requires sending the entire database on every HB operation, which may not be feasible with large backups. And, there would be no historical versions available, which is a safety risk.

When db-history-days is set to 0 (zero), it is similar to setting it to 1 (one) in that only the most recent version is available to the recover command. But it is more efficient because changes are still sent as a series of small incrementals. This is how HashBackup operated before db-history-days (August 2024).

As a guide to remote space usage, a setting of 0 uses about 15% the space of hb.db, 1 uses about 20%, 7 uses about 40%, 15 uses 80%, 30 uses 100%, and so on.

The recover command tries to restore the most recent version of the HB database and runs an integrity check to make sure it doesn’t have problems. To recover an older version of the database, use the --check option. With an optional number added, --check N will restore up to N versions of the database and install the most recent version that passes an integrity test.

Offline Recover

HashBackup’s main backup directory can also be regenerated without using the recover command. The steps are:

  1. Create a new, empty directory

  2. Copy the key.conf into the new directory

  3. Copy all arc.* and hb.db.N files

  4. Run any HB command with -c and the new directory

HB will see that there is no hb.db and use the hb.db.N files to regenerate it. To recover older versions, use the HBCHECK environment variable, for example:

# HBCHECK=5 hb versions -c newbackupdir

If arc files were also copied to the new directory, use the hb config command to change cache-size-limit to -1, meaning all arc files are local. If arc files were not copied, a dest.conf file will be needed. The inex.conf file may need to be restored if it was customized.

Notes

  • The recover command fetches the backup database and archive files; it does not fetch individual files or directories that you may have backed up. For example, you can’t restore the /etc directory using the recover command; use the get command instead.

  • To recover all files from a remote destination, create a new directory with only the dest.conf and key.conf files, then run recover -c with this new directory.

  • If the local archive cache size is limited with the cache-size-limit config option, recover will not download archive files unless -a is used. Instead, archive files will be downloaded as needed.

Recover Session

In this example, the HashBackup development server’s backup is recovered to a cloud-based Linux VM. The server is an 80GB Mac Mini with daily backups since 2010, so there are over 3600 versions.

First the key.conf and dest.conf files are copied to an empty directory named bs on the VM, then the recover command is used to recover the backup database from Backblaze B2. The -n option prevents downloading arc files because they would require 50GB of space and this small single-CPU VM only has about 10GB of space available.

At each step of the database recovery, files are verified with cryptographic signatures, and after the database is reconstructed, a final signature is verified. An integrity check verifies that the rebuilt database passes consistency checks.

The download and reconstruction takes about 30 seconds and the database consistency check takes about 90 seconds. The cache size limit is then set to 0 because this VM has very limited disk space.

there may be skips in the hb.db.N file sequences! 
# time hb recover -c bs -n
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Using destinations in dest.conf

Recovering backup files from destination: b2
Files will be copied to: /root/bs

Proceed with recovery? y

Removed /root/bs/dest.db
Getting dest.db from b2
Getting hb.db from b2
Queueing hb.db files

Waiting for /root/bs/hb.db.6386
Loading hb.db.6386
Verified hb.db.6386 signature

Waiting for /root/bs/hb.db.6385
Loading hb.db.6385
Verified hb.db.6385 signature

Loading hb.db.6384
Verified hb.db.6384 signature

Loading hb.db.6383
Verified hb.db.6383 signature

Loading hb.db.6382
Verified hb.db.6382 signature

Loading hb.db.6381
Verified hb.db.6381 signature

Loading hb.db.6380
Verified hb.db.6380 signature

Loading hb.db.6379
Verified hb.db.6379 signature

Loading hb.db.6378
Verified hb.db.6378 signature

Loading hb.db.6376
Verified hb.db.6376 signature

Loading hb.db.6375
Verified hb.db.6375 signature

Loading hb.db.6364
Verified hb.db.6364 signature

Loading hb.db.6363
Verified hb.db.6363 signature

Verified hb.db signature
Checking db integrity
Removing hb.db.N files
Archives not recovered with -n option

Backup files recovered to: /root/bs
Verify the backup with the selftest command:
  $ hb selftest -c bs
If inex.conf was customized, restore it with the hb get command.

real	     1m57.403s
user	     0m53.856s
sys	     0m17.106s

# du -hsc bs
1.1G	     bs
1.1G	     total

# hb config -c bs cache-size-limit 0
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Current config version: 3631

Set cache-size-limit to 0 (was -1) for future backups

Finding Files

Sometimes in a disaster situation you’ll need some files in a hurry, but maybe can’t remember exactly where they are. A wildcard hb ls command is used here to find the bsd VM in the recovered build server backup. It takes only half a second to find the files using a wildcard.

# time hb ls -c bs -l 'bsd764*'
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Most recent backup version: 3630
Showing most recent version, use -ad for all
3630 drwxr-xr-x    1  501   20        0 2021-07-10 17:04:18 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:14 2021-07-11 06:02:12 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/564d12c0-4305-5360-09a8-24a25cfcabe9.vmem.lck
3630 -rw-------    1  501   20 787939328 2021-12-17 07:01:53 2021-12-17 07:02:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk
3630 -rw-------    1  501   20 1139015680 2021-12-17 07:00:20 2021-12-17 07:02:39 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s002.vmdk
3630 -rw-------    1  501   20 1231224832 2021-12-17 07:01:22 2021-12-17 07:03:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s003.vmdk
3630 -rw-------    1  501   20 1811283968 2021-12-17 07:00:20 2021-12-17 07:03:49 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s004.vmdk
3471 -rw-------    1  501   20   196608 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s005.vmdk
3630 -rw-------    1  501   20     8684 2021-12-16 08:00:28 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.nvram
3471 -rw-------    1  501   20      570 2021-07-10 17:14:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk.lck
2760 drwxrwxrwx    1  501   20        0 2019-03-13 03:11:42 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmem.lck
2015 -rw-------    1  501   20        0 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmsd
2760 -rw-r--r--    1  501   20  1123098 2019-03-13 03:11:45 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmss
3471 -rwxr-xr-x    1  501   20     2274 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:08 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx.lck
2015 -rw-------    1  501   20      261 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmxf
3471 -rw-r--r--    1  501   20    71440 2021-07-08 02:42:22 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware-0.log
3471 -rw-r--r--    1  501   20    70782 2021-06-27 21:10:50 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware-1.log
3471 -rw-r--r--    1  501   20    90281 2021-06-19 18:57:18 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware-2.log
3630 -rw-r--r--    1  501   20    97241 2021-12-16 17:00:08 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware.log
3630 -rw-------    1  501   20 787939328 2021-12-17 07:01:53 2021-12-17 07:02:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk
3630 -rw-------    1  501   20 1139015680 2021-12-17 07:00:20 2021-12-17 07:02:39 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s002.vmdk
3630 -rw-------    1  501   20 1231224832 2021-12-17 07:01:22 2021-12-17 07:03:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s003.vmdk
3630 -rw-------    1  501   20 1811283968 2021-12-17 07:00:20 2021-12-17 07:03:49 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s004.vmdk
3471 -rw-------    1  501   20   196608 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s005.vmdk
3630 -rw-------    1  501   20     8684 2021-12-16 08:00:28 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.nvram
3471 -rw-------    1  501   20      570 2021-07-10 17:14:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk.lck
2760 drwxrwxrwx    1  501   20        0 2019-03-13 03:11:42 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmem.lck
2015 -rw-------    1  501   20        0 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmsd
2760 -rw-r--r--    1  501   20  1123098 2019-03-13 03:11:45 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmss
3471 -rwxr-xr-x    1  501   20     2274 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:08 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx.lck
2015 -rw-------    1  501   20      261 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmxf

real 0m0.528s
user 0m0.458s
sys  0m0.065s

Restoring Files

Now we can restore one of the VMs from the backup. Here is a restore of just 1 file:

# time hb get -c bs /Users/jim/Documents/Virtual\ Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Most recent backup version: 3630
Restoring most recent version
Using destinations in dest.conf
Warning: destination is disabled: usbcopy
Increased cache to 2.1 GB
Using local files

Planning cache...
  Items: 1 - 787 MB
  Scan time: 5.7s, 5s
  Plan time: 0.4s, 0s
  Download size: 3.8 GB
  Peak cache size: 2.1 GB
  Disk free space: 13 GB, 62% free
  Mem:  84 MB

Restoring bsd764-s001.vmdk to /root
/root/bsd764-s001.vmdk
Wait for arc.2.6 - 714 MB 29s
Wait for arc.0.5 - 630 MB 4s
Wait for arc.0.11 - 359 MB 8s
Wait for arc.334.0 - 1.0 GB 33s
Unable to set extended attributes: Operation not supported: /root/bsd764-s001.vmdk
Restored /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk to /root/bsd764-s001.vmdk
1 errors

real	2m14.878s
user	0m41.384s
sys	0m13.521s

# ls -l /root/bsd764-s001.vmdk
-rw------- 1 501 games 787939328 Dec 17 07:01 /root/bsd764-s001.vmdk

Notes about the restore:

  • More data was downloaded than is shown with "Wait" messages. Only files that take more than a few seconds to download show a "Wait" message.

  • Why did arc.2.6, a 714MB file, take 29 seconds to download, but arc.0.5, a 630MB file, took only 4 seconds? HashBackup uses "selective downloading" to minimize its downloads. This means it only downloads the parts of files that it needs. In this case, it likely needed more data from the file that took longer to download.

  • The extended attribute error occurs because this Linux VM filesystem does not have extended attributes enabled but the original server did. But the data recovery still works.

  • The ls command after the restore shows that the restored file is the correct length, matching the previous hb ls output.

  • The file is restored with the original numeric userid and groupid, but since groups are different on this VM, it shows as games.