Recover

Recovers the backup directory from a remote destination in a disaster situation. This command is used when the disk containing your local backup directory fails or the whole system fails or is unavailable for some reason such as a fire, theft, or lighting strike.

The recover command does not restore individual files like /etc/hosts. To do that, use the Get command.

Options

$ hb recover -c backupdir [destname] [-a] [-n] [-o]
                                     [--force] [--check]

The recover command downloads backup files from the remote destination and copies them to the local backup directory.

To do a full recover, create an empty backup directory, copy the original backup’s key.conf and dest.conf files there, then run the recover command. Do not run the init command before recover as this creates a new key.conf file.

The --force option starts recovery without a confirmation prompt.

The -n option skips downloading of archive files. Normally archive files are recovered if the cache-size-limit config option is -1 (the default), but this may require dowloading a lot of data. In a disaster recovery situation, only some critical user files may need to be restored immediately. With -n, the local backup directory can be recovered quickly, the hb get command can be used to restore the critical user files, downloading only the required archive data, then the recover command without -n can be used later to recover all archive files. Another way to download arc files is to set the cache-size-limit config option to -1 and run selftest with either the -v3 or -v4 options. This can be combined with --inc to download arc files over time.

The -a option (archives) forces downloading of archive files even if the cache-size-limit config option is >= 0. This is useful if you previously didn’t want all archive files stored locally, but now are switching to a local NAS and do want all archive files stored locally. HB will set the cache-size-limit config option to -1 after the recover. Another easy way to fetch all remote archives is to set the cache-size-limit config option to -1 and then run selftest -v3.

The -o option (overwrite) downloads files even if they already exist in the local backup directory. This is useful if the local copies are suspect for some reason.

The --check option recovers the main database in a slower but more careful way. This is only used if a normal recover cannot recover the latest database for some reason. The resulting database may not be the latest version.

To recover your backup, you must have a copy of the dest.conf and key.conf files from your backup directory. These files are never copied to remote destinations. The dest.conf file is required to connect to a destination containing a copy of your backup, and the key.conf file is required to decode your encrypted backup data.

You MUST copy your dest.conf and key.conf files to one or more USB flash drives, or print these files, and store the copies in several locations separate from the backup data. If your backup is passphrase protected, you must also know the passphrase.

Test Recover

Some customers do test recovers to verify their backup, which is a great idea. To prevent accidentally overwriting remote data following a test recover, a file testonly.conf is created in the backup directory. While this file is present, downloads work but destinations cannot be modified (no upload or remove). This is necessary because after a test recover, both the real and test backup directories match the unique ID HashBackup uses to prevent accidental overwrites of remote data. An operation in the test directory that modifies the backup could unintentionally modify the remote data, potentially causing problems for the production backup. The testonly.conf file prevents that and allows tests to be run without allowing remote data to be changed.

For real recovers of the local backup directory, delete the testonly.conf file.

Notes

  • The recover command fetches the backup database and archive files; it does not fetch individual files or directories that you may have backed up. For example, you can’t restore the /etc directory using the recover command; use the get command instead.

  • To recover all files from a remote destination, create a new directory with only the dest.conf and key.conf files, then run recover -c with this new directory.

  • If the local archive cache size is limited with the cache-size-limit config option, recover will not download archive files unless -a is used.

Recover Session

In this example, the HashBackup development server’s backup is recovered to a cloud-based Linux VM. The server is an 80GB Mac Mini with daily backups since 2010, so there are over 3600 versions.

First the key.conf and dest.conf files are copied to an empty directory named bs on the VM, then the recover command is used to recover the backup database from Backblaze B2. The -n option prevents downloading arc files because they would require 50GB of space and this small single-CPU VM only has about 10GB of space available.

At each step of the database recovery, files are verified with cryptographic hashes, and after the database is reconstructed, a final checksum is verified. An integrity check verifies that the rebuilt database passes consistency checks.

The download and reconstruction takes about 30 seconds and the database consistency check takes about 90 seconds. The cache size limit is then set to 0 because this VM has very limited disk space.

It’s normal to have skips in the hb.db.N file sequences! 
# time hb recover -c bs -n
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Using destinations in dest.conf

Recovering backup files from destination: b2
Files will be copied to: /root/bs

Proceed with recovery? y

Removed /root/bs/dest.db
Getting dest.db from b2
Getting hb.db from b2
Queueing hb.db files

Waiting for /root/bs/hb.db.6386
Loading hb.db.6386
Verified hb.db.6386 signature

Waiting for /root/bs/hb.db.6385
Loading hb.db.6385
Verified hb.db.6385 signature

Loading hb.db.6384
Verified hb.db.6384 signature

Loading hb.db.6383
Verified hb.db.6383 signature

Loading hb.db.6382
Verified hb.db.6382 signature

Loading hb.db.6381
Verified hb.db.6381 signature

Loading hb.db.6380
Verified hb.db.6380 signature

Loading hb.db.6379
Verified hb.db.6379 signature

Loading hb.db.6378
Verified hb.db.6378 signature

Loading hb.db.6376
Verified hb.db.6376 signature

Loading hb.db.6375
Verified hb.db.6375 signature

Loading hb.db.6364
Verified hb.db.6364 signature

Loading hb.db.6363
Verified hb.db.6363 signature

Verified hb.db signature
Checking db integrity
Removing hb.db.N files
Archives not recovered with -n option

Backup files recovered to: /root/bs
Verify the backup with the selftest command:
  $ hb selftest -c bs
If inex.conf was customized, restore it with the hb get command.

real	     1m57.403s
user	     0m53.856s
sys	     0m17.106s

# du -hsc bs
1.1G	     bs
1.1G	     total

# hb config -c bs cache-size-limit 0
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Current config version: 3631

Set cache-size-limit to 0 (was -1) for future backups

Finding Files

Sometimes in a disaster situation you’ll need some files in a hurry, but maybe can’t remember exactly where they are. A wildcard hb ls command is used here to find the bsd VM in the recovered build server backup. It takes only half a second to find the files using a wildcard.

# time hb ls -c bs -l 'bsd764*'
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Most recent backup version: 3630
Showing most recent version, use -ad for all
3630 drwxr-xr-x    1  501   20        0 2021-07-10 17:04:18 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:14 2021-07-11 06:02:12 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/564d12c0-4305-5360-09a8-24a25cfcabe9.vmem.lck
3630 -rw-------    1  501   20 787939328 2021-12-17 07:01:53 2021-12-17 07:02:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk
3630 -rw-------    1  501   20 1139015680 2021-12-17 07:00:20 2021-12-17 07:02:39 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s002.vmdk
3630 -rw-------    1  501   20 1231224832 2021-12-17 07:01:22 2021-12-17 07:03:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s003.vmdk
3630 -rw-------    1  501   20 1811283968 2021-12-17 07:00:20 2021-12-17 07:03:49 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s004.vmdk
3471 -rw-------    1  501   20   196608 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s005.vmdk
3630 -rw-------    1  501   20     8684 2021-12-16 08:00:28 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.nvram
3471 -rw-------    1  501   20      570 2021-07-10 17:14:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk.lck
2760 drwxrwxrwx    1  501   20        0 2019-03-13 03:11:42 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmem.lck
2015 -rw-------    1  501   20        0 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmsd
2760 -rw-r--r--    1  501   20  1123098 2019-03-13 03:11:45 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmss
3471 -rwxr-xr-x    1  501   20     2274 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:08 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx.lck
2015 -rw-------    1  501   20      261 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmxf
3471 -rw-r--r--    1  501   20    71440 2021-07-08 02:42:22 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware-0.log
3471 -rw-r--r--    1  501   20    70782 2021-06-27 21:10:50 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware-1.log
3471 -rw-r--r--    1  501   20    90281 2021-06-19 18:57:18 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware-2.log
3630 -rw-r--r--    1  501   20    97241 2021-12-16 17:00:08 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/vmware.log
3630 -rw-------    1  501   20 787939328 2021-12-17 07:01:53 2021-12-17 07:02:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk
3630 -rw-------    1  501   20 1139015680 2021-12-17 07:00:20 2021-12-17 07:02:39 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s002.vmdk
3630 -rw-------    1  501   20 1231224832 2021-12-17 07:01:22 2021-12-17 07:03:16 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s003.vmdk
3630 -rw-------    1  501   20 1811283968 2021-12-17 07:00:20 2021-12-17 07:03:49 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s004.vmdk
3471 -rw-------    1  501   20   196608 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s005.vmdk
3630 -rw-------    1  501   20     8684 2021-12-16 08:00:28 2021-12-17 07:04:41 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.nvram
3471 -rw-------    1  501   20      570 2021-07-10 17:14:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmdk.lck
2760 drwxrwxrwx    1  501   20        0 2019-03-13 03:11:42 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmem.lck
2015 -rw-------    1  501   20        0 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmsd
2760 -rw-r--r--    1  501   20  1123098 2019-03-13 03:11:45 2019-03-13 06:07:04 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmss
3471 -rwxr-xr-x    1  501   20     2274 2021-07-10 17:04:14 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx
3471 drwxrwxrwx    1  501   20        0 2021-07-10 17:04:08 2021-07-11 06:04:26 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmx.lck
2015 -rw-------    1  501   20      261 2010-10-20 18:53:45 2017-01-08 07:22:35 /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764.vmxf

real 0m0.528s
user 0m0.458s
sys  0m0.065s

Restoring Files

Now we can restore one of the VMs from the backup. Here is a restore of just 1 file:

# time hb get -c bs /Users/jim/Documents/Virtual\ Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Most recent backup version: 3630
Restoring most recent version
Using destinations in dest.conf
Warning: destination is disabled: usbcopy
Increased cache to 2.1 GB
Using local files

Planning cache...
  Items: 1 - 787 MB
  Scan time: 5.7s, 5s
  Plan time: 0.4s, 0s
  Download size: 3.8 GB
  Peak cache size: 2.1 GB
  Disk free space: 13 GB, 62% free
  Mem:  84 MB

Restoring bsd764-s001.vmdk to /root
/root/bsd764-s001.vmdk
Wait for arc.2.6 - 714 MB 29s
Wait for arc.0.5 - 630 MB 4s
Wait for arc.0.11 - 359 MB 8s
Wait for arc.334.0 - 1.0 GB 33s
Unable to set extended attributes: Operation not supported: /root/bsd764-s001.vmdk
Restored /Users/jim/Documents/Virtual Machines.localized/bsd764.vmwarevm/bsd764-s001.vmdk to /root/bsd764-s001.vmdk
1 errors

real	2m14.878s
user	0m41.384s
sys	0m13.521s

# ls -l /root/bsd764-s001.vmdk
-rw------- 1 501 games 787939328 Dec 17 07:01 /root/bsd764-s001.vmdk

Notes about the restore:

  • More data was downloaded than is shown with "Wait" messages. Only files that take more than a few seconds to download show a "Wait" message.

  • Why did arc.2.6, a 714MB file, take 29 seconds to download, but arc.0.5, a 630MB file, took only 4 seconds? HashBackup uses "selective downloading" to minimize its downloads. This means it only downloads the parts of files that it needs. In this case, it likely needed more data from the file that took longer to download.

  • The arc.0.5 wait message shows that, because of deduplication, reconstructing this file needed data from the initial backup of the server. This is the 5th arc file in the version 0 (initial) backup of the server from Oct 10, 2010, as shown by this hb versions command:

    [root@hbtest ~]# hb versions -c bs -a | head
HashBackup #2628 Copyright 2009-2021 HashBackup, LLC
Backup directory: /root/bs
Showing all versions
     0 0(root) 2010-10-10 19:38:35 - 2010-10-10 20:30:21     51m 45s 8.4 GB
     1 0(root) 2010-10-10 21:04:01 - 2010-10-10 21:13:03      9m  2s
     2 0(root) 2010-11-27 04:52:30 - 2010-11-27 05:47:40     55m  9s 3.6 GB
     3 0(root) 2010-11-27 15:05:25 - 2010-11-27 15:05:25          0s
     4 0(root) 2010-12-05 16:46:59 - 2010-12-05 16:55:48      8m 49s  43 MB
     5 0(root) 2010-12-05 17:35:41 - 2010-12-05 17:35:41          0s
     6 0(root) 2010-12-09 23:28:40 - 2010-12-09 23:33:51      5m 10s  19 MB

    The size of the version 0 backup is now only 8.4GB because over the years, data has been superceeded by later backups. Over time, the earliest backups become smaller. The initial backup would have been around 80GB in 2010.

  • The extended attribute error occurs because this Linux VM filesystem does not have extended attributes enabled but the original server did. But the data recovery still works.

  • The ls command after the restore shows that the restored file is the correct length, matching the previous hb ls output.

  • The file is restored with the original numeric userid and groupid, but since groups are different on this VM, it shows as games.