Config

An admin passphrase restricts certain actions unless the passphrase is entered. This option is used to:

Unlike all other config options, the current admin passphrase cannot be displayed: it is stored as a hash code so displaying it is impossible. Also unlike other options, a new admin passphrase cannot be listed on the command line. Instead, wait for the prompt to enter the new passphrase, then enter it twice:

After an admin passphrase is set, the environment variable HB_ADMIN_PASSPHRASE can be used to pass the admin passphrase to HashBackup for automated operations. For example, to change the passphrase on a backup that currently has one, and enter the new passphrase on the keyboard:

To change the passphrase using only environment variables, without keyboard input:

Specifies the maximum size of archive files. The default is 100mb. The minimum size is 10K, used for internal testing. At least 2 x arc-size-limit bytes of free disk space are required in the local backup directory.

When arc-size-limit is changed, the new limit is used on the next backup but HashBackup does not resize existing arc files. Future retain and rm operations will use the new limit when combining older, small arc files into larger arc files.

HashBackup may go slightly over the limit specified because when the limit is reached, especially on a multi-core system and/or with large block sizes. If a hard limit is needed for some reason, specify 5-10MB less or 3-4 block sizes less, whichever is greater, and run some experiments to make sure your hard limit isn’t exceeded.

For multi-TB backups with mostly large files that do not change often, a large archive size may be more efficient. A practical limit is 4GB because many online storage services do not allow file sizes over 5GB without special handling. Also, a limit higher than 4GB doubles RAM usage when creating restore plans (when cache-size-limit is set) because 64-bit file offsets are required. Be aware that some online services have a maximum upload file size.

For VM image backups with a lot of dedup, a high rate of change, and/or small block sizes, a smaller arc size may be more efficient because it can be packed more often. A smaller arc size may also reduce disk space cache requirements during a restore, if local disk space is tight. This is only a concern when cache-size-limit is >= 0 and the destination does not support selective downloads (retrieving parts of arc files), so entire arc files are downloaded.

As user files are removed from older backups with the rm or retain commands, "holes" are created in arc files. If enough data is removed, more than pack-percent-free, archives are automatically packed to remove empty space (see pack-* options). If all data from an archive is removed, the archive is deleted without a pack operation. This is why smaller archive files can be more space efficient than larger files: when data is removed, smaller arc files increase the probability that entire arc files can be removed without a pack.

A disadvantage of very small arc files, besides creating many arc files for large backups, is that very small arc files may defeat download optimizations. For example, with 1MB arc files, the maximum request size to remote storage services is limited to 1MB. A backup of a 1GB file will create 1000 arc files and downloading this will require 1000 x 1MB remote requests. If the storage service has high latency (delay for each request) it can cause performance problems. Using arc-size-limit below 4MB is not recommended, and in general, arc-size-limit should be somewhat proportional to your backup size.

When True, backup will save Linux file attributes, also called flags in BSD Unix. On Linux, file attributes are set with the chattr command and displayed with lsattr. They are little used and poorly implemented on Linux, requiring an open file descriptor and an ioctl call. This can cause permission problems, especially in shared hosting environments, so the default is False.

File attributes are not the same as extended attributes, also called xattrs. Extended attributes are always backed up if present. Extended attributes are handled by the attr, setfattr, and setfattr commands on Linux, as well as the Linux ACL commands.

The variable block size, default is auto. Can be set to 16K, 32K, 64K, 128K, 256K, 512K or 1M. The effective average block size is usually 1.5 times larger. Smaller block sizes dedup better than larger block sizes but also cause more block metadata to be stored in the database and can make restores slower, so there is a trade-off. Previously the variable block size was 32K, but auto scales better because it uses a larger block size for larger files.

If this option is changed on an existing backup at rev r, files changed after rev r will be saved with the new block size and cannot dedup against files saved before rev r with a different block size. A block size warning is displayed during backup when this happens. Once files are saved with the new block size, dedup will occur on subsequent backups when files change. This option cannot be used to set a large fixed block size. To do that, use -B4M for example on the backup command line.

Sets the backup block size for specific filename extensions (suffixes), overriding backup’s -B command line option and other block size settings. For example, with this option set to '-B4M mov,avi -V128K .sql -B16K ibd -B23K xyz', backup will use:

Commas and periods are optional. The value must be quoted since it contains spaces. Unlike most config options, this option uses 1024 as a multiplier so 8K means 8192 bytes.

Specifies the maximum amount of space to use in the local backup directory to cache archive files. By default, this is -1, meaning keep local copies of all archive files. Setting cache-size-limit >= 0 will restrict the number of archives kept locally. Remote destinations must be setup in dest.conf or this config option is ignored.

The value can be set two ways:

A reasonable value for this limit is the average size of your incremental backups. A reasonable minimum is the highest workers setting for any destination plus 1. The default number of workers is 4, so the minimum would be 5, but a more reasonable limit would be 1GB for 100MB arc files. If you plan to use the mount command extensively, a larger cache will prevent multiple downloads of the same data.

The cache size limit is more of a desired goal than a hard limit. The limit may often be exceeded by the size of 1 archive file, and may need to be greatly exceeded for restores to prevent downloading the same archive file more than once. The mount command respects the limit but might have to download the same data more than once.

If you can afford the disk space, keeping a copy of archives locally has many advantages:

Thanks to compression and dedup, total backup space is usually about 50%-100% the size of the original data, even when multiple versions are maintained.

Examples of cache size limits:

If cache-size-limit was -1 (all arc files kept locally) and then is changed to 10GB for example, the next backup will trim the cache down to 10GB after the backup completes. You can force this trim to happen immediately by backing up a small file like /dev/null after changing cache-size-limit.

If there are multiple destinations configured in dest.conf, an arc file must be stored successfully on all destinations before the local copy can be deleted. For configurations where 2 USB drives are setup and 1 is always offsite, there must be enough local cache space to hold all backups that occur before swapping drives.

When True, copy the hb program to remote destinations. The default is False. The hb program is always copied to the local backup directory as hb#b, where b is the build number. If using HB only occasionally, more as an archive, it is important to set this true so that you will have a copy of HB stored with your archive.

Controls when a database integrity check occurs. The default is selftest. This is fine when backups are run and stored on enterprise-class hardware: non-removable drives, ECC RAM, and hardwired networks.

For extra safety, set this option to upload on consumer-grade hardware using removable drives, non-ECC RAM, and wireless networks where there is a higher possibility of database damage. This will verify database integrity before each upload to destinations, preventing a damaged local database from overwriting a good version stored remotely.

Keep incremental database backup files (hb.db.N) for this many days before the latest backup. The default is 3. Combined with recover --check, this allows recovering earlier versions of the main backup database. The minimum value is 1 so there is always at least one older version of the database stored on remotes.

A read-only config option that is unique for each backup database

A read-only config option with the database revision level

The amount of RAM to be used for the dedup table, similar to the -D backup option. The -D option overrides this config option. See Dedup Info for more detailed information about sizing the dedup table. The default is 0, meaning dedup is disabled. A reasonable value is 1gb. The dedup table does not immediately use all memory specified; it starts small and doubles in size when necessary, up to this limit. The backup command shows how full the current dedup table is, and how full it is compared to the maximum table size. When the dedup table is full and cannot be expanded, backups will continue to work correctly. Dedup is still very effective even with a full table. Unlike most config options, this option uses 1024 as a multiplier so 100M means 104,857,600 bytes.

List of commands that should be disabled, separated by commas; all other commands are enabled. If the admin-passphrase option is set, these commands ask for the passphrase and run only if it is correctly entered. If there is no admin-passphrase set, these commands refuse to run. The upgrade and init commands cannot be disabled because when these commands execute there is no database to check whether they are enabled or disabled. To cancel this option, use the value '' (two single quotes).

SECURITY NOTE: Disabling commands without admin-passphrase set is a poor configuration since commands can easily be re-enabled.

List of commands that should be enabled, separated by commas; all other commands are disabled. An example would be to enable only the backup command. Then it would not be possible to list, restore, or remove files without the admin passphrase. To cancel this option, use the value '' (two single quotes).

SECURITY NOTE: Enabling some commands without setting an admin-passphrase is a poor configuration since more commands can easily be enabled.

List of file extensions, with or without dots, separated by spaces or commas. Files with these extensions will not be backed up.

List of filenames separated by spaces or commas. Directories containing any of these files will not have their contents backed up, other than the directory itself and the tag file(s). Typical values are .nobackup and CACHEDIR.TAG Each entry in this list causes an extra stat system call for every directory in the backup, so keep the size of this list to a minimum.

List of file extensions of files that should not be compressed. Most common extensions are already programmed into the backup program, but less common extensions can be added. Extensions may or may not have dots, and are separated by spaces or commas.

List of file extensions of files with data that does not dedup well, for example, photos. If exact copies of files are backed up, they will be still be deduped if dedup is enabled.

The minimum age of an archive before it is packed. Some storage services charge penalties for early removal of files, for example, Amazon S3 Infrequent Access and Google Nearline. This setting prevents packing archives until they have aged, avoiding the delete penalty charge, but you are still paying more for storage of data that perhaps could have been deleted earlier. All other conditions must be satisfied before an archive is packed. The default for pack-age-days is 30. To disable this feature, set it to zero. Higher numbers will reduce the number of pack operations but may increase the backup storage requirements. If you do not pay delete penalties and have low or zero download fees, a free download allowance, or have cache-size-limit set to -1 (local copy of arc files), setting this to zero will cause more frequent packing to lower storage costs, improve restore times, and lower restore cache requirements.

This option also controls the frequency of packing operations. However, if a packing operation cannot complete because of a download or time limit, it will resume the next time rm or retain are run. If set to zero, rm and retain will try to pack archives on every run.

The minimum free space in an archive before it is packed. The default is 1MB. This setting prevents packing very small archives when data is removed, ie, once an archive gets down to 1MB (the default setting), it is not packed until it is empty, then is deleted. All other conditions such as pack-percent-free must also be satisifed before an archive is packed. To disable this feature, set it to 4096 (the minimum value). Higher numbers will increase the backup storage requirements and reduce the number of pack operations.

The minimum size of an arc file before it is merged into a larger arc file. The default is 1MB, meaning that when possible, consecutive arc files smaller than 1MB will be merged together. For combining to occur, packing must occur (see the other pack-* config settings). For technical reasons, only consecutive arc files can be merged into larger arc files.

The maximum amount of data that can be downloaded during a single run of rm or retain for packing. The default is 950MB. To prevent any downloading of remote arc files for packing, set this limit to 0 (but see below). For "unlimited" downloading, set the limit to a high value like 1TB. This limit only applies when cache-size-limit is >= 0, meaning that not all arc files are stored locally; local arc files are packed without a download.

It is recommended that pack-download-limit not be set to zero. When files are removed from the backup with rm and retain, "holes" are created in arc files. Over time, this causes older backup data to be stored inefficiently, making restores slower. Packing reorganizes arc files into more efficient storage. To reduce the amount of packing, raise pack-percent-free to a high number like 95, meaning that 95% of an arc file must be free before it is packed. This will prevent nearly all downloading except tiny files smaller than pack-combine-min and very inefficient arc files with mostly empty space.

If pack-download-limit is set to a value smaller than the largest archive in the backup, a warning may be displayed that this archive cannot be packed. If more space is freed within the archive, it will eventually be packed if selective download is available on a destination.

When archive files have this percentage or more as free space, the archive is packed to recover space. This is used by rm and retain after they have deleted backup data. The default is 50. If cache-size-limit is >= 0 and you pay for downloaded data, consider raising this percentage to avoid frequent packing or set pack-download-limit. Most storage services have very high download charges compared to their storage charge. If cache-size-limit is -1 (the default), packing more often is fine because no download is required. Higher numbers increase the backup storage requirements but reduce the number of pack operations and reduce the number of downloads when cache-size-limit is >= 0.

When set to normal (the default), HB sends new data before deleting old data, for example, during packing. This preserves the integrity of remote backup areas. But if a remote disk becomes full it may be necessary to delete the old data before sending new data. To enable this behavior, set to unsafe.

Another way to handle a full remote backup area is to delete recent backup versions with rm -r. This quickly removes entire versions and their archives and makes room for a retain operation to remove older data. The next backup will re-save the recent files that were deleted if they are still present in the filesystem.

Adds an extra tme period to each -s interval as a retention safety cushion, so -s7d4w becomes -s8d5w The default is True.

The number of days to keep shard output in the sout subdirectory. The default is 30 days. Set to 0 to keep all shard output.

If set to true before the first backup, no arc files are created by the backup command. This allows modeling backup options such as -B (blocks size) and -D (dedup table size), even for very large backups, without using a lot of disk space. Simulated backups also run faster because there is less I/O. Incremental backups, rm, and retain all work correctly, and the stats command can be used to view statistics showing the backup space that would be used by a real backup. The pack-…​ keywords control whether the simulated arc files are packed when files are removed with rm or retain.

Differences for a simulated backup: