Releases

bump expiration date

rm, retain, selftest: fix nested transaction with new destination

versions: add -t option

rm, retain, selftest: if the first command after creating a new destination was rm, retain, or selftest, an error "OperationalError: cannot start a transaction within a transaction" could occur.

versions: a new -t option selects versions by tag. Used by itself, -t shows all versions with a tag. An optional argument shows only versions with a tag containing the argument. Thanks Boris!

rm: remove file path with no log record

selftest: correct "no parent" error if possible

backup: avoid path with no log errors w/bad timestamps

rm: in #3280, rm would remove a pathname with no log record iff it was a directory path and had children. Now it will also remove the path if it is a file or directory with no children. Thanks Ware!

selftest: in #3211 (July 2024), a new test and warning was added to ensure all files' parent directories were saved in each version, to avoid problems with ls. These warnings usually occurred when a backup was interrupted, and that problem was also corrected in #3211. Now, selftest --fix can often correct these warnings by copying a previous version’s backup of a missing directory. Thanks Ware!

backup: backing up files with goofy timestamps caused "path with no log record" errors in selftest. Thanks Ware!

rm, selftest: removing paths with no log record

rm, selftest: if a path exists but has no log record (file data), selftest would give a warning about an unused path. With --fix it would delete the pathname, but only if there were no files underneath. rm would not delete the pathname, but would try to run selftest --fix. Now, the selftest message has been changed, it is an error instead of a warning, and rm will remove the extra pathname, also removing any files underneath. Thanks Christian!

release fix: rm not removing directory files (Thanks Christian!)

release fix: mount bug in #3275 (Thanks Bugz & Christian!)

database upgrade to dbrev 37 + 38 + 39 + 40

db: always use backup directory for temp files

backup: upgrade encryption to AES-256

db: db-history-days keeps database history

backup: arc-size-limit controls size of hb.db.N files

backup: allow saving more than 2 billion files

backup: save parent directories in order

trim, versions: blocks and space aren’t tied to versions

trim: showed one file twice near the end

backup: pack small recent arc files < pack-combine-min

dest: fix rare setid traceback

clear: add -r option to remove recent versions

retain: 2.5x faster checking directories

retain, compare: add -v0 option to suppress filenames

clear: --reset caused tracebacks

simulated-backup improvements

audit: replaced by log files

readkey: random "wrong key" errors

shards: fix commands that ask questions

rename: works with shards + bug fix

init: add and remove shards

dest.conf: remove support for maxsize

this release does an automatic database upgrade to remove the limits on maximum files backed up (was 2B) and maximum blocks per file (was 4B). Databases on dbrev 32 or later (Feb 2019) can be upgraded.

backup: arc files and database incrementals (hb.db.N files) are now encrypted with AES-256. Existing arc files remain encrypted with AES-128. Key files are unchanged since they’ve always been 256-bit.

db: sometimes the database was using OS temp space (/var/tmp, /usr/tmp, or /tmp) for operations like rebuilding an index or sorting. If the backup database was a bit large and temp space was limited, this could cause database failures like "OperationalError: database or disk is full", often during a database upgrade, without any indication that limited temp space was the actual problem. Now, database temp files are created in the same directory as the main database.

db: the db-history-days config option specifies how many days of database history are kept on destinations with hb.db.N files. These files are used by recover to reconstruct the local backup directory, including the hb.db database, if lost because of theft, fire, drive failure, etc. It was always possible to recover the latest version of the database with the recover command, but it was somewhat hit-and-miss whether older versions could be recovered with the --check option, and how recent they would be. Now, it is always possible to recover at least db-history-days recent versions of the backup database. The default has been changed to 30 versions. See the config and recover pages for more details about db-history-days.

backup: the arc-size-limit config option also controls the size of hb.db.N files, used to backup the main database. Previously these files could be up to 4GB by default. Thanks Mehmet!

backup: if 2,147,483,647 (size of a 32-bit integer) files were saved, some commands raised an error "OverflowError: signed integer is greater than maximum". From internal testing.

backup: unmodified parent directories with modified subfiles were saved on the way out of the tree. But if the backup is interrupted, the saved, modified files can’t be seen by ls because the parent directories haven’t been saved. Now parent directories are always saved before any descendants, preventing ls problems and avoiding the new selftest error for files without parent directories. From internal testing.

trim, versions: the sections of these reports that tied backup space to versions have been removed. Over time, blocks (and space) do not have a strong association to the original version, and it is confusing to show for example that there are no files in version x (they’ve been removed by retain), but there is still space used by version x because other files are referencing those blocks.

trim: when generating the list of files to trim, a path near the end could be shown twice.

backup: if the last arc file from the previous backup is smaller than pack-combine-min (default 1MB), or there are a string of recent small arc files, maybe because of small hourly backups, backup will combine them into a larger arc file.

dest: a setid command could cause a rare traceback, "AttributeError: 'NoneType' object has no attribute 'endswith'" This occurred if the previous command had files to remove, was interrupted, and the next command (setid) tried to finish the removes. Thanks Bugz!

clear: previously, the rm command could remove one version from the backup with -r, but that feature was removed because it was hard to understand, eg, files were sometimes moved into the next backup. But it is sometimes useful to remove the last backup as if it never happened. The clear command now takes a -r option and will remove any backups on or after this version, as if they never occurred.

retain: after improving a database query, retain is 2.5x faster when there are a large number of directories affected. On a backup of 10M files where 25% were removed with -m1, retain runtime is reduced from 22 minutes to 8.5 minutes with an SSD drive.

retain: the -v0 option suppresses showing filenames when retaining a specific list of files or directories. The -v0 option to the compare command now also suppresses filenames and only shows the final summary.

clear: commands after clear --reset would fail with a traceback "error: cannot convert argument to integer" and the only way to fix it was to run hb init.

more things work with simulated-backup set, for example:

audit: the audit command still works for a few months, showing a message that it has been replaced by files in the logs subdirectory, but it should be removed from any scripts since the logs directory has better auditing information. The audit command will be removed in a future release.

readkey: the crypto library HashBackup uses for public key encryption (readkey only!) had a bug when data began with a zero byte. When the readkey command was used without "on" or "off", it would randomly, about 0.4% of the time, report that the key passphrase or readkey.conf was wrong, when in fact, it was fine.

shards: commands that ask questions, like hb clear without --force asking "Are you sure?", were not asking the question with shards.

rename: now works with shards, and a bug was fixed: renaming /tmp to /tmp2 would fail with an error about renaming a file to a subdirectory.

init: the number of shards can now be increased or decreased. If shards are decreased from 4 to 3 for example, the backup history saved on shard 4 will be lost. If shards are increased from 3 to 4, shards 1-3 will keep their backup history and the next backup will distribute new files among all 4 shards. Shard 4 will save more files in the next backup because ~8% of the live files in shards 1-3 will be re-saved to shard 4.

dest.conf: in #3015, June 2022, support for the maxsize keyword in dest.conf was removed. If files were already uploaded with maxsize, HB advised to update the backup with "dest noparts". With this release, support for files uploaded with maxsize has been removed.

database upgrade to dbrev 36

shards: command log files and timestamps

backup: extended attributes 3x smaller and 3x faster

stats: new statistics

get: use default responses with shards

get: not always fatal if shard doesn’t contain path

trim: handle -i with shards

rm: better exit code handling

export: support shards

config: shard-output-days defaults to 7 vs 30

rekey: handle interrupted rekey with shards

backup: handle deleted directories on Mac

compare: support shards

compare: # of files sometimes too big

disk space errors on 16TiB filesystems

selftest: extra blockid in dedup table

versions: corrected file statistics

selftest, backup: arc blockids overlap error

selftest: set --iso if no local arcs and no destinations

ls: add -i for case insensitive searches

ls: drop glob character class (bracket) pattern

ls: add -h option for human-readable file sizes

ls: add -x for regular expression searches

compare: compare acl and extended attributes

rm: remove owned versions, even if not all

selftest: "no parent" warnings

hb: reduce memory usage with large backups

backup: don’t resolve symlinks on command line

ls, get: symlinks might be directories

this release does an automatic database upgrade to ensure the hb.db and hash.db files are "mated". If they are not, hash.db is rebuilt. At one site, the hash.db and hb.db files were copied around and became mismatched, causing dedup to fail silently though the backup was fine otherwise. This also gave a non-critical selftest error. Thanks Mehmet!

shards: HB now creates regular timestamped log files in each shard subdirectory as with non-sharded backups. Before this change, the log file in backupdir/logs was timestamped, but the timestamp was the time the logs were gathered, not the time files were backed up (for example). The sout logs are mostly redundant now, but will be kept for a while to make sure nothing falls through the cracks.

backup: HB previously did not store extended attributes very efficiently, because historically, they were small and unimportant. But recently Apple started adding large (up to 6MB) extended attributes to container directories, and a backup of ~/Library/Containers creates a 469 MB hb.db, with 99% or 466 MB of that being used to store extended attributes. With this db upgrade, extended attributes are stored more efficiently, about 3x smaller, and it’s 3x faster to backup the Container directory. Thanks Kim!

stats: some new statistics are shown:

get: questions are sometimes asked, such as "File exists, okay to overwrite?", and these usually have default responses. With shards, the default responses were not being accepted and a shard would sometimes go into an infinite loop asking the question. This could be seen in the sout files, though on the terminal screen, it appeared that the restore was hung.

get: shards could exit with a fatal error if they didn’t contain a requested pathname, but sometimes this is expected and not an error. When restoring a single file, the exit code would be 1 even though the file was correctly restored. When multiple file pathnames in different shards were on the command line, not all files were restored because some shards exited prematurely with a "Pathname is not in backup" fatal error.

trim: trim works with shards to generate reports if the -i option is not used. The -i option (interactive trim) can be used with individual shards by adding /sN to the -c option. This will interactively trim shard N. The -i option cannot be used to trim multiple shards, and now instead of raising an error 'Inappropriate ioctl for device', the error advises to trim shards individually.

rm: when removing multiple paths, the exit status was only for the last path removed, so if the 1st remove failed and the 2nd worked, the exit status was zero even though there was a failure. Also, it is not necessarily an error if a pathname is not found in a particular shard, so it should not always affect the exit status.

export: shards are supported, either exporting the entire backup, creating an export file for each shard, or exporting a single shard by adding /sN to export’s -c option, where N is the shard number.

config: shard-output-days defaults to 7 (was 30) and instead of using the time the job was started to calculate its age, the last time the job wrote output is used so a long-running job does not have its output removed prematurely.

rekey: an interrupted rekey is automatically rolled back to the previous state on the next command. With shards, a successful rekey after a rollback had an incorrect key file.

backup: Mac filesystems, both HFS and APFS, can fail if the parent directory is deleted by another process during directory traversal. Backup would report an error like: Unable to backup [#3083 backup 2588 2593]: No such file or directory: <pathname>. This is true and not so bad, but it also broke traversal so badly that backup marked all files in all parent path directories as deleted. A workaround now prevents this. Thanks Arthur, this was a tough one!

compare: now works with shards. Also, the total number of files compared statistic was sometimes too big, because each component of each pathname was counted instead of just the pathname.

on filesystems with 16TiB or a multiple of that, to 16TiB plus 2 times arc-size-limit of free space, HB stopped, warning that there was not enough free space. Thanks Boris!

selftest: many warning messages "extra blockid in dedup table" were incorrectly shown on large backups with more than 2 billion blocks. This was a selftest bug: both the dedup table and backup were actually fine.

versions: the "File Space" statistic could incorrectly be shown as a negative number if hard links were removed from the backup either with the rm or retain commands. The file counts were too high when directories were removed.

selftest, backup: huge backups with more than 4 billion blocks could cause a selftest error "arc blockids overlap", with one of the blockids shown as 4294967295. This error is legitimate: the mistake actually occurred during backups. From internal testing.

selftest: if no local arc files and no destinations, set --iso to test backup in isolation without throwing errors for every arc file and every block

get: the --cache option now works with shards instead of only 1 shard working and the rest getting an error that the backup directory is locked.

ls: a new -i option enables case-insensitive (ignore case) pathname searches. The default for Apple is case-insensitive, the default for Linux is case-sensitive, like their filesystems. Thanks Arthur!

ls: support has been dropped for character-class glob wildcard patterns (brackets). Most users don’t understand them, and it was difficult to search for filenames actually containing square brackets. For example, a pattern of abc[xyz-abc]def would not match a filename with this exact name but raised an error because z is greater than a. And a pattern like '[BASIC]', which should be an easy search, actually means "any filename containing the letters B, A, S, I, or C", which could be hundreds of thousands of files in a large backup. The * (multiple wild characters) and ? (single wild character) wildcards are still supported. Thanks Andy!

ls: add -h option (human readable) to show file sizes in KB, MB, GB

ls: a new -x option shows all paths in the backup, whether deleted or not. When followed by a regular expression, only matching pathnames are shown. -x is usually slower than glob (no -x) patterns, but can do more complex filtering with patterns that may cross pathname components.

$ hb ls -c backupdir -x 'flower.*tree'
/User/jim/blueflowers/and/browntrees

compare: compare acl and extended attributes, add -i options to ignore either of these when comparing. Also compares and notes symbolic link targets with "slink"; previously these showed up as ctime and mtime changes only. The output has changed slightly, using periods instead of askterisk for each field listed with -v2, to look a bit cleaner. Thanks Arthur!

rm: previously, when a path was removed, rm required the user to own all versions of the path, ie, they created all backups of that path. But if user A created version 1 and user B created version 2, neither user could remove the version of the path they created. Now, users can remove any path they create. From internal testing.

backup, selftest: ideally, when a file is saved, all of its parent directories are also saved. This is necessary for the ls command to "see" all files. Selftest now checks for this and issues a warning. There were two previous causes of this problem. The first is related to excluding directories but backing up some sub-contents on the command line. This occurred commonly with inex.conf, and has been fixed. The second is when a parent directory was not modified since the last backup, but its subcontents were changed. Backup would normally save the parent directories on its way out of the tree, but if it was interrupted before the parent was saved, the changed files underneath would not be visible to the ls command even though they were backed up; they were restorable by pathname. This new warning can be ignored, the affected files can be removed with the rm -r command (be sure to use the -r option!), or they will naturally expire from the backup with the retain command.

hb: memory usage has been reduced, without reduced performance. If you notice a significant performance decrease with this release, please let us know.

backup: previously, pathnames on the command line containing symbolic links were resolved to the ultimate target, and that’s how the pathname was saved. Now the pathnames are backed up as-is, except that on Mac systems, the pathname might be changed to the actual letter case as stored on disk, ie, backing up /users/jim will actually save /Users/jim to avoid confusion. Thanks Ware!

ls: if /a/b/c is backed up and /a/b is a symlink, ls was not showing /a/b/c. Thanks Ware!

get: if /a/b/c is backed up and /a/b is a symlink, get /a/b did not restore /a/b/c. The target of /a/b must be an existing directory at the time /a/b/c is restored. If /a/b points to a target that has not yet been restored or is not a directory, then /a/b/c cannot be restored either. Thanks Ware!

retire FreeBSD and 32-bit Linux builds

stats: extended metadata lengths were wrong

backup: exclude backup directory with -c symlink

backup: notice for previously seen inodes

hb: DatabaseError: Disk I/O Error exception

the FreeBSD and 32-bit Linux builds of HashBackup are being retired. There were only a handful of upgrades last quarter for 32-bit Linux, so it is hardly used. The FreeBSD build was used by about 2.5% of users, but has had compatibility issues with current FreeBSD releases for about a year.

stats: the maximum extended metadata length shown could be much smaller than the actual maximum. This also affected the --xmeta option, which shows the length of the largest metadata items: extended attributes, ACLs, and symbolic links. This is mainly on MacOS, where Apple sets extended attributes up to 6MB on some directories.

backup: if any -c path component was a symlink, and the local backup directory was beneath a directory being saved, the local backup directory was saved instead of being excluded as it should have been. Thanks Ware!

backup: with -v3, a notice is given for previously seen inodes

hb: some unusual system configurations with chroots and read-only partitions could cause a DatabaseError: Disk I/O Error exception. Thanks Sarah!

get: restore failed on a single deleted path

avoid recursive selftest

ls: fix wildcard bugs for absolute pathnames

ls: give hint for square brackets in pattern

ls: fix sequential scans

get: if one path is being restored and that path is not in the current backup version (it was previously deleted), a message shows the previously-saved version and asks whether to restore from that version. But the restore was actually using the latest version, so nothing was restored. Adding -r with the last-saved version would fix the problem, but that is no longer needed. Thanks Henrik!

on some database errors, selftest runs automatically to try to repair the error. If selftest triggers a database error, don’t run selftest to try to correct it since that causes an infinite loop and forks many processes. Thanks Ware!

ls: if the ls search pattern began with slash (absolute pathname) and used wildcards, ls might say "File not in backup" when the file was actually there. Thanks Andy!

ls: square brackets in the ls search pattern specify a character class (a single character). If an actual pathname contains [xyz-def], it is not possible to search for these 9 characters with ls using brackets. Also, this is an illegal character class because z-d is not a valid character range (though d-z is), causing ls to fail with "error: bad character range". Now, if the pattern contains square brackets and is not a valid regular expression or does not match anything, a hint to change brackets to question marks (match any 1 character) is given. Thanks Andy!

ls: if a search pattern is too complex, ie, more than 9 pathname components without a leading slash, ls did a sequential search of the backup pathnames instead of using database indexes. These scans were not always showing matching filenames. Sequential scans were also used if more than 20% of pathnames matched. This was removed because benchmarks showed that a sequential scan was slower even if 80% of pathnames matched.

compare: round ctime before doing comparisons

compare: the compare command could show differences due to floating point rounding errors. Filesystems store times as two integer fields, seconds and nanoseconds, while Python returns times as floating point seconds, so there can be slight conversion errors between the two formats. The compare command also should not show a difference if ctime is different but mtime is the same. This has not yet been fixed.

dest: suggest verify if downloaded file size is wrong

rm: interrupted --secure could cause selftest errors

rm: don’t reuse arc filenames when packing

rm: interrupted packing could cause selftest errors

dest: when a file is downloaded but the file size is not the expected size, the warning message now suggests running dest verify. This will force the file to be re-synced on the next backup if there is a good copy on another destination.

rm: if rm --secure was interrupted at just the right time, it could cause selftest errors on the last arc file packed. From internal testing.

rm: packing the last arc file in a version could reuse the arc filename when packing other arc files. Depending on the order files are packed, this could delete an arc file that should have been kept, cause an error "dest: can’t access file for put" when uploading the arc file, and cause a selftest error. This bug is unusual, happening 15 times in the past year. Thanks Bugz!

rm: when arc files are empty, a pack operation deletes them. If HB was interrupted before the database committed it could cause "required arc file is missing" selftest errors. From internal testing.

bump expiration date

backup, dest sync: respect cache-size-limit

log: avoid traceback when summarizing huge log files

config: allow lower pack-percent-free >= 1% vs 10%

rename: can’t rename to a subdirectory

rm: traceback removing a dangling pathname

backup, dest sync: cache-size-limit was sometimes not being honored during sync operations. Thanks for the detailed notes Christian!

log: when hb log summarizes log files, it creates a .zip file for each month. For sites with huge logs, the .zip file might be more than 4GB, and this caused a traceback while summarizing the logs. Now .zip files larger han 4GB are created without errors. Thanks Bugz!

config: pack-percent-free previously had a lower limit of 10 (percent) but is now 1. Lowering pack-percent-free causes more frequent arc file packing, saving backup space. It should only be lowered if cache-size-limit is -1 (there is a local copy of all arc files), or the first destination in dest.conf does not have download fees. If you can pack files without triggering high download fees, it’s safe to pack more frequently, though with the low cost of object storage, it’s usually not worth it. Thanks David!

rename: if a directory was renamed to a subdirectory of itself, eg, hb rename /a /a/b, then /a would become inaccessible and selftest would either detect errors in the paths or get stuck. Rename now throws an error in this case. Thanks Rodrigo!

rm: trying to remove an existing pathname that had no associated data caused a traceback rather than the correct error, "Path exists but no files found; run selftest --fix: pathname". Thanks BugZ!

Traceback (most recent call last):
  File "/hb.py", line 303, in <module>
  File "/rm.py", line 1127, in main
  File "/rm.py", line 1024, in rmpath
TypeError: raise: arg 3 must be a traceback or None

rm, selftest: prevent & fix selftest error blockids overlap

ls: strip pathname pattern trailing slashes

trim: prevent hang when destinations configured

trim: Exception: No pathname for pathid 2141069

trim: always add excludes to inex.conf

log: summarize w/o log dir in read-only backup dir

backup: better error on bad wildcard in inex.conf

rm, selftest: if the last block of a backup is deleted with either rm or retain and enough data is deleted to trigger a database compression, then after the next backup, selftest could report an error like "Error: arc.1.0 blockids overlap arc.0.4: 458 ⇐ 486". Selftest was previously unable to correct this error with --fix; now it can. Database compression was changed so this error should not occur. Thanks Yannick!

ls: strip trailing slashes from pathname pattern to avoid "Path not in backup" errors. Thanks Neal!

trim: when items are deleted and deletes are committed, trim could hang if destinations are configured. It did all of its work and the backup was fine, but it got stuck closing an already-closed database. Thanks David!

trim: if a subdirectory and its parent were both marked for delete, trim failed with "Exception: No pathname for pathid xxx" and no trim changes occurred. Now, if /a and /a/b are both deleted, /a is deleted and /a/b gets deleted because it is a subdirectory of /a. Thanks Bugz!

trim: if a path is marked exclude, it was only added to inex.conf if it was already excluded. Now excludes are always added, whether they were previously excluded or not. From internal testing.

log: if the backup directory is read-only, the logs directory doesn’t exist, and hb log is asked to summarize the logs, return 0 (no errors) rather than throwing a Permission denied error trying to create the logs directory. Thanks Bugz!

backup: a bad wildcard in inex.conf, for example [x-a], caused a traceback with no context and a bug report. Now a better error message is displayed before aborting, showing the bad wildcard. Thanks Bugz!