Preview

The preview release is installed with hb upgrade -p. Using preview releases for non-critical backups helps find and solve problems early. The official release is sent to the preview area so hb upgrade -p gets the official release when it’s available. Thank you for your help!

Preview Notes

Apr 9 #3067: official release
Apr 9 #3066: bump expiration date
Mar 6 #3065: sync: avoid overfilling cache
Feb 25 #3063: rm: traceback removing dangling pathname
Feb 15 #3062: error if rename to subdirectory
Jan 30 #3061: pack-percent-free > 1% instead of 10%
Jan 20 #3059: create large .zip files for huge logs
Jan 11 #3058: official release

#3067 Apr 9, 2023

Summary:

backup, dest sync: respect cache-size-limit
log: avoid traceback when summarizing huge log files
config: allow lower pack-percent-free >= 1% vs 10%
rename: can’t rename to a subdirectory
rm: traceback removing a dangling pathname

Details:

backup, dest sync: cache-size-limit was sometimes not being honored during sync operations. Thanks for the detailed notes Christian!
log: when hb log summarizes log files, it creates a .zip file for each month. For sites with huge logs, the .zip file might be more than 4GB, and this caused a traceback while summarizing the logs. Now .zip files larger han 4GB are created without errors. Thanks Bugz!
config: pack-percent-free previously had a lower limit of 10 (percent) but is now 1. Lowering pack-percent-free causes more frequent arc file packing, saving backup space. It should only be lowered if cache-size-limit is -1 (there is a local copy of all arc files), or the first destination in dest.conf does not have download fees. If you can pack files without triggering high download fees, it’s safe to pack more frequently, though with the low cost of object storage, it’s usually not worth it. Thanks David!
rename: if a directory was renamed to a subdirectory of itself, eg, hb rename /a /a/b, then /a would become inaccessible and selftest would either detect errors in the paths or get stuck. Rename now throws an error in this case. Thanks Rodrigo!

rm: trying to remove an existing pathname that had no associated data caused a traceback rather than the correct error, "Path exists but no files found; run selftest --fix: pathname". Thanks BugZ!

Traceback (most recent call last):
  File "/hb.py", line 303, in <module>
  File "/rm.py", line 1127, in main
  File "/rm.py", line 1024, in rmpath
TypeError: raise: arg 3 must be a traceback or None

#3058 Jan 11, 2023

Summary:

rm, selftest: prevent & fix selftest error blockids overlap
ls: strip pathname pattern trailing slashes
trim: prevent hang when destinations configured
trim: Exception: No pathname for pathid 2141069
trim: always add excludes to inex.conf
log: summarize w/o log dir in read-only backup dir
backup: better error on bad wildcard in inex.conf

Details:

rm, selftest: if the last block of a backup is deleted with either rm or retain and enough data is deleted to trigger a database compression, then after the next backup, selftest could report an error like "Error: arc.1.0 blockids overlap arc.0.4: 458 ⇐ 486". Selftest was previously unable to correct this error with --fix; now it can. Database compression was changed so this error should not occur. Thanks Yannick!
ls: strip trailing slashes from pathname pattern to avoid "Path not in backup" errors. Thanks Neal!
trim: when items are deleted and deletes are committed, trim could hang if destinations are configured. It did all of its work and the backup was fine, but it got stuck closing an already-closed database. Thanks David!
trim: if a subdirectory and its parent were both marked for delete, trim failed with "Exception: No pathname for pathid xxx" and no trim changes occurred. Now, if /a and /a/b are both deleted, /a is deleted and /a/b gets deleted because it is a subdirectory of /a.
trim: if a path is marked exclude, it was only added to inex.conf if it was already excluded. Now excludes are always added, whether they were previously excluded or not.
log: if the backup directory is read-only, the logs directory doesn’t exist, and hb log is asked to summarize the logs, return 0 (no errors) rather than throwing a Permission denied error trying to create the logs directory. Thanks Bugz!
backup: a bad wildcard in inex.conf, for example [x-a], caused a traceback with no context and a bug report. Now a better error message is displayed before aborting, showing the bad wildcard.

History

#3058 Jan 11, 2023

Jan 11 #3058: official release
Jan 11 #3058: prevent and fix selftest error: blockids overlap
Jan 11 #3057: bump expiration date
Jan 5 #3056: inex.conf: better error on bad wildcard
Dec 14 #3055: log: summarize w/o log dir in RO backup dir
Dec 1 #3054: trim: double db close causing hang
Nov 27 #3053: trim: always add excludes to inex.conf
Nov 27 #3053: trim: traceback if parent and subdir deleted
Nov 10 #3052: ls: strip pathname pattern trailing slashes
Nov 1 #3051: experiment: preload database for get

#3050 Sep 9, 2022

Sep 9 #3050: official release
Sep 9 #3049: get: remove experimental inex exclusion
Sep 7 #3048: Bugz: strip pathnames if background thread fails
Sep 5 #3047: mount: revert back to #3040; couldn’t fix FreeBSD FUSE
Sep 5 #3047: get: traceback during restore
Sep 5 #3046: backup, get, compare: round mtime more aggressively
Aug 31 #3042: mount: adjust for FreeBSD 12+
Aug 25 #3040: get: honor inex.conf excludes for restore

#3039 Aug 17, 2022

Aug 17 #3039: official release: no attribute 'SIGINFO' on Linux

#3038 Aug 14, 2022

Aug 14 #3038: official release
Aug 7 #3037: backup: Exception: Bug: varblock in mixed threads

#3036 Jul 29, 2022

Jul 29 #3036: official release
Jul 24 #3035: log: forward siginfo signal
Jul 21 #3033: recover: arc file download rename race condition
Jul 14 #3032: official release

#3032 Jul 14, 2022

Jul 14 #3032: official release
Jul 14 #3032: db upgrade: remove .uporig if dbrev is correct, check disk space before upgrade
Jul 14 #3031: dest needput: revise conditions
Jul 8 #3030: selftest: enable dev testing of -v0 and -v1
Jul 4 #3029: remove leading slash from inline tracebacks
Jul 3 #3028: db upgrade: remove cached arc files to make space (held)
Jul 3 #3027: backup: reorder mtime tests
Jul 2 #3026: backup: cache was not always getting cleaned up
Jul 2 #3025: backup, get: handle int resolutions on mtimes
Jul 1 #3024: dest noparts: skip files with null info
Jun 30 #3023: selftest: only fix dedup table with --fix
Jun 27 #3022: get: rtrim → rstrip. Thanks Bugz!
Jun 27 #3021: recover: no such table: destfiles. Thanks Bugz!
Jun 25 #3020: dest.conf: better handling of keywords with missing values
Jun 25 #3019: dest: re-wrap verify help to match other subcommands
Jun 22 #3018: log: handle no such process in sig handler. Thanks Bugz!
Jun 22 #3017: handle -r with missing/empty versions better. Thanks Bugz!
Jun 21 #3016: backup: no partial backup warning for char devices
Jun 18 #3015: official release

#3015 Jun 18, 2022

Jun 18 #3015: official release
Jun 16 #3012: get: traceback restoring deleted files
Jun 16 #3011: backup: distinguish interupts from timeouts
Jun 12 #3010: send ctrl-c and sigterm to commands
Jun 12 #3009: dest nopart: exit on ctrl-c and kill -term
Jun 7 #3007: destinations: ending support for maxsize
Jun 6 #3006: cleanup mount change in #3005
Jun 5 #3005: mount: reading variable-block files could crash mount
Jun 3 #3004: backup, mount: minor message changes
Jun 2 #3003: destcmd: don’t require setid after dest rename. Thanks Preston!
Jun 2 #3002: mount: remove mount timeout for slow destinations
Jun 2 #3001: mount: use mkdirp to create --cache directory. Thanks Bugz!
May 31 #2999: mount: remove a debug stack trace
May 31 #2998: recover: create testonly.conf file
May 30 #2997: compress inline traceback a bit more
May 30 #2996: dest.conf: integer keyword defaults
May 30 #2995: auto-run selftest in several more situations
May 30 #2994: mount improvements. Thanks Preston!
May 25 #2993: clear: no "if you continue" with --force
May 25 #2992: remove Rackspace Cloudfiles remnants
May 25 #2991: log: traceback if program completes during summarizing. Thanks Bugz!
May 24 #2990: security disclosure from 2022-02-21 published
May 24 #2990: handle rotating USB drives better
May 23 #2989: get: restore pathnames can start with ../
May 22 #2988: selftest, rm: change cache initialization
May 19 #2985: dir dest: was requiring too much free disk space
May 19 #2984: generalize reading passphrases
May 19 #2983: selftest: --fix corrects a few more errors
May 18 #2982: trim: flush stderr so 1st pathname isn’t cleared
May 18 #2981: trim: traceback "getting called too often"
May 18 #2979: log: prevent status line escape sequences in logs
May 17 #2978: dest rename: new command

#2976 May 17, 2022

May 17 #2976: official release
May 17 #2975: disabled destinations being selected
May 16 #2973: recover: use multiple destinations if necessary
May 15 #2970: clear progress reports on stderr write
May 9 #2969: versions: File Space was sometimes incorrectly blank
May 8 #2968: backup: traceback when rebuilding dedup table
May 7 #2967: selftest: upload hb.db if it was changed
May 7 #2966: ls: show date deleted
May 7 #2965: retain: use new status line code
May 6 #2964: db upgrade: error vs traceback for permission failures
May 5 #2963: hb: auto selftest may need to run 4 times, not 3
May 4 #2962: get: handle some fringe error situations
May 2 #2958: dest: remove sync after verify is complete
May 1 #2957: recover: show better stats during download
May 1 #2956: upgrade: add --- End release notes ---
May 1 #2955: hb: auto run selftest --fix for missing arc files
May 1 #2954: rm/retain: if download fails try other destinations
May 1 #2953: hb: bug fix for #2949: Unable to download archive arc.0.0: AttributeError("'module' object has no attribute 'report'"
Apr 30 #2952: log: if log disappears don’t traceback on rename
Apr 30 #2951: hb: forward SIGTERM so backup sees it
Apr 29 #2950: selftest: remove read count difference notice
Apr 29 #2949: add a couple more wait status lines
Apr 29 #2948: backup: better wait time stats
Apr 29 #2947: backup: show status msg while waiting for cache space
Apr 28 #2946: backup: set timelimit flag on SIGTERM to stop sooner
Apr 28 #2945: backup: moved and enabled signal (SIGTERM) handler
Apr 28 #2943: backup: don’t set checkpoint to last file
Apr 28 #2942: backup: maxtime interrupt on large files
Apr 27 #2941: backup: show maxwait stop goal
Apr 27 #2940: backup: delay maxwait after maxtime expires
Apr 26 #2939: use statline() and progrep() for status lines
Apr 26 #2937: get: early termination could hang with limited cache
Apr 25 #2936: selftest: more uniform -v5 progress reports
Apr 25 #2935: selftest: --sample fix, status line
Apr 25 #2934: misc.statline() and progrep()
Apr 24 #2933: Bugz: don’t send tracebacks in development
Apr 24 #2932: dest: setid arg not required if only 1 destination
Apr 24 #2931: set DESTID carefully to avoid a setid in edge cases
Apr 24 #2930: hb: avoid traceback if db can’t be opened
Apr 23 #2929: backup: further reduce --maxtime overruns
Apr 22 #2927: backup: close/unlock db during final sync
Apr 22 #2926: config: doesn’t need to lock the backup directory
Apr 22 #2924: rm: don’t run sync to delete arc files
Apr 21 #2923: misc.timeinterval() accepts multiple time periods
Apr 21 #2922: misc.holdquit() signal handler
Apr 21 #2921: misc.intty(), misc.outtty()
Apr 21 #2920: log: -X didn’t always exclude lines
Apr 17 #2919: backup: --maxtime/--maxwait more accurate
Apr 17 #2918: handle negative time intervals in show.vistime()
Apr 17 #2917: backup: poll destinations while waiting for cache
Apr 16 #2916: init: avoid traceback if -c is a file. Thanks Bugz!
Apr 16 #2915: hb: disable stderr redirection for automated tests
Apr 15 #2914: recover: HMAC signature verification failed
Apr 14 #2913: config: conditionally display enable/disable-commands note
Apr 14 #2912: config: flush stdout before reading admin-passphrase to avoid confusing output; handle EOF (ctrl-d) when entering admin-passphrase to avoid traceback - thanks Bugz!
Apr 14 #2911: don’t send tracebacks from automated tests
Apr 13 #2910: hb: don’t update exit code in tests (sync test)

#2909 Apr 12, 2022

Apr 12 #2909: official release
Apr 12 #2909: config: traceback on invalid or read-only key
Apr 12 #2908: hb command --help shows command help. Thanks Arthur!
Apr 11 #2907: close db after integrity check
Apr 8 #2905: hb, selftest: try to correct malformed database
Apr 8 #2903: trim: show message when -i isn’t used (maybe forgot it)
Apr 7 #2902: trim: traceback with -s
Apr 7 #2901: trim: % character causing help traceback
Apr 6 #2900: Bugz: don’t skip line if at beginning of log
Apr 6 #2899: Bugz: handle extremely long traceback (internal testing)
Apr 6 #2898: Bugz: flush stdout and delay to get end of traceback

#2897 Apr 6, 2022

Apr 6 #2897: official release
Apr 6 #2897: dest: cPickle.loads() type mismatch. Thanks Ware!
Apr 6 #2896: trim: -n always a percentage, better time left estimate
Apr 5 #2895: trim: -n % limit was getting applied twice
Apr 5 #2894: trim: limit versions with -n too
Apr 5 #2893: trim: -n defaults to top 50% of backup
Apr 5 #2891: log: ctrl-c does not log as _FAIL or _RUN
Apr 5 #2890: trim: check backup ownership for keep & delete
Apr 4 #2889: trim: reorganized code, dash should have been r for remove, reinstate -n w/default 1000 for -i, 100000 otherwise, add g command to go to (search for) pathname, add i command to show file details
Apr 4 #2886: trim: remove blank line when paging with space bar
Apr 4 #2885: trim: change --paths option to -p
Apr 4 #2884: trim: add note in show deletes about smaller files
Apr 3 #2883: trim: unique data too low, needs float division
Apr 3 #2882: trim: more accurate (but slower) scan for unique data, show time left

#2881 Apr 3, 2022

Apr 3 #2881: official release
Apr 3 #2881: db upgrade: fix AttributeError: 'NoneType' object has no attribute 'items'
Apr 2 #2877: ssh: handle dd unavailable on remote

#2875 Apr 1, 2022

Apr 1 #2875: official release
Mar 31 #2874: selftest: fix %% in progress string; upgrade: show current version
Mar 31 #2873: tune: tune paths, fix and re-enable block commits, speed up tune with many arc files
Mar 30 #2872: tune: disable block commits: interrupt testing found a bug
Mar 30 #2871: compare: not showing all changes, mostly deletes
Mar 29 #2870: trim: space key, fix suspend
Mar 29 #2869: trim: typo in #2861 added duplicate entries to trim list
Mar 28 #2868: export: workaround NFS creating .nfs files for large databases
Mar 28 #2866: backup: was not deleting old hb.db.N files
Mar 27 #2865: backup: sync remote-to-remote after backup vs at the start
Mar 26 #2864: tune: don’t delete version records; versions suppresses them
Mar 25 #2863: db upgrade: tune failed on old backups
Mar 25 #2862: arcmb didn’t work out, removed it
Mar 25 #2861: trim: --paths option to sort by pathname
Mar 25 #2860: rm: show user bytes deleted
Mar 22 #2858: selftest: performance in Paths II w/huge pathids
Mar 22 #2857: trim: can’t keep kid if parent deleted
Mar 22 #2856: export: delete key.conf.0
Mar 22 #2855: db upgrade: set arcmb, use MB not MiB
Mar 22 #2854: trim: save keeps in db for future trims
Mar 21 #2853: trim: line numbers, l and / don’t unstack
Mar 21 #2852: trim: <, >, digits
Mar 21 #2851: trim: l (el) list directory
Mar 21 #2850: trim: K keeps whole directories
Mar 21 #2849: trim: skip kids if parents deleted
Mar 20 #2848: backup: large file tracking, db upgrade changes
Mar 20 #2847: versions: simplify code
Mar 20 #2846: get: atexit traceback after restore. Thanks David!
Mar 19 #2845: trim: check files > 1MB to improve speed
Mar 19 #2843: trim: slashes on directories
Mar 18 #2842: trim: more -i options, fixed -s0
Mar 18 #2841: trim: flush stdout before changing tty mode
Mar 18 #2840: trim: can’t delete / with -i
Mar 18 #2839: backup: checked / saved progress off by 1
Mar 18 #2838: trim: more -i options
Mar 17 #2837: stats: bug in db library gave negative stats w/10GB db
Mar 16 #2834: export: close db before delete to avoid NFS errors
Mar 16 #2830: trim: suspend on ctrl-z
Mar 15 #2829: backup: round mtimes for comparisons
Mar 15 #2828: Bugz: map pathnames to numbers for privacy
Mar 14 #2827: trim: -i option added for interactive removal
Mar 14 #2826: trim: -d option removed; always shows directories
Mar 14 #2825: backup: error count 1 off if destinations halted
Mar 14 #2824: backup: show large files and recent errors
Mar 10 #2823: backup: --maxwait off disables destinations
Mar 9 #2822: log: show from & to dates in log summary
Mar 8 #2821: don’t pickle command for audit log
Mar 8 #2820: db upgrade changes
Mar 7 #2818: backup: save char block devices, tb in errors
Mar 7 #2817: yesno: delay writing to /dev/tty until stdout flushes
Mar 7 #2816: selftest: print block progress by space used
Mar 5 #2814: new base17 dbid
Mar 5 #2813: rm: always repack cblocks with --secure
Mar 3 #2805: trim: new command to find large files in backup
Mar 3 #2804: log: always disable stdout buffering for timestamps
Mar 3 #2803: tune: gave error if last backup was interrupted
Mar 2 #2802: backup: remove last version if nothing saved
Mar 2 #2801: recover is now logged
Mar 2 #2800: recover: be friendly if no destination on command line
Mar 2 #2799: backup: print warning if more than 4 billion blocks
Mar 1 #2798: backup: don’t crash if more than 4 billion blocks
Mar 1 #2797: Bugz: include end of log with tracebacks
Mar 1 #2796: db: clear journal before closing db for upgrade
Mar 1 #2795: versions: write errors to stderr not stdout
Mar 1 #2794: recover: static build requires sys.exit(), not exit()
Mar 1 #2793: improve readkey on message, shared import loop fix
Mar 1 #2792: db upgrade changes
Feb 28 #2791: traceback if logs summarized while program running
Feb 28 #2790: hb -c w/o backupdir should error, not traceback
Feb 28 #2789: handle stdout redirected on yes/no question
Feb 26 #2788: tune: divide by zero if no data saved
Feb 26 #2787: versions: better output, -l obsolete
Feb 26 #2786: compare: round mtimes
Feb 26 #2784: DATABASE UPGRADE
Feb 26 #2783: review readkey help
Feb 26 #2782: prep rekey for db upgrade
Feb 26 #2781: prep export for db upgrade
Feb 26 #2780: don’t try to clear journals on every open
Feb 22 #2779: traceback if database locked
Feb 22 #2775: add sparse support for APFS
Feb 22 #2774: default dedup is 100MB
Feb 22 #2773: shard exit code incorrect

#2772 Feb 21, 2022

Feb 21 #2772: official release
Feb 21 #2772: security issue, disclosure in 90 days
Feb 21 #2771: stats: split block and ref db space
Feb 21 #2770: better error for -c
Feb 21 #2769: handle halted destinations better
Feb 20 #2766: log when -c isn’t used
Feb 20 #2765: show source of backup directory
Feb 17 #2764: mount: make intermediate directories
Feb 17 #2762: add --iso option to selftest for tune testing
Feb 17 #2760: better permission error messages
Feb 17 #2759: compare traceback on empty directories

#2758 Feb 16, 2022

Feb 16 #2758: official release
Feb 16 #2756: symlink bug in Dir destination
Feb 16 #2754: remove progress bars from log files
Feb 15 #2752: cleanup tune arguments
Feb 15 #2751: log changes disabled all progress bars
Feb 15 #2744: add tune command w/progress
Feb 14 #2740: remove tune db precheck
Feb 14 #2739: remove unused code from tune
Feb 14 #2738: update release notes for #2735
Feb 13 #2737: rm: error message if path not found with -r
Feb 13 #2735: remove feature to remove entire versions
Feb 13 #2734: double audit logging
Feb 12 #2731: create log directory in init, log permission issues
Feb 11 #2730: rm: KeyError traceback
Feb 11 #2729: rm: remove db commit
Feb 11 #2725: separate preview notes
Feb 11 #2724: sort log summaries
Feb 11 #2723: log log → log: don’t log the log command even if users request it, to avoid a traceback when finished

#2722 Feb 10, 2022

Feb 10 #2722: released
Feb 10 #2720: log -s option is obsolete
Feb 9 #2716: add instructions for upgrading old dbrevs
Feb 9 #2715: remove hidden command handling (not used)
Feb 9 #2714: print copyright heading in parent, log in child
Feb 9 #2713: copyright heading missing for some commands
Feb 9 #2712: write /tmp cafile if needed for SSL w/o -c
Feb 9 #2711: remove support for dbrev < 32 (Feb 16, 2019)
Feb 9 #2710: log: pass back real exit code
Feb 8 #2704+: not so much fun with version control
Feb 6 #2703: selftest --inc underflow
Feb 2 #2702: hb log -s didn’t work
Feb 2 #2701: fix "Is a directory" in rmx #2683. Thanks Ware!
Feb 2 #2700: log every command
Feb 1 #2694: log perf, log clear, log command
Jan 31 #2692: ls -1 w/o -r shows last version
Jan 31 #2690: rm: incorrect accounting when removing blocks
Jan 31 #2689: selftest: don’t delete bad block w/o --fix
Jan 28 #2688: auto run selftest --fix for hash mismatch
Jan 27 #2682: minor formatting changes for stats --xmeta
Jan 26 #2681: fix dbstat build error on non-OSX. Thanks Evert!
Jan 25 #2680: release failed arc download from cache
Jan 22 #2679: NFS spans.tmp bug. Thanks Paul!
Jan 20 #2678: log -x0 and -x1