Preview

The preview release is installed with hb upgrade -p. Using preview releases for non-critical backups helps find and solve problems early. The official release is sent to the preview area so hb upgrade -p gets the official release when it’s available. Thank you for your help!

Preview Notes

  • Nov 30 #3082: get: restore fails on single deleted file/directory

  • Nov 4 #3081: log: no error on file remove if already gone (race)

  • Oct 31 #3080: avoid recursive selftest

  • Oct 23 #3079: ls: fix sequential scans

  • Oct 21 #3078: ls: fix wildcard bugs, give hint for square bracket error in pattern

  • Oct 14 #3077: official release

#3082 Nov 30, 2023

Summary:

  • get: restore failed on a single deleted path

  • avoid recursive selftest

  • ls: fix wildcard bugs for absolute pathnames

  • ls: give hint for square brackets in pattern

  • ls: fix sequential scans

Details:

  • get: if one path is being restored and that path is not in the current backup version (it was previously deleted), a message shows the previously-saved version and asks whether to restore from that version. But the restore was actually using the latest version, so nothing was restored. Adding -r with the last-saved version would fix the problem, but that is no longer needed. Thanks Henrik!

  • on some database errors, selftest runs automatically to try to repair the error. If selftest triggers a database error, don’t run selftest to try to correct it since that causes an infinite loop and forks many processes. Thanks Ware!

  • ls: if the ls search pattern began with slash (absolute pathname) and used wildcards, ls might say "File not in backup" when the file was actually there. Thanks Andy!

  • ls: square brackets in the ls search pattern specify a character class (a single character). If an actual pathname contains [xyz-def], it is not possible to search for these 9characters with ls using brackets. Also, this is an illegal character class because z-d is not a valid character range (though d-z is), causing ls to fail with "error: bad character range". Now, if the pattern contains square brackets and is not a valid regular expression or does not match anything, a hint to change brackets to question marks (match any 1 character) is given. Thanks Andy!

  • ls: if a search pattern is too complex, ie, more than 9 pathname components without a leading slash, ls does a sequential search of the backup pathnames instead of using database indexes. These scans were not always showing matching filenames. Sequential scans were also used if more than 20% of pathnames matched. This was removed because benchmarks showed that a sequential scan was slower even if 80% of pathnames matched.

History

#3077 Oct 14, 2023

  • Oct 14 #3077: official release

  • Oct 14 #3076: compare: round ctime before doing comparisons

#3075 Sep 1, 2023

  • Sep 1 #3075: official release

  • Aug 17 #3073: dest: suggest verify if downloaded file size is wrong

  • Jul 16 #3072: mount: some mount errors gave "global name 'strerr' is not defined" instead of the actual error

  • Jul 15 #3071: rm: interrupted packing could cause selftest errors

  • Jul 15 #3070: rm: don’t reuse arc filenames when packing

  • Jul 14 #3069: rm: interrupted --secure could cause selftest errors

#3068 Jul 14, 2023

  • Jul 14 #3068: official release

  • Jul 13 #3068: bump expiration date

#3067 Apr 9, 2023

  • Apr 9 #3067: official release

  • Apr 9 #3066: bump expiration date

  • Mar 6 #3065: sync: avoid overfilling cache

  • Feb 25 #3063: rm: traceback removing dangling pathname

  • Feb 15 #3062: error if rename to subdirectory

  • Jan 30 #3061: pack-percent-free > 1% instead of 10%

  • Jan 20 #3059: create large .zip files for huge logs

  • Jan 11 #3058: official release

#3058 Jan 11, 2023

  • Jan 11 #3058: official release

  • Jan 11 #3058: prevent and fix selftest error: blockids overlap

  • Jan 11 #3057: bump expiration date

  • Jan 5 #3056: inex.conf: better error on bad wildcard

  • Dec 14 #3055: log: summarize w/o log dir in RO backup dir

  • Dec 1 #3054: trim: double db close causing hang

  • Nov 27 #3053: trim: always add excludes to inex.conf

  • Nov 27 #3053: trim: traceback if parent and subdir deleted

  • Nov 10 #3052: ls: strip pathname pattern trailing slashes

  • Nov 1 #3051: experiment: preload database for get

#3050 Sep 9, 2022

  • Sep 9 #3050: official release

  • Sep 9 #3049: get: remove experimental inex exclusion

  • Sep 7 #3048: Bugz: strip pathnames if background thread fails

  • Sep 5 #3047: mount: revert back to #3040; couldn’t fix FreeBSD FUSE

  • Sep 5 #3047: get: traceback during restore

  • Sep 5 #3046: backup, get, compare: round mtime more aggressively

  • Aug 31 #3042: mount: adjust for FreeBSD 12+

  • Aug 25 #3040: get: honor inex.conf excludes for restore

#3039 Aug 17, 2022

  • Aug 17 #3039: official release: no attribute 'SIGINFO' on Linux

#3038 Aug 14, 2022

  • Aug 14 #3038: official release

  • Aug 7 #3037: backup: Exception: Bug: varblock in mixed threads

#3036 Jul 29, 2022

  • Jul 29 #3036: official release

  • Jul 24 #3035: log: forward siginfo signal

  • Jul 21 #3033: recover: arc file download rename race condition

  • Jul 14 #3032: official release

#3032 Jul 14, 2022

  • Jul 14 #3032: official release

  • Jul 14 #3032: db upgrade: remove .uporig if dbrev is correct, check disk space before upgrade

  • Jul 14 #3031: dest needput: revise conditions

  • Jul 8 #3030: selftest: enable dev testing of -v0 and -v1

  • Jul 4 #3029: remove leading slash from inline tracebacks

  • Jul 3 #3028: db upgrade: remove cached arc files to make space (held)

  • Jul 3 #3027: backup: reorder mtime tests

  • Jul 2 #3026: backup: cache was not always getting cleaned up

  • Jul 2 #3025: backup, get: handle int resolutions on mtimes

  • Jul 1 #3024: dest noparts: skip files with null info

  • Jun 30 #3023: selftest: only fix dedup table with --fix

  • Jun 27 #3022: get: rtrim → rstrip. Thanks Bugz!

  • Jun 27 #3021: recover: no such table: destfiles. Thanks Bugz!

  • Jun 25 #3020: dest.conf: better handling of keywords with missing values

  • Jun 25 #3019: dest: re-wrap verify help to match other subcommands

  • Jun 22 #3018: log: handle no such process in sig handler. Thanks Bugz!

  • Jun 22 #3017: handle -r with missing/empty versions better. Thanks Bugz!

  • Jun 21 #3016: backup: no partial backup warning for char devices

  • Jun 18 #3015: official release

#3015 Jun 18, 2022

  • Jun 18 #3015: official release

  • Jun 16 #3012: get: traceback restoring deleted files

  • Jun 16 #3011: backup: distinguish interupts from timeouts

  • Jun 12 #3010: send ctrl-c and sigterm to commands

  • Jun 12 #3009: dest nopart: exit on ctrl-c and kill -term

  • Jun 7 #3007: destinations: ending support for maxsize

  • Jun 6 #3006: cleanup mount change in #3005

  • Jun 5 #3005: mount: reading variable-block files could crash mount

  • Jun 3 #3004: backup, mount: minor message changes

  • Jun 2 #3003: destcmd: don’t require setid after dest rename. Thanks Preston!

  • Jun 2 #3002: mount: remove mount timeout for slow destinations

  • Jun 2 #3001: mount: use mkdirp to create --cache directory. Thanks Bugz!

  • May 31 #2999: mount: remove a debug stack trace

  • May 31 #2998: recover: create testonly.conf file

  • May 30 #2997: compress inline traceback a bit more

  • May 30 #2996: dest.conf: integer keyword defaults

  • May 30 #2995: auto-run selftest in several more situations

  • May 30 #2994: mount improvements. Thanks Preston!

  • May 25 #2993: clear: no "if you continue" with --force

  • May 25 #2992: remove Rackspace Cloudfiles remnants

  • May 25 #2991: log: traceback if program completes during summarizing. Thanks Bugz!

  • May 24 #2990: security disclosure from 2022-02-21 published

  • May 24 #2990: handle rotating USB drives better

  • May 23 #2989: get: restore pathnames can start with ../

  • May 22 #2988: selftest, rm: change cache initialization

  • May 19 #2985: dir dest: was requiring too much free disk space

  • May 19 #2984: generalize reading passphrases

  • May 19 #2983: selftest: --fix corrects a few more errors

  • May 18 #2982: trim: flush stderr so 1st pathname isn’t cleared

  • May 18 #2981: trim: traceback "getting called too often"

  • May 18 #2979: log: prevent status line escape sequences in logs

  • May 17 #2978: dest rename: new command

#2976 May 17, 2022

  • May 17 #2976: official release

  • May 17 #2975: disabled destinations being selected

  • May 16 #2973: recover: use multiple destinations if necessary

  • May 15 #2970: clear progress reports on stderr write

  • May 9 #2969: versions: File Space was sometimes incorrectly blank

  • May 8 #2968: backup: traceback when rebuilding dedup table

  • May 7 #2967: selftest: upload hb.db if it was changed

  • May 7 #2966: ls: show date deleted

  • May 7 #2965: retain: use new status line code

  • May 6 #2964: db upgrade: error vs traceback for permission failures

  • May 5 #2963: hb: auto selftest may need to run 4 times, not 3

  • May 4 #2962: get: handle some fringe error situations

  • May 2 #2958: dest: remove sync after verify is complete

  • May 1 #2957: recover: show better stats during download

  • May 1 #2956: upgrade: add --- End release notes ---

  • May 1 #2955: hb: auto run selftest --fix for missing arc files

  • May 1 #2954: rm/retain: if download fails try other destinations

  • May 1 #2953: hb: bug fix for #2949: Unable to download archive arc.0.0: AttributeError("'module' object has no attribute 'report'"

  • Apr 30 #2952: log: if log disappears don’t traceback on rename

  • Apr 30 #2951: hb: forward SIGTERM so backup sees it

  • Apr 29 #2950: selftest: remove read count difference notice

  • Apr 29 #2949: add a couple more wait status lines

  • Apr 29 #2948: backup: better wait time stats

  • Apr 29 #2947: backup: show status msg while waiting for cache space

  • Apr 28 #2946: backup: set timelimit flag on SIGTERM to stop sooner

  • Apr 28 #2945: backup: moved and enabled signal (SIGTERM) handler

  • Apr 28 #2943: backup: don’t set checkpoint to last file

  • Apr 28 #2942: backup: maxtime interrupt on large files

  • Apr 27 #2941: backup: show maxwait stop goal

  • Apr 27 #2940: backup: delay maxwait after maxtime expires

  • Apr 26 #2939: use statline() and progrep() for status lines

  • Apr 26 #2937: get: early termination could hang with limited cache

  • Apr 25 #2936: selftest: more uniform -v5 progress reports

  • Apr 25 #2935: selftest: --sample fix, status line

  • Apr 25 #2934: misc.statline() and progrep()

  • Apr 24 #2933: Bugz: don’t send tracebacks in development

  • Apr 24 #2932: dest: setid arg not required if only 1 destination

  • Apr 24 #2931: set DESTID carefully to avoid a setid in edge cases

  • Apr 24 #2930: hb: avoid traceback if db can’t be opened

  • Apr 23 #2929: backup: further reduce --maxtime overruns

  • Apr 22 #2927: backup: close/unlock db during final sync

  • Apr 22 #2926: config: doesn’t need to lock the backup directory

  • Apr 22 #2924: rm: don’t run sync to delete arc files

  • Apr 21 #2923: misc.timeinterval() accepts multiple time periods

  • Apr 21 #2922: misc.holdquit() signal handler

  • Apr 21 #2921: misc.intty(), misc.outtty()

  • Apr 21 #2920: log: -X didn’t always exclude lines

  • Apr 17 #2919: backup: --maxtime/--maxwait more accurate

  • Apr 17 #2918: handle negative time intervals in show.vistime()

  • Apr 17 #2917: backup: poll destinations while waiting for cache

  • Apr 16 #2916: init: avoid traceback if -c is a file. Thanks Bugz!

  • Apr 16 #2915: hb: disable stderr redirection for automated tests

  • Apr 15 #2914: recover: HMAC signature verification failed

  • Apr 14 #2913: config: conditionally display enable/disable-commands note

  • Apr 14 #2912: config: flush stdout before reading admin-passphrase to avoid confusing output; handle EOF (ctrl-d) when entering admin-passphrase to avoid traceback - thanks Bugz!

  • Apr 14 #2911: don’t send tracebacks from automated tests

  • Apr 13 #2910: hb: don’t update exit code in tests (sync test)

#2909 Apr 12, 2022

  • Apr 12 #2909: official release

  • Apr 12 #2909: config: traceback on invalid or read-only key

  • Apr 12 #2908: hb command --help shows command help. Thanks Arthur!

  • Apr 11 #2907: close db after integrity check

  • Apr 8 #2905: hb, selftest: try to correct malformed database

  • Apr 8 #2903: trim: show message when -i isn’t used (maybe forgot it)

  • Apr 7 #2902: trim: traceback with -s

  • Apr 7 #2901: trim: % character causing help traceback

  • Apr 6 #2900: Bugz: don’t skip line if at beginning of log

  • Apr 6 #2899: Bugz: handle extremely long traceback (internal testing)

  • Apr 6 #2898: Bugz: flush stdout and delay to get end of traceback

#2897 Apr 6, 2022

  • Apr 6 #2897: official release

  • Apr 6 #2897: dest: cPickle.loads() type mismatch. Thanks Ware!

  • Apr 6 #2896: trim: -n always a percentage, better time left estimate

  • Apr 5 #2895: trim: -n % limit was getting applied twice

  • Apr 5 #2894: trim: limit versions with -n too

  • Apr 5 #2893: trim: -n defaults to top 50% of backup

  • Apr 5 #2891: log: ctrl-c does not log as _FAIL or _RUN

  • Apr 5 #2890: trim: check backup ownership for keep & delete

  • Apr 4 #2889: trim: reorganized code, dash should have been r for remove, reinstate -n w/default 1000 for -i, 100000 otherwise, add g command to go to (search for) pathname, add i command to show file details

  • Apr 4 #2886: trim: remove blank line when paging with space bar

  • Apr 4 #2885: trim: change --paths option to -p

  • Apr 4 #2884: trim: add note in show deletes about smaller files

  • Apr 3 #2883: trim: unique data too low, needs float division

  • Apr 3 #2882: trim: more accurate (but slower) scan for unique data, show time left

#2881 Apr 3, 2022

  • Apr 3 #2881: official release

  • Apr 3 #2881: db upgrade: fix AttributeError: 'NoneType' object has no attribute 'items'

  • Apr 2 #2877: ssh: handle dd unavailable on remote

#2875 Apr 1, 2022

  • Apr 1 #2875: official release

  • Mar 31 #2874: selftest: fix %% in progress string; upgrade: show current version

  • Mar 31 #2873: tune: tune paths, fix and re-enable block commits, speed up tune with many arc files

  • Mar 30 #2872: tune: disable block commits: interrupt testing found a bug

  • Mar 30 #2871: compare: not showing all changes, mostly deletes

  • Mar 29 #2870: trim: space key, fix suspend

  • Mar 29 #2869: trim: typo in #2861 added duplicate entries to trim list

  • Mar 28 #2868: export: workaround NFS creating .nfs files for large databases

  • Mar 28 #2866: backup: was not deleting old hb.db.N files

  • Mar 27 #2865: backup: sync remote-to-remote after backup vs at the start

  • Mar 26 #2864: tune: don’t delete version records; versions suppresses them

  • Mar 25 #2863: db upgrade: tune failed on old backups

  • Mar 25 #2862: arcmb didn’t work out, removed it

  • Mar 25 #2861: trim: --paths option to sort by pathname

  • Mar 25 #2860: rm: show user bytes deleted

  • Mar 22 #2858: selftest: performance in Paths II w/huge pathids

  • Mar 22 #2857: trim: can’t keep kid if parent deleted

  • Mar 22 #2856: export: delete key.conf.0

  • Mar 22 #2855: db upgrade: set arcmb, use MB not MiB

  • Mar 22 #2854: trim: save keeps in db for future trims

  • Mar 21 #2853: trim: line numbers, l and / don’t unstack

  • Mar 21 #2852: trim: <, >, digits

  • Mar 21 #2851: trim: l (el) list directory

  • Mar 21 #2850: trim: K keeps whole directories

  • Mar 21 #2849: trim: skip kids if parents deleted

  • Mar 20 #2848: backup: large file tracking, db upgrade changes

  • Mar 20 #2847: versions: simplify code

  • Mar 20 #2846: get: atexit traceback after restore. Thanks David!

  • Mar 19 #2845: trim: check files > 1MB to improve speed

  • Mar 19 #2843: trim: slashes on directories

  • Mar 18 #2842: trim: more -i options, fixed -s0

  • Mar 18 #2841: trim: flush stdout before changing tty mode

  • Mar 18 #2840: trim: can’t delete / with -i

  • Mar 18 #2839: backup: checked / saved progress off by 1

  • Mar 18 #2838: trim: more -i options

  • Mar 17 #2837: stats: bug in db library gave negative stats w/10GB db

  • Mar 16 #2834: export: close db before delete to avoid NFS errors

  • Mar 16 #2830: trim: suspend on ctrl-z

  • Mar 15 #2829: backup: round mtimes for comparisons

  • Mar 15 #2828: Bugz: map pathnames to numbers for privacy

  • Mar 14 #2827: trim: -i option added for interactive removal

  • Mar 14 #2826: trim: -d option removed; always shows directories

  • Mar 14 #2825: backup: error count 1 off if destinations halted

  • Mar 14 #2824: backup: show large files and recent errors

  • Mar 10 #2823: backup: --maxwait off disables destinations

  • Mar 9 #2822: log: show from & to dates in log summary

  • Mar 8 #2821: don’t pickle command for audit log

  • Mar 8 #2820: db upgrade changes

  • Mar 7 #2818: backup: save char block devices, tb in errors

  • Mar 7 #2817: yesno: delay writing to /dev/tty until stdout flushes

  • Mar 7 #2816: selftest: print block progress by space used

  • Mar 5 #2814: new base17 dbid

  • Mar 5 #2813: rm: always repack cblocks with --secure

  • Mar 3 #2805: trim: new command to find large files in backup

  • Mar 3 #2804: log: always disable stdout buffering for timestamps

  • Mar 3 #2803: tune: gave error if last backup was interrupted

  • Mar 2 #2802: backup: remove last version if nothing saved

  • Mar 2 #2801: recover is now logged

  • Mar 2 #2800: recover: be friendly if no destination on command line

  • Mar 2 #2799: backup: print warning if more than 4 billion blocks

  • Mar 1 #2798: backup: don’t crash if more than 4 billion blocks

  • Mar 1 #2797: Bugz: include end of log with tracebacks

  • Mar 1 #2796: db: clear journal before closing db for upgrade

  • Mar 1 #2795: versions: write errors to stderr not stdout

  • Mar 1 #2794: recover: static build requires sys.exit(), not exit()

  • Mar 1 #2793: improve readkey on message, shared import loop fix

  • Mar 1 #2792: db upgrade changes

  • Feb 28 #2791: traceback if logs summarized while program running

  • Feb 28 #2790: hb -c w/o backupdir should error, not traceback

  • Feb 28 #2789: handle stdout redirected on yes/no question

  • Feb 26 #2788: tune: divide by zero if no data saved

  • Feb 26 #2787: versions: better output, -l obsolete

  • Feb 26 #2786: compare: round mtimes

  • Feb 26 #2784: DATABASE UPGRADE

  • Feb 26 #2783: review readkey help

  • Feb 26 #2782: prep rekey for db upgrade

  • Feb 26 #2781: prep export for db upgrade

  • Feb 26 #2780: don’t try to clear journals on every open

  • Feb 22 #2779: traceback if database locked

  • Feb 22 #2775: add sparse support for APFS

  • Feb 22 #2774: default dedup is 100MB

  • Feb 22 #2773: shard exit code incorrect

#2772 Feb 21, 2022

  • Feb 21 #2772: official release

  • Feb 21 #2772: security issue, disclosure in 90 days

  • Feb 21 #2771: stats: split block and ref db space

  • Feb 21 #2770: better error for -c

  • Feb 21 #2769: handle halted destinations better

  • Feb 20 #2766: log when -c isn’t used

  • Feb 20 #2765: show source of backup directory

  • Feb 17 #2764: mount: make intermediate directories

  • Feb 17 #2762: add --iso option to selftest for tune testing

  • Feb 17 #2760: better permission error messages

  • Feb 17 #2759: compare traceback on empty directories

#2758 Feb 16, 2022

  • Feb 16 #2758: official release

  • Feb 16 #2756: symlink bug in Dir destination

  • Feb 16 #2754: remove progress bars from log files

  • Feb 15 #2752: cleanup tune arguments

  • Feb 15 #2751: log changes disabled all progress bars

  • Feb 15 #2744: add tune command w/progress

  • Feb 14 #2740: remove tune db precheck

  • Feb 14 #2739: remove unused code from tune

  • Feb 14 #2738: update release notes for #2735

  • Feb 13 #2737: rm: error message if path not found with -r

  • Feb 13 #2735: remove feature to remove entire versions

  • Feb 13 #2734: double audit logging

  • Feb 12 #2731: create log directory in init, log permission issues

  • Feb 11 #2730: rm: KeyError traceback

  • Feb 11 #2729: rm: remove db commit

  • Feb 11 #2725: separate preview notes

  • Feb 11 #2724: sort log summaries

  • Feb 11 #2723: log log → log: don’t log the log command even if users request it, to avoid a traceback when finished

#2722 Feb 10, 2022

  • Feb 10 #2722: released

  • Feb 10 #2720: log -s option is obsolete

  • Feb 9 #2716: add instructions for upgrading old dbrevs

  • Feb 9 #2715: remove hidden command handling (not used)

  • Feb 9 #2714: print copyright heading in parent, log in child

  • Feb 9 #2713: copyright heading missing for some commands

  • Feb 9 #2712: write /tmp cafile if needed for SSL w/o -c

  • Feb 9 #2711: remove support for dbrev < 32 (Feb 16, 2019)

  • Feb 9 #2710: log: pass back real exit code

  • Feb 8 #2704+: not so much fun with version control

  • Feb 6 #2703: selftest --inc underflow

  • Feb 2 #2702: hb log -s didn’t work

  • Feb 2 #2701: fix "Is a directory" in rmx #2683. Thanks Ware!

  • Feb 2 #2700: log every command

  • Feb 1 #2694: log perf, log clear, log command

  • Jan 31 #2692: ls -1 w/o -r shows last version

  • Jan 31 #2690: rm: incorrect accounting when removing blocks

  • Jan 31 #2689: selftest: don’t delete bad block w/o --fix

  • Jan 28 #2688: auto run selftest --fix for hash mismatch

  • Jan 27 #2682: minor formatting changes for stats --xmeta

  • Jan 26 #2681: fix dbstat build error on non-OSX. Thanks Evert!

  • Jan 25 #2680: release failed arc download from cache

  • Jan 22 #2679: NFS spans.tmp bug. Thanks Paul!

  • Jan 20 #2678: log -x0 and -x1