Preview

The preview release is installed with hb upgrade -p. Using preview releases for non-critical backups helps find and solve problems early. The official release is sent to the preview area so hb upgrade -p gets the official release when it’s available. Thank you for your help!

Preview Notes

  • Apr 23 #3102: rekey: handle interrupted rekey with shards

  • Apr 23 #3101: shards: show failing shard numbers

  • Apr 23 #3100: config: shard-output-days defaults to 7 vs 30

  • Apr 22 #3099: shards: use pid for sout files

  • Apr 22 #3098: export: support shards

  • Apr 20 #3097: rm: better exit code handling

  • Apr 20 #3096: trim: handle -i with shards

  • Apr 18 #3093: get: use default responses with shards

  • Apr 18 #3092: get: not always fatal if shard doesn’t contain path

  • Apr 14 #3091: official release

#3102 Apr 23, 2024

Summary:

  • get: use default responses with shards

  • get: not always fatal if shard doesn’t contain path

  • trim: handle -i with shards

  • rm: better exit code handling

  • export: support shards

  • config: shard-output-days defaults to 7 vs 30

  • rekey: handle interrupted rekey with shards

Details:

  • get: questions are sometimes asked, such as "File exists, okay to overwrite?", and these usually have default responses. With shards, the default responses were not being accepted and a shard would sometimes go into an infinite loop asking the question. This could be seen in the sout files, though on the terminal screen, it appeared that the restore was hung.

  • get: shards could exit with a fatal error if they didn’t contain a requested pathname, but sometimes this is expected and not an error. When restoring a single file, the exit code would be 1 even though the file was correctly restored. When multiple file pathnames in different shards were on the command line, not all files were restored because some shards exited with a "Pathname is not in backup" fatal error.

  • trim: trim works with shards to generate reports if the -i option is not used. The -i option (interactive trim) can be used with individual shards by adding /sN to the -c option. This will interactively trim shard N. The -i option cannot be used to trim multiple shards, and now instead of raising an error 'Inappropriate ioctl for device', the error advises to trim shards individually.

  • rm: when removing multiple paths, the exit status was only for the last path removed, so if the 1st path remove failed and the 2nd worked, the exit status was zero even though there was a failure. Also, it is not necessarily an error if a pathname is not found in a particular shard, so it should not always affect the exit status.

  • export: shards are supported, either by exporting the entire backup, creating an export file for each shard, or by exporting a single shard by adding /sN to export’s -c option, where N is the shard number.

  • config: shard-output-days defaults to 7 (was 30) and instead of using the time the job was started to calculate its age, the last time the job wrote output is used so that a long-running job does not have its output removed prematurely.

  • rekey: an interrupted rekey is automatically rolled back to the previous state on the next command. With shards, a successful rekey after a rollback had an incorrect key file.

History

#3091 Apr 14, 2024

  • Apr 14 #3091: official release

  • Mar 18 #3090: selftest: fix overlapping blocks error message

  • Mar 18 #3088: stats: extended metadata lengths were wrong

  • Feb 21 #3087: hb: DatabaseError: Disk I/O Error exception

  • Feb 16 #3086: backup: debug for Harvard/MIT Disk I/O Error

  • Feb 2 #3084: backup: exclude backup directory with -c symlink

#3083 Jan 12, 2024

  • Jan 12 2024 #3083: official release

  • Nov 30 #3082: get: restore fails on single deleted file/directory

  • Nov 4 #3081: log: no error on file remove if already gone (race)

  • Oct 31 #3080: avoid recursive selftest

  • Oct 23 #3079: ls: fix sequential scans

  • Oct 21 #3078: ls: fix wildcard bugs, give hint for square bracket error in pattern

#3077 Oct 14, 2023

  • Oct 14 #3077: official release

  • Oct 14 #3076: compare: round ctime before doing comparisons

#3075 Sep 1, 2023

  • Sep 1 #3075: official release

  • Aug 17 #3073: dest: suggest verify if downloaded file size is wrong

  • Jul 16 #3072: mount: some mount errors gave "global name 'strerr' is not defined" instead of the actual error

  • Jul 15 #3071: rm: interrupted packing could cause selftest errors

  • Jul 15 #3070: rm: don’t reuse arc filenames when packing

  • Jul 14 #3069: rm: interrupted --secure could cause selftest errors

#3068 Jul 14, 2023

  • Jul 14 #3068: official release

  • Jul 13 #3068: bump expiration date

#3067 Apr 9, 2023

  • Apr 9 #3067: official release

  • Apr 9 #3066: bump expiration date

  • Mar 6 #3065: sync: avoid overfilling cache

  • Feb 25 #3063: rm: traceback removing dangling pathname

  • Feb 15 #3062: error if rename to subdirectory

  • Jan 30 #3061: pack-percent-free > 1% instead of 10%

  • Jan 20 #3059: create large .zip files for huge logs

  • Jan 11 #3058: official release

#3058 Jan 11, 2023

  • Jan 11 #3058: official release

  • Jan 11 #3058: prevent and fix selftest error: blockids overlap

  • Jan 11 #3057: bump expiration date

  • Jan 5 #3056: inex.conf: better error on bad wildcard

  • Dec 14 #3055: log: summarize w/o log dir in RO backup dir

  • Dec 1 #3054: trim: double db close causing hang

  • Nov 27 #3053: trim: always add excludes to inex.conf

  • Nov 27 #3053: trim: traceback if parent and subdir deleted

  • Nov 10 #3052: ls: strip pathname pattern trailing slashes

  • Nov 1 #3051: experiment: preload database for get

#3050 Sep 9, 2022

  • Sep 9 #3050: official release

  • Sep 9 #3049: get: remove experimental inex exclusion

  • Sep 7 #3048: Bugz: strip pathnames if background thread fails

  • Sep 5 #3047: mount: revert back to #3040; couldn’t fix FreeBSD FUSE

  • Sep 5 #3047: get: traceback during restore

  • Sep 5 #3046: backup, get, compare: round mtime more aggressively

  • Aug 31 #3042: mount: adjust for FreeBSD 12+

  • Aug 25 #3040: get: honor inex.conf excludes for restore

#3039 Aug 17, 2022

  • Aug 17 #3039: official release: no attribute 'SIGINFO' on Linux

#3038 Aug 14, 2022

  • Aug 14 #3038: official release

  • Aug 7 #3037: backup: Exception: Bug: varblock in mixed threads

#3036 Jul 29, 2022

  • Jul 29 #3036: official release

  • Jul 24 #3035: log: forward siginfo signal

  • Jul 21 #3033: recover: arc file download rename race condition

  • Jul 14 #3032: official release

#3032 Jul 14, 2022

  • Jul 14 #3032: official release

  • Jul 14 #3032: db upgrade: remove .uporig if dbrev is correct, check disk space before upgrade

  • Jul 14 #3031: dest needput: revise conditions

  • Jul 8 #3030: selftest: enable dev testing of -v0 and -v1

  • Jul 4 #3029: remove leading slash from inline tracebacks

  • Jul 3 #3028: db upgrade: remove cached arc files to make space (held)

  • Jul 3 #3027: backup: reorder mtime tests

  • Jul 2 #3026: backup: cache was not always getting cleaned up

  • Jul 2 #3025: backup, get: handle int resolutions on mtimes

  • Jul 1 #3024: dest noparts: skip files with null info

  • Jun 30 #3023: selftest: only fix dedup table with --fix

  • Jun 27 #3022: get: rtrim → rstrip. Thanks Bugz!

  • Jun 27 #3021: recover: no such table: destfiles. Thanks Bugz!

  • Jun 25 #3020: dest.conf: better handling of keywords with missing values

  • Jun 25 #3019: dest: re-wrap verify help to match other subcommands

  • Jun 22 #3018: log: handle no such process in sig handler. Thanks Bugz!

  • Jun 22 #3017: handle -r with missing/empty versions better. Thanks Bugz!

  • Jun 21 #3016: backup: no partial backup warning for char devices

  • Jun 18 #3015: official release

#3015 Jun 18, 2022

  • Jun 18 #3015: official release

  • Jun 16 #3012: get: traceback restoring deleted files

  • Jun 16 #3011: backup: distinguish interupts from timeouts

  • Jun 12 #3010: send ctrl-c and sigterm to commands

  • Jun 12 #3009: dest nopart: exit on ctrl-c and kill -term

  • Jun 7 #3007: destinations: ending support for maxsize

  • Jun 6 #3006: cleanup mount change in #3005

  • Jun 5 #3005: mount: reading variable-block files could crash mount

  • Jun 3 #3004: backup, mount: minor message changes

  • Jun 2 #3003: destcmd: don’t require setid after dest rename. Thanks Preston!

  • Jun 2 #3002: mount: remove mount timeout for slow destinations

  • Jun 2 #3001: mount: use mkdirp to create --cache directory. Thanks Bugz!

  • May 31 #2999: mount: remove a debug stack trace

  • May 31 #2998: recover: create testonly.conf file

  • May 30 #2997: compress inline traceback a bit more

  • May 30 #2996: dest.conf: integer keyword defaults

  • May 30 #2995: auto-run selftest in several more situations

  • May 30 #2994: mount improvements. Thanks Preston!

  • May 25 #2993: clear: no "if you continue" with --force

  • May 25 #2992: remove Rackspace Cloudfiles remnants

  • May 25 #2991: log: traceback if program completes during summarizing. Thanks Bugz!

  • May 24 #2990: security disclosure from 2022-02-21 published

  • May 24 #2990: handle rotating USB drives better

  • May 23 #2989: get: restore pathnames can start with ../

  • May 22 #2988: selftest, rm: change cache initialization

  • May 19 #2985: dir dest: was requiring too much free disk space

  • May 19 #2984: generalize reading passphrases

  • May 19 #2983: selftest: --fix corrects a few more errors

  • May 18 #2982: trim: flush stderr so 1st pathname isn’t cleared

  • May 18 #2981: trim: traceback "getting called too often"

  • May 18 #2979: log: prevent status line escape sequences in logs

  • May 17 #2978: dest rename: new command

#2976 May 17, 2022

  • May 17 #2976: official release

  • May 17 #2975: disabled destinations being selected

  • May 16 #2973: recover: use multiple destinations if necessary

  • May 15 #2970: clear progress reports on stderr write

  • May 9 #2969: versions: File Space was sometimes incorrectly blank

  • May 8 #2968: backup: traceback when rebuilding dedup table

  • May 7 #2967: selftest: upload hb.db if it was changed

  • May 7 #2966: ls: show date deleted

  • May 7 #2965: retain: use new status line code

  • May 6 #2964: db upgrade: error vs traceback for permission failures

  • May 5 #2963: hb: auto selftest may need to run 4 times, not 3

  • May 4 #2962: get: handle some fringe error situations

  • May 2 #2958: dest: remove sync after verify is complete

  • May 1 #2957: recover: show better stats during download

  • May 1 #2956: upgrade: add --- End release notes ---

  • May 1 #2955: hb: auto run selftest --fix for missing arc files

  • May 1 #2954: rm/retain: if download fails try other destinations

  • May 1 #2953: hb: bug fix for #2949: Unable to download archive arc.0.0: AttributeError("'module' object has no attribute 'report'"

  • Apr 30 #2952: log: if log disappears don’t traceback on rename

  • Apr 30 #2951: hb: forward SIGTERM so backup sees it

  • Apr 29 #2950: selftest: remove read count difference notice

  • Apr 29 #2949: add a couple more wait status lines

  • Apr 29 #2948: backup: better wait time stats

  • Apr 29 #2947: backup: show status msg while waiting for cache space

  • Apr 28 #2946: backup: set timelimit flag on SIGTERM to stop sooner

  • Apr 28 #2945: backup: moved and enabled signal (SIGTERM) handler

  • Apr 28 #2943: backup: don’t set checkpoint to last file

  • Apr 28 #2942: backup: maxtime interrupt on large files

  • Apr 27 #2941: backup: show maxwait stop goal

  • Apr 27 #2940: backup: delay maxwait after maxtime expires

  • Apr 26 #2939: use statline() and progrep() for status lines

  • Apr 26 #2937: get: early termination could hang with limited cache

  • Apr 25 #2936: selftest: more uniform -v5 progress reports

  • Apr 25 #2935: selftest: --sample fix, status line

  • Apr 25 #2934: misc.statline() and progrep()

  • Apr 24 #2933: Bugz: don’t send tracebacks in development

  • Apr 24 #2932: dest: setid arg not required if only 1 destination

  • Apr 24 #2931: set DESTID carefully to avoid a setid in edge cases

  • Apr 24 #2930: hb: avoid traceback if db can’t be opened

  • Apr 23 #2929: backup: further reduce --maxtime overruns

  • Apr 22 #2927: backup: close/unlock db during final sync

  • Apr 22 #2926: config: doesn’t need to lock the backup directory

  • Apr 22 #2924: rm: don’t run sync to delete arc files

  • Apr 21 #2923: misc.timeinterval() accepts multiple time periods

  • Apr 21 #2922: misc.holdquit() signal handler

  • Apr 21 #2921: misc.intty(), misc.outtty()

  • Apr 21 #2920: log: -X didn’t always exclude lines

  • Apr 17 #2919: backup: --maxtime/--maxwait more accurate

  • Apr 17 #2918: handle negative time intervals in show.vistime()

  • Apr 17 #2917: backup: poll destinations while waiting for cache

  • Apr 16 #2916: init: avoid traceback if -c is a file. Thanks Bugz!

  • Apr 16 #2915: hb: disable stderr redirection for automated tests

  • Apr 15 #2914: recover: HMAC signature verification failed

  • Apr 14 #2913: config: conditionally display enable/disable-commands note

  • Apr 14 #2912: config: flush stdout before reading admin-passphrase to avoid confusing output; handle EOF (ctrl-d) when entering admin-passphrase to avoid traceback - thanks Bugz!

  • Apr 14 #2911: don’t send tracebacks from automated tests

  • Apr 13 #2910: hb: don’t update exit code in tests (sync test)

#2909 Apr 12, 2022

  • Apr 12 #2909: official release

  • Apr 12 #2909: config: traceback on invalid or read-only key

  • Apr 12 #2908: hb command --help shows command help. Thanks Arthur!

  • Apr 11 #2907: close db after integrity check

  • Apr 8 #2905: hb, selftest: try to correct malformed database

  • Apr 8 #2903: trim: show message when -i isn’t used (maybe forgot it)

  • Apr 7 #2902: trim: traceback with -s

  • Apr 7 #2901: trim: % character causing help traceback

  • Apr 6 #2900: Bugz: don’t skip line if at beginning of log

  • Apr 6 #2899: Bugz: handle extremely long traceback (internal testing)

  • Apr 6 #2898: Bugz: flush stdout and delay to get end of traceback

#2897 Apr 6, 2022

  • Apr 6 #2897: official release

  • Apr 6 #2897: dest: cPickle.loads() type mismatch. Thanks Ware!

  • Apr 6 #2896: trim: -n always a percentage, better time left estimate

  • Apr 5 #2895: trim: -n % limit was getting applied twice

  • Apr 5 #2894: trim: limit versions with -n too

  • Apr 5 #2893: trim: -n defaults to top 50% of backup

  • Apr 5 #2891: log: ctrl-c does not log as _FAIL or _RUN

  • Apr 5 #2890: trim: check backup ownership for keep & delete

  • Apr 4 #2889: trim: reorganized code, dash should have been r for remove, reinstate -n w/default 1000 for -i, 100000 otherwise, add g command to go to (search for) pathname, add i command to show file details

  • Apr 4 #2886: trim: remove blank line when paging with space bar

  • Apr 4 #2885: trim: change --paths option to -p

  • Apr 4 #2884: trim: add note in show deletes about smaller files

  • Apr 3 #2883: trim: unique data too low, needs float division

  • Apr 3 #2882: trim: more accurate (but slower) scan for unique data, show time left

#2881 Apr 3, 2022

  • Apr 3 #2881: official release

  • Apr 3 #2881: db upgrade: fix AttributeError: 'NoneType' object has no attribute 'items'

  • Apr 2 #2877: ssh: handle dd unavailable on remote

#2875 Apr 1, 2022

  • Apr 1 #2875: official release

  • Mar 31 #2874: selftest: fix %% in progress string; upgrade: show current version

  • Mar 31 #2873: tune: tune paths, fix and re-enable block commits, speed up tune with many arc files

  • Mar 30 #2872: tune: disable block commits: interrupt testing found a bug

  • Mar 30 #2871: compare: not showing all changes, mostly deletes

  • Mar 29 #2870: trim: space key, fix suspend

  • Mar 29 #2869: trim: typo in #2861 added duplicate entries to trim list

  • Mar 28 #2868: export: workaround NFS creating .nfs files for large databases

  • Mar 28 #2866: backup: was not deleting old hb.db.N files

  • Mar 27 #2865: backup: sync remote-to-remote after backup vs at the start

  • Mar 26 #2864: tune: don’t delete version records; versions suppresses them

  • Mar 25 #2863: db upgrade: tune failed on old backups

  • Mar 25 #2862: arcmb didn’t work out, removed it

  • Mar 25 #2861: trim: --paths option to sort by pathname

  • Mar 25 #2860: rm: show user bytes deleted

  • Mar 22 #2858: selftest: performance in Paths II w/huge pathids

  • Mar 22 #2857: trim: can’t keep kid if parent deleted

  • Mar 22 #2856: export: delete key.conf.0

  • Mar 22 #2855: db upgrade: set arcmb, use MB not MiB

  • Mar 22 #2854: trim: save keeps in db for future trims

  • Mar 21 #2853: trim: line numbers, l and / don’t unstack

  • Mar 21 #2852: trim: <, >, digits

  • Mar 21 #2851: trim: l (el) list directory

  • Mar 21 #2850: trim: K keeps whole directories

  • Mar 21 #2849: trim: skip kids if parents deleted

  • Mar 20 #2848: backup: large file tracking, db upgrade changes

  • Mar 20 #2847: versions: simplify code

  • Mar 20 #2846: get: atexit traceback after restore. Thanks David!

  • Mar 19 #2845: trim: check files > 1MB to improve speed

  • Mar 19 #2843: trim: slashes on directories

  • Mar 18 #2842: trim: more -i options, fixed -s0

  • Mar 18 #2841: trim: flush stdout before changing tty mode

  • Mar 18 #2840: trim: can’t delete / with -i

  • Mar 18 #2839: backup: checked / saved progress off by 1

  • Mar 18 #2838: trim: more -i options

  • Mar 17 #2837: stats: bug in db library gave negative stats w/10GB db

  • Mar 16 #2834: export: close db before delete to avoid NFS errors

  • Mar 16 #2830: trim: suspend on ctrl-z

  • Mar 15 #2829: backup: round mtimes for comparisons

  • Mar 15 #2828: Bugz: map pathnames to numbers for privacy

  • Mar 14 #2827: trim: -i option added for interactive removal

  • Mar 14 #2826: trim: -d option removed; always shows directories

  • Mar 14 #2825: backup: error count 1 off if destinations halted

  • Mar 14 #2824: backup: show large files and recent errors

  • Mar 10 #2823: backup: --maxwait off disables destinations

  • Mar 9 #2822: log: show from & to dates in log summary

  • Mar 8 #2821: don’t pickle command for audit log

  • Mar 8 #2820: db upgrade changes

  • Mar 7 #2818: backup: save char block devices, tb in errors

  • Mar 7 #2817: yesno: delay writing to /dev/tty until stdout flushes

  • Mar 7 #2816: selftest: print block progress by space used

  • Mar 5 #2814: new base17 dbid

  • Mar 5 #2813: rm: always repack cblocks with --secure

  • Mar 3 #2805: trim: new command to find large files in backup

  • Mar 3 #2804: log: always disable stdout buffering for timestamps

  • Mar 3 #2803: tune: gave error if last backup was interrupted

  • Mar 2 #2802: backup: remove last version if nothing saved

  • Mar 2 #2801: recover is now logged

  • Mar 2 #2800: recover: be friendly if no destination on command line

  • Mar 2 #2799: backup: print warning if more than 4 billion blocks

  • Mar 1 #2798: backup: don’t crash if more than 4 billion blocks

  • Mar 1 #2797: Bugz: include end of log with tracebacks

  • Mar 1 #2796: db: clear journal before closing db for upgrade

  • Mar 1 #2795: versions: write errors to stderr not stdout

  • Mar 1 #2794: recover: static build requires sys.exit(), not exit()

  • Mar 1 #2793: improve readkey on message, shared import loop fix

  • Mar 1 #2792: db upgrade changes

  • Feb 28 #2791: traceback if logs summarized while program running

  • Feb 28 #2790: hb -c w/o backupdir should error, not traceback

  • Feb 28 #2789: handle stdout redirected on yes/no question

  • Feb 26 #2788: tune: divide by zero if no data saved

  • Feb 26 #2787: versions: better output, -l obsolete

  • Feb 26 #2786: compare: round mtimes

  • Feb 26 #2784: DATABASE UPGRADE

  • Feb 26 #2783: review readkey help

  • Feb 26 #2782: prep rekey for db upgrade

  • Feb 26 #2781: prep export for db upgrade

  • Feb 26 #2780: don’t try to clear journals on every open

  • Feb 22 #2779: traceback if database locked

  • Feb 22 #2775: add sparse support for APFS

  • Feb 22 #2774: default dedup is 100MB

  • Feb 22 #2773: shard exit code incorrect

#2772 Feb 21, 2022

  • Feb 21 #2772: official release

  • Feb 21 #2772: security issue, disclosure in 90 days

  • Feb 21 #2771: stats: split block and ref db space

  • Feb 21 #2770: better error for -c

  • Feb 21 #2769: handle halted destinations better

  • Feb 20 #2766: log when -c isn’t used

  • Feb 20 #2765: show source of backup directory

  • Feb 17 #2764: mount: make intermediate directories

  • Feb 17 #2762: add --iso option to selftest for tune testing

  • Feb 17 #2760: better permission error messages

  • Feb 17 #2759: compare traceback on empty directories

#2758 Feb 16, 2022

  • Feb 16 #2758: official release

  • Feb 16 #2756: symlink bug in Dir destination

  • Feb 16 #2754: remove progress bars from log files

  • Feb 15 #2752: cleanup tune arguments

  • Feb 15 #2751: log changes disabled all progress bars

  • Feb 15 #2744: add tune command w/progress

  • Feb 14 #2740: remove tune db precheck

  • Feb 14 #2739: remove unused code from tune

  • Feb 14 #2738: update release notes for #2735

  • Feb 13 #2737: rm: error message if path not found with -r

  • Feb 13 #2735: remove feature to remove entire versions

  • Feb 13 #2734: double audit logging

  • Feb 12 #2731: create log directory in init, log permission issues

  • Feb 11 #2730: rm: KeyError traceback

  • Feb 11 #2729: rm: remove db commit

  • Feb 11 #2725: separate preview notes

  • Feb 11 #2724: sort log summaries

  • Feb 11 #2723: log log → log: don’t log the log command even if users request it, to avoid a traceback when finished

#2722 Feb 10, 2022

  • Feb 10 #2722: released

  • Feb 10 #2720: log -s option is obsolete

  • Feb 9 #2716: add instructions for upgrading old dbrevs

  • Feb 9 #2715: remove hidden command handling (not used)

  • Feb 9 #2714: print copyright heading in parent, log in child

  • Feb 9 #2713: copyright heading missing for some commands

  • Feb 9 #2712: write /tmp cafile if needed for SSL w/o -c

  • Feb 9 #2711: remove support for dbrev < 32 (Feb 16, 2019)

  • Feb 9 #2710: log: pass back real exit code

  • Feb 8 #2704+: not so much fun with version control

  • Feb 6 #2703: selftest --inc underflow

  • Feb 2 #2702: hb log -s didn’t work

  • Feb 2 #2701: fix "Is a directory" in rmx #2683. Thanks Ware!

  • Feb 2 #2700: log every command

  • Feb 1 #2694: log perf, log clear, log command

  • Jan 31 #2692: ls -1 w/o -r shows last version

  • Jan 31 #2690: rm: incorrect accounting when removing blocks

  • Jan 31 #2689: selftest: don’t delete bad block w/o --fix

  • Jan 28 #2688: auto run selftest --fix for hash mismatch

  • Jan 27 #2682: minor formatting changes for stats --xmeta

  • Jan 26 #2681: fix dbstat build error on non-OSX. Thanks Evert!

  • Jan 25 #2680: release failed arc download from cache

  • Jan 22 #2679: NFS spans.tmp bug. Thanks Paul!

  • Jan 20 #2678: log -x0 and -x1