Benchmarks

These benchmarks are designed to give a feel for how HashBackup performs in different situations.  Many of these tests are not "real world" tests, but are designed to be easily compared with other backup software to test limits, memory usage, disk usage, and/or raw performance.

Backup 1M zero-length files in a single directory

Summary: 3m 14s to backup 1M zero-length files, using 113MB of RAM and creating a backup database of around 118MB.

[jim@ms ~]$ /usr/bin/time -l hb log backup -c hb bigdir -v0

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb

Backup start: 2019-12-25 13:51:01

Copied HB program to /Users/jim/hb/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 194.3s, 3m 14s

CPU:  183.4s, 3m 3s, 94%

Mem:  113 MB

Checked: 1000006 paths, 522 bytes, 522 bytes

Saved: 1000006 paths, 522 bytes, 522 bytes

Excluded: 0

Dupbytes: 0

Compression: 47%, 1.9:1

Efficiency: 0.00 MB reduced/cpusec

Space: +272 bytes, 147 KB total

No errors

      194.60 real       117.94 user        65.97 sys

 113348608  maximum resident set size

         0  average shared memory size

         0  average unshared data size

         0  average unshared stack size

     67448  page reclaims

         0  page faults

         0  swaps

     69712  block input operations

        53  block output operations

         0  messages sent

         0  messages received

         0  signals received

     69823  voluntary context switches

       540  involuntary context switches

[jim@ms ~]$ ls -l hb

total 270584

-rw-r--r--  1 jim  staff        320 Dec 25 13:54 arc.0.0

-rw-r--r--  1 jim  staff     282097 Dec 25 13:51 cacerts.crt

-rw-r--r--  1 jim  staff      36864 Dec 25 13:51 dest.db

-rw-r--r--  1 jim  staff    3146512 Dec 25 13:54 hash.db

-rwxr-xr-x  1 jim  staff   17168072 Dec 25 13:50 hb#2461

-rw-r--r--  1 jim  staff  117882880 Dec 25 13:54 hb.db

-rw-r--r--  1 jim  staff          6 Dec 25 13:51 hb.lock

-rw-r--r--  1 jim  staff        522 Dec 25 13:49 inex.conf

-r--------  1 jim  staff        338 Dec 25 13:49 key.conf

drwxr-xr-x  3 jim  staff        102 Dec 25 13:54 logs


Backup 1M zero-length files in one directory using 8 shards

This is similar to the previous test, but uses shards to run 8 parallel backups.  This system is a 4-core (8 hyperthread) Intel i7 Mac Mini.  The -p0 backup option is used to disable threading in each backup process since this is handled via sharding.  Sharding is transparent in HashBackup after it is enabled with the init command.

HashBackup evenly divides the 1M files in a large directory between 8 backup processes.  This division of work is stable, even when files are added or deleted, so that incremental backups work as expected.
 
Summary: 56 seconds to save 1M empty files using 8 parallel backup processes.  Each process used around 80MB of RAM, or 640MB altogether.  Each backup database is around 15MB, or 120MB altogether - similar to the non-sharded backup.

A test with 4 shards took 69 seconds, with each process using 104MB of RAM, or 416MB altogether.

[jim@ms ~]$ hb init -c hb --shards 8

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb

Permissions set for owner access only

Created key file /Users/jim/hb/key.conf

Key file set to read-only

Setting include/exclude defaults: /Users/jim/hb/inex.conf

Initializing shards

Shards 1-8 initialized


VERY IMPORTANT: your backup is encrypted and can only be accessed with

the encryption key, stored in the file:


    /Users/jim/hb/key.conf


You MUST make copies of this file and store them in secure locations,

separate from your computer and backup data.  If your hard drive fails, 

you will need this key to restore your files.  If you have setup remote

destinations in dest.conf, that file should be copied too.

        

Backup directory initialized

[jim@ms ~]$ /usr/bin/time -l hb log backup -c hb bigdir -v0 -p0

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC


Shard output: /Users/jim/hb/sout/backup.1

Start shard #1: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s1

Start shard #2: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s2

Start shard #3: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s3

Start shard #4: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s4

Start shard #5: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s5

Start shard #6: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s6

Start shard #7: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s7

Start shard #8: hb backup bigdir -v0 -p0 -c /Users/jim/hb/s8


--------

Shard #1

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s1

Shard #1, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s1/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 55.1s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 124816 paths, 0 bytes, 0 bytes

Saved: 124816 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #1 successful


--------

Shard #2

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s2

Shard #2, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s2/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 55.1s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 124829 paths, 0 bytes, 0 bytes

Saved: 124829 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #2 successful


--------

Shard #3

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s3

Shard #3, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s3/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 54.6s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 124764 paths, 0 bytes, 0 bytes

Saved: 124764 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #3 successful


--------

Shard #4

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s4

Shard #4, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s4/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 55.1s

CPU:  46.0s, 83%

Mem:  80 MB

Checked: 125700 paths, 0 bytes, 0 bytes

Saved: 125700 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #4 successful


--------

Shard #5

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s5

Shard #5, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s5/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 55.0s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 125135 paths, 0 bytes, 0 bytes

Saved: 125135 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #5 successful


--------

Shard #6

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s6

Shard #6, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s6/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 54.7s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 125306 paths, 522 bytes, 522 bytes

Saved: 125306 paths, 522 bytes, 522 bytes

Excluded: 0

Dupbytes: 0

Compression: 47%, 1.9:1

Efficiency: 0.00 MB reduced/cpusec

Space: +272 bytes, 147 KB total

No errors

Shard #6 successful


--------

Shard #7

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s7

Shard #7, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s7/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 55.0s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 124901 paths, 0 bytes, 0 bytes

Saved: 124901 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #7 successful


--------

Shard #8

--------

HashBackup #2461 Copyright 2009-2019 HashBackup, LLC

Backup directory: /Users/jim/hb/s8

Shard #8, 12% of files

Backup start: 2019-12-25 14:19:56

Using destinations in dest.conf

Copied HB program to /Users/jim/hb/s8/hb#2461

This is backup version: 0

Dedup not enabled; use -Dmemsize to enable

Backup started


Time: 55.0s

CPU:  45.8s, 83%

Mem:  80 MB

Checked: 124583 paths, 0 bytes, 0 bytes

Saved: 124583 paths, 0 bytes, 0 bytes

Excluded: 0

No errors

Shard #8 successful


Shard summary for backup: 8 worked, 0 failed


       56.19 real       262.95 user       107.33 sys

  80199680  maximum resident set size

         0  average shared memory size

         0  average unshared data size

         0  average unshared stack size

    313104  page reclaims

         0  page faults

         0  swaps

     70261  block input operations

        40  block output operations

         0  messages sent

         0  messages received

         0  signals received

    245803  voluntary context switches

    922071  involuntary context switches

[jim@ms ~]$ du -ksc hb/*

276 hb/cacerts.crt

4 hb/dest.conf

3076 hb/hash.db

144 hb/hb.db

4 hb/hb.lock

4 hb/inex.conf

4 hb/key.conf

12 hb/logs

34236 hb/s1

34236 hb/s2

34228 hb/s3

34332 hb/s4

34268 hb/s5

34304 hb/s6

34260 hb/s7

34212 hb/s8

32 hb/sout

277632 total

[jim@ms ~]$ ls -l hb/s1

total 68472

lrwxr-xr-x  1 jim  staff        14 Dec 25 14:19 cacerts.crt -> ../cacerts.crt

lrwxr-xr-x  1 jim  staff        12 Dec 25 14:19 dest.conf -> ../dest.conf

-rw-r--r--  1 jim  staff     36864 Dec 25 14:19 dest.db

-rw-r--r--  1 jim  staff   3146512 Dec 25 14:20 hash.db

-rwxr-xr-x  1 jim  staff  17168072 Dec 25 13:50 hb#2461

-rw-r--r--  1 jim  staff  14680064 Dec 25 14:20 hb.db

-rw-r--r--  1 jim  staff         6 Dec 25 14:19 hb.lock

lrwxr-xr-x  1 jim  staff        12 Dec 25 14:19 inex.conf -> ../inex.conf

lrwxr-xr-x  1 jim  staff        11 Dec 25 14:19 key.conf -> ../key.conf


Comments