Commands‎ > ‎

Count

Scan one or more directory trees and show statistics including file counts by file type, by size, the top N files and directories by size, how the files would be divided into shards, and how file sizes are distributed.

    $ hb count pathname ... [-d] [-n] [-tN] [--sample p] [--shards N or M-N] [-X]

At least one pathname is required.  Other options:

-d    shows a file size distribution table
-n    disables reading file sizes and is 2-3x faster
--shards N    shows how files would be divided into N shards, by count and size
--shards M-N  shows how files would be sharded for all values M through N
--random p    random sample a percentage p of files (by file count, not by size)
--sample p    sample (like backup) a percentage p of files (by file count, not by size)
-tN   shows the top N files and directories by size
-X    cross filesystems

The --sample option is not a true random sample and will give the same results every time.  It is designed to be compatible with backup's --sample option, which samples in this fixed way to support incremental backup testing.  Sampling is based on a hash of the filename so is not always accurate, especially if many files have the same name.  For random sampling, using the --random option.

Example output from an OSX (Mac) /Applications directory:

# hb count /Applications -d -t10 --shard 2-4
HashBackup build #2295 Copyright 2009-2019 HashBackup, LLC

Paths: 214796
Avg path len: 100
Max path len: 249
Max pathname: /Applications/Adobe Photoshop Elements 11/Support Files/Adobe Photoshop Elements Editor.app/Contents/Required/LogTransportLaunch.app/Contents/Frameworks/LogTransport.framework/Versions/A/Support/LogTransport.app/Contents/_CodeSignature/CodeResources

Types:
  dir      |  51542 |          
  file     | 161483 |     5 GiB
  symlink  |   1771 |    38 KiB

Shard counts:
  2: 81434 49% | 81820 50%
  3: 54351 33% | 54610 33% | 54293 33%
  4: 40866 25% | 40981 25% | 40736 24% | 40671 24%

Shard sizes:
  2: 3.0 GB 52% | 2.6 GB 47%
  3: 1.9 GB 34% | 1.8 GB 33% | 1.8 GB 32%
  4: 1.1 GB 19% | 1.3 GB 23% | 1.4 GB 24% | 1.8 GB 32%

Top 10 files:
   210 MB /Applications/ABBYY FineReader for ScanSnap 4.0/FineReader for ScanSnap Preferences.app/Contents/Frameworks/libFREngine.dylib
   185 MB /Applications/VMware Fusion.app/Contents/Library/isoimages/antivirus.iso
   138 MB /Applications/iPhoto.app/Contents/Resources/PointOfInterest.db
   127 MB /Applications/Firefox.app/Contents/MacOS/XUL
    97 MB /Applications/Mnemosyne.app/Contents/MacOS/libopenblasp-r0.3.0.dev.dylib
    96 MB /Applications/Mnemosyne.app/Contents/MacOS/QtWebEngineCore
    88 MB /Applications/VMware Fusion.app/Contents/Resources/naos-1.0.vmwarevm/naos.vmdk
    65 MB /Applications/Adobe Photoshop Elements 11/Support Files/Adobe Photoshop Elements Editor.app/Contents/MacOS/Adobe Photoshop Elements Editor
    63 MB /Applications/VMware Fusion.app/Contents/Library/isoimages/windows.iso
    62 MB /Applications/VMware Fusion.app/Contents/Library/isoimages/linux.iso

Top 10 dirs:
   368 MB /Applications/VMware Fusion.app/Contents/Library/isoimages
   299 MB /Applications/Mnemosyne.app/Contents/MacOS
   210 MB /Applications/iPhoto.app/Contents/Resources
   210 MB /Applications/ABBYY FineReader for ScanSnap 4.0/FineReader for ScanSnap Preferences.app/Contents/Frameworks
   140 MB /Applications/ABBYY FineReader for ScanSnap 4.0/FineReader for ScanSnap Preferences.app/Contents/fre
   140 MB /Applications/ABBYY FineReader for ScanSnap 4.0/FineReader for ScanSnap Preferences.app/Contents/freppc
   136 MB /Applications/Firefox.app/Contents/MacOS
    94 MB /Applications/VMware Fusion.app/Contents/Resources/naos-1.0.vmwarevm
    86 MB /Applications/VMware Fusion.app/Contents/Library
    67 MB /Applications/VMware Fusion.app/Contents/Frameworks

File size distribution:
Cumm Count |  Pct | FILE COUNT |~Cumm Count | ~Pct | FILE SIZE | Cumm Space |  Pct | SPACE USED |~Cumm Space | ~Pct
-----------+------+------------+------------+------+-----------+------------+------+------------+------------+-----
     35682 |  21% |      35682 |     163254 | 100% |    <1 KiB |     13 MiB |   0% |     13 MiB |      5 GiB | 100%
     51666 |  31% |      15984 |     127572 |  78% |     1 KiB |     36 MiB |   0% |     22 MiB |      5 GiB |  99%
     81377 |  49% |      29711 |     111588 |  68% |     2 KiB |    126 MiB |   2% |     90 MiB |      5 GiB |  99%
    122176 |  74% |      40799 |      81877 |  50% |     4 KiB |    348 MiB |   6% |    221 MiB |      5 GiB |  97%
    141618 |  86% |      19442 |      41078 |  25% |     8 KiB |    563 MiB |  10% |    215 MiB |      4 GiB |  93%
    150924 |  92% |       9306 |      21636 |  13% |    16 KiB |    769 MiB |  14% |    205 MiB |      4 GiB |  89%
    157135 |  96% |       6211 |      12330 |   7% |    32 KiB |      1 GiB |  19% |    269 MiB |      4 GiB |  85%
    159976 |  97% |       2841 |       6119 |   3% |    64 KiB |      1 GiB |  23% |    242 MiB |      4 GiB |  80%
    161574 |  98% |       1598 |       3278 |   2% |   128 KiB |      1 GiB |  28% |    274 MiB |      4 GiB |  76%
    162333 |  99% |        759 |       1680 |   1% |   256 KiB |      1 GiB |  33% |    266 MiB |      3 GiB |  71%
    162736 |  99% |        403 |        921 |   0% |   512 KiB |      2 GiB |  38% |    274 MiB |      3 GiB |  66%
    162967 |  99% |        231 |        518 |   0% |     1 MiB |      2 GiB |  44% |    324 MiB |      3 GiB |  61%
    163088 |  99% |        121 |        287 |   0% |     2 MiB |      2 GiB |  51% |    346 MiB |      2 GiB |  55%
    163176 |  99% |         88 |        166 |   0% |     4 MiB |      3 GiB |  60% |    497 MiB |      2 GiB |  48%
    163223 |  99% |         47 |         78 |   0% |     8 MiB |      3 GiB |  70% |    527 MiB |      2 GiB |  39%
    163241 |  99% |         18 |         31 |   0% |    16 MiB |      4 GiB |  77% |    389 MiB |      1 GiB |  29%
    163247 |  99% |          6 |         13 |   0% |    32 MiB |      4 GiB |  83% |    301 MiB |      1 GiB |  22%
    163251 |  99% |          4 |          7 |   0% |    64 MiB |      4 GiB |  90% |    391 MiB |    900 MiB |  16%
    163254 | 100% |          3 |          3 |   0% |   128 MiB |      5 GiB | 100% |    508 MiB |    508 MiB |   9%

214796 names in 22s, 9747 names/s

The file size distribution table is easier to understand by focusing on the 3 main columns:
  • FILE SIZE is a range of file sizes, from this size to the next size
  • FILE COUNT is the number of files that are in this size range
  • SPACE USED is the total bytes used by all files in this size range
To the left and right of FILE COUNT and SPACE USED are cumulative counts and percentages.  The left set are ascending, the right are descending.  Some observations about /Applications from this table:
  • from line 1 col 2, 21% of files are less than 1K; together they occupy around 13MiB
  • from line 14 col 4, only 166 files are 4MB or larger; from the last column, they occupy 48% of /Applications
  • line 4 shows there are 122,176 files less than 8K, this is 74% of the total file count, but only 6% of the space, at 348 MiB
  • the last line shows there are 3 files over 128 MiB, they use 508 MiB, and they use 9% of the space in /Applications
Comments