Hello!

Questions / comments / suggestions / bug reports are always welcome and appreciated!

Here is a fairly typical support question and answer session about HashBackup, to give new customers an idea of the kind of support we like to provide.

From: Thomas Neuber
To: info@hashbackup.com
Subject: Question regarding a special backup option --no-inode
Date: Monday, May 13, 2024 11:14 AM

Hi,

I have been using your software - HashBackup - for a very long time. Recently, you abandoned the 32-bit Linux version, that I used on my (very) old Synology NAS. It does not support the 64-bit Linux version. Therefore, I migrated all saved data to a new hardware, which is now 64-bit capable. I want to continue with existing backup history. The inodes of saved files has been changed, permissions and probably some timestamps. Do you recommend to continue the backup with the -no-inode option? I want to backup those files that have been changed since last working backup from the old system, but I want to avoid a full backup.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Monday, May 13, 2024 12:59 PM

Hi Thomas! Could you explain a bit more what you are backing up? It’s not clear to me if you are backing up the files residing on your NAS or if you are using your NAS as storage and backing up files from other computer systems attached via network to your NAS.

Jim

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Monday, May 13, 2024 1:33 PM

Hi Jim! I am backing up files residing on my NAS. I think, the biggest part of it is my photo archive containing approximately 3 TB of photos and videos, but also other documents, Maildir etc. The backup does not contain backup images from other attached computer systems.

Because of the large size and number of files, I want to avoid a new full backup. I have a local copy of that backup and one copy at Backblaze. It would last around 18 month and consumes a lot of additional space, I think.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Monday, May 13, 2024 2:36 PM

Hi Thomas - since you copied data from your old NAS to your new one, some file attributes are probably changed, as you mentioned. The --no-ino option doesn’t help you in this case: it is designed to handle the situation where, on some networked file systems, the inode number changes randomly. This happens because the file server doesn’t have fixed inode numbers and assigns them as files are opened and closed. The --no-ino option ignores these changing inode numbers, but HB will still look at all the other file attributes, and if any of them have changed, it will try to backup the file.

If you have always backed up with dedup enabled, then even though HB will read every file, it will not create much new data in the way of arc files, because most of it will dedup. If you’re not sure you’ve always had dedup enabled on every backup (it used to be disabled by default), you could send me an export of your backup or the output of hb stats and I can probably tell.

Questions:

Let me know and I can give you some advice.

Jim

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Monday, May 13, 2024 3:30 PM

Hi Jim!

#1. Almost all files have the same pathnames.

#2. The dedup memory was configured to 200m. It was set with the config command. I did not use the -D option.

#3. The old system reports with "cat /proc/meminfo"

The new system reports with "cat /proc/meminfo"

#4. The hash.db consumes 201M on disk.

Are you interested in some statistics?

Maybe, some more statistic?

I created the backup job on daily basis with --maxtime 16h. If it takes several days, it would not be a problem. As I understood, I should not use --no-ino. The inodes have been changed once but are stable from now on. It may take same time to catch up and dedup will help.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Monday, May 13, 2024 7:00 PM

Hi Thomas!

On Mon, May 13, 2024, at 3:30 PM, Thomas Neuber wrote:

This is good: HB can dedup better if the pathname is the same

This was a good size for your old system, but according to hb stats, your dedup table is full, and not all blocks are in the dedup table. You have 50M blocks total, but only 17M are in the dedup table.

On your new system, you have a lot more RAM available: 1.2GB is completely free, and 2.4GB is available, in total, if needed. This is because 2.4 - 1.2 = 1.2 GB is being used for file system buffering, also shown as "Cached". You want to have as many blocks as possible in the dedup table, so I would increase dedup-mem to 2GB. When you do that, your next backup should say "Expanding dedup table". To test this, just backup 1 file.

Then I would backup a few of your oldest files / photos, and see how HB does at deduping them. If it seems to do well, do a directory of files. You might want to monitor your NAS during the backup and see if it is paging. If so, 2GB is too big and you might need to decrease it to 1.5GB. Once your small test backups are working well, do your whole backup.

The other thing I will mention is that you probably have block-size set to 32K since that was the default for a long time. The default for new backups is "auto", which means HB will select the block size for each file, and it will most likely be much larger. The advantage of using a larger block size is that you will have fewer blocks. So instead of having 50M blocks in hb.db, you might only have 5-10M. Also, all of the blocks will fit into a smaller dedup table.

HOWEVER: the big disadvantage of changing the block size on an existing backup is that it is like starting over, because you cannot dedup against previous versions if the block size changes. To go this route, I would probably start completely over, and that may not be something you want to do. I’m only mentioning it because it would be more efficient going forward.

If you have any questions, just ask! Let me know if this works for you.

Jim

PS: you are right, --no-ino should not be used. It doesn’t hurt much, but one of the side effects of --no-ino is that hard links are backed up as separate files instead of being saved as hard links. By definition, a hard link is when two files have the same inode number, and if inode numbers are being ignored or randomly generated, it’s not possible to determine hard link relationships.

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 10:12 AM

Hey Thomas, I was thinking today that 2GB is probably a bit too big for dedup-mem. You have ~50M blocks in your backup, and were storing 17M blocks with dedup-mem at 200M. So you need about 600M to store all 50M blocks. If you set dedup-mem to 1GB, you can store all your blocks with room for future growth, and you will have more RAM available for the OS to cache pages from the hb.db database during the backup.

Let me know how it goes!

Jim

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 10:22 AM

Hi Jim, I read your message this morning and came back from work few minutes ago and wanted to start with your suggestions. I will let you know about the results. I am pretty sure, that it will work. As I have learn in last seven or eight years, you created a pretty good piece of software.

Thomas

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 11:49 AM

Hi Jim! I changed the dedup-mem to 1g and left the block size as it was. The first backup run expanded the dedup table. I specified a really old file. It was saved beside the inex.conf. I was not aware of changes in the inex.conf. The specified file has 25 bytes and the inex.conf has 296 bytes. The backup size increased by 288 bytes. The second backup saved a complete directory containing 30 files with 27 mb. The second backup size was 240 bytes. That means, it works perfect.

You did a great job with the tool and thank you for the great support.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 1:19 PM

Hi Thomas! Thanks for letting me know it all worked out okay.

One other thing you might want to think about: when you finish the backup of your new server, it shouldn’t create much new backup data, as you have seen. But it will create a lot of new data in your hb.db database because your file attributes like ctime and mtime were changed during the copy from the old server. It will probably double your database size if every file is backed up again.

For things like photos, I’m guessing there is no history of previous versions that you want to keep (unlike with documents), so having a version 0 of a photo and now having the same file stored in version 2000 (or whatever) is not useful. What you could do is run retain after your new backup is completed, either on specific directories like photos or on the whole backup, and use the -m1 option to say "I only want 1 copy of a pathname in the backup". HB will keep the most recent copy and delete the older copies. If a file is marked as deleted in the backup (it was deleted from your NAS), HB will also keep those; only the -x option controls when they are deleted from the backup.

Example:

Without pathnames, retain will remove the version history for all files in the backup.

Jim

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 1:40 PM

Hi Jim! Thank you for your advice. I use a cycle of backup, retain and test. But keeping only the latest backup, I was not aware of. That sounds really interesting, especially for things like photos. This is something that I should really consider, keeping only one or two copies of certain files.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 2:03 PM

Hey Thomas! After thinking about it some more, I realized I goofed a bit. In your hb.db, 80% is "blocks", 4% is "refs", and 3% is "files". When you backup the new server and it dedups everything, "files" and "refs" might double, but "blocks" will not. So your hb.db should only grow about 7%.

Jim

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Tuesday, May 14, 2024 2:28 PM

Hi Jim! I started already my backup procedure. It has been running for 20 minutes. Following the log output, it looks pretty good. The CPU and memory consumption seems to be moderate.

115M dest.db
1.1G hash.db
3.8G hb.db

A 7% bigger hb.db would not hurt. My original concern was about the amount of new arc files.

I will let you know the result. My backup has maxtime of 16h. Let us see how far we get within 16 hours and/or how many days we will need to get in sync. Last backup was a month ago. A lot of new files have been added for the last month.

Thomas

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Wednesday, May 15, 2024 12:47 AM

Hi Jim! The backup is still running. The process of collecting and backing up changed files seems to be completed, but upload is in progress. The latest revision created 43g arc files. That looks reasonable for the changes during the last month without backup. The deduplication works really good.

Thank you for that great tool.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Wednesday, May 15, 2024 9:56 AM

Hi Thomas! I’m glad the backup of your new NAS went smoothly! Thanks for letting me know!

I was thinking, it might be interesting to publish our email exchange on the HB website, to show potential new HashBackup users an example of customer support. I’d remove your personal information (use just your first name) and would have to reformat things. Would you mind if I tried that? I can let you take a look first before publishing anything.

Thanks! Jim

From: Thomas Neuber
To: Jim Wilcoxson
Subject: AW: Question regarding a special backup option --no-inode
Date: Wednesday, May 15, 2024 11:25 AM

Hi Jim!

Yes, you can publish this conversation on your website. It does not contain any confidential information. Yes, please reduce my personal information like my private email address.

The backup log file says, that the backup failed. I would say: completed successfully. Yes, the upload to Backblaze was stopped after reaching the time limitation.

The retain log

A test run is still in progress.

Thomas

From: Jim Wilcoxson
To: Thomas Neuber
Subject: Re: Question regarding a special backup option --no-inode
Date: Wednesday, May 15, 2024 2:21 PM

Hi Thomas! Thanks for sending the results. Your backup seems to have worked very well. It pointed out a very small buglet; where it says:

2024-05-15 Wed 15:04:47| Interrupted before all archives were copied

The message should have been:

Time limit on --maxwait before all archives were copied.

But since you used --maxtime instead of --maxwait, it gave the other error message, which should only occur on ctrl-c. Easy to fix though.

Thanks for permission to use the email. I’ll send a link to review the web page when I get it formatted.

Jim