Integration of checksum-xattr support

[jluebbe]

The rsync-patches repository contained a checksum-xattr.diff until the update to v.3.2.7 and was removed as it "Lacks negotiated-checksum support".

We'd be interested in using this approach for RAUC to avoid reading the source file data in cases where the target already has the correct file contents. Compared to checksum-reading.diff, it seems to be very simple, likely because it doesn't need to parse and cache the rsyncsums files.

We'd rebase checksum-xattr to current master and add negotiated-checksum support. Would you be open to integrating this into rsync itself or are there significant changes that would need to be done before that would be acceptable?

[WayneD]

Sorry, I forgot to reply. The xattr-based code is not a very good method of checksum caching as it tweaks the file's ctime and thus makes it hard to know if the file changed content w/o an mtime change. Thus, this code will never make it into a production rsync.

The db patch was a much better method of checksum caching, but all the new checksums made it quite out-of-date. I actually used that code in a corporate setting a number of years ago (allowing checksums on a public mirror), and it worked very nicely. It would be cool if that could be reworked into something that handles any checksum type and had better local-db support than SQLite (which has a huge problem with shared access from multiple processes).

At one time I had imagined a project to write a checksum-server that would allow web-like queries for returning checksum info for any file on the host (which could use a single-threaded db-access layer to make SQLite happy, or use any other backing DB method the user desires). Such a server would require using unix sockets in order for the server to be able to determine the user info of the requesting process and thus be able to limit access to checksum info based on source-file read permissions. Such a nice separation of checksum server (with multi-db support) would make it very easy for rsync to just support that one checksum-caching method, and thus not bloat the standard rsync install with db-library dependencies.

One complicating factor would be how to add new checksums in a trustworthy manner. For instance, if the user asks for a checksum that isn't present, the caller could indicate if it wanted the server to fork a process to compute the checksum when it was unknown (and thus the server would not have to trust the requesting user to send in accurate checksum data). Another possibility would be to trust incoming checksum info from a process if the requesting user has write permissions on the source file. This would allow a program like rsync that is always computing file checksums as it copies files to send-off checksum updates to the server even when --checksum wasn't specified.

[jluebbe]

Thanks for the background information!

I've looked at the db patch and it has a lot of complexity compared to checksum-xattr or even checksum-reading. For example, the checksum-reading and db patches try to detect if the checksum in the separate checksum-file could be outdated (by looking at ctime or inode numbers).

I'm not quite sure I understand the issue with ctime changing when setting the checksum xattr. As far as I can see, unpatched rsync currently doesn't use a file's ctime. Perhaps I should describe why I don't think this is a big issue:

The approach taken in checksum-xattr could be regarded as a middle ground between the default "quick check" mode (file size and mtime) and the existing --checksum mode: In quick check mode, content changes might not be detected if the mtime is restored after writing to the file. When running with --checksum, we can be sure that any content changes are detected, because we generate a fresh checksum on demand.

For cases where the file tree on the sending side is well under control and no files are modified in place (i.e. a Debian mirror or my software update use-case with RAUC), the program managing the file tree can also generate or update the checksums, so that one can reasonably assume them to be correct. Any problem that could cause wrong checksums could cause bad file contents just as easily.

The benefit would be to avoid the large amount of I/O when the receiver wants to make sure it has the correct contents (e.g. because something might have modified file contents but not mtime on the client). So the result from the receiver's perspective would be to ensure that contents and metadata match what the sender stored in its file tree.

As it's not as strict in cases where the files on the sending side might have been modified (e.g. normal user file systems), a separate command-line option sounds appropriate (--checksum-xattr?). It would enable --checksum mode and also opportunistically use the user.rsync. xattr to avoid the calculation.