Fix Android bug on FAT32/exFAT SD card time resolution and mtime caching

[lmagyar]

As I wrote in #351 (comment), Android lies about the lastModifiedTime. Even the specs says, that if the time resolution is lower than what we ask, the next getLastModifiedTime will give back the "truncated" value, this is simply not true (tested on real phone, Android 9.0 and emulator). It gives back the truncated values only when he wants, usually hours later. This is an unpredictable nightmare.

This fix:

• adds a new method to the StorageManagerUtil class that detects SAF filesystems from /proc/mounts, it detects:
  • FAT16, FAT32 -> 2000ms
  • exFAT -> 10ms 2000ms
  • default 1ms
• moves the fileSystemView property from SafSshFile up to SafFile, so all SAF file can call the new getTimeResolution() from SafFileSystemView

Update:

• also implemented for roSAF and poFS
• poFS alters mtime only for files on the SD card's, ie. under path eg. /storage/XXXX-XXXX

Maybe the default 1ms is overkill, in theory FTP has sub-second resolution, though we can modify StorageManagerUtil to return only 1000 vs 2000 ms, in this case no overload needed in SafSshFileSystemView to limit sub second resolution to the 1s resolution of sftp messages.

[lmagyar]

Not the latest logs, but some proof it works.
Emulator:

getFilesystemTimeResolutionForSftp(content://com.android.externalstorage.documents/tree/17FB-0715...
  found mount point /mnt/media_rw/17FB-0715 with type vfat

Real FAT32 card:

getFilesystemTimeResolutionForSftp(content://com.android.externalstorage.documents/tree/80AB-022A...
  found mount point /mnt/media_rw/80AB-022A with type sdfat with option fs=vfat:32

The fs=exfat sample in the code comment is from a real exFAT 64GB card.

[wolpi]

I'm really looking forward to that merge script 😄

[lmagyar]

I'm really looking forward to that merge script 😄

The Python sync? 😄 It is working, I've added remote management (start/stop, with Automate or HomeAssistant) for Tailscale, pFTPd, Termux/sshd services (the later for inotifywait, to catch the changes on the fly). I need a few hours net to finalize it on to a v0.1 quality level, though with a newborn and a relocation I hardly find time to sleep. 😨

I will work on the tech comments above on next week.

[wolpi]

Happy coding between moving boxes and with new born on arm 😄

[lmagyar]

Happy coding between moving boxes and with new born on arm 😄

Yeah, this description is quite accurate. :)

Just a note:

Android inaccurate on file mod. times not only on files we have uploaded, but on eg. new pictures. When we download a fresh new picture through plain-old FS from the SD card, it will have an inaccurate timestamp, but a few hours later it will be rounded to 2s (in case of FAT32) and my sync script will download it again. Not a big issue, doesn't happen too often, but an issue, especially when I delete the downloaded file, and the delete and "modification" causes a conflict.

So this solution should be extended to RoSaf and plain-old FS (when we access files under /fs/storage/XXXX-XXXX). Not a big problem, but it needs some time.

[lmagyar]

After some "working, but I don't like it" versions, this is the minimalistic version:

• StorageManagerUtil.getFilesystemTimeResolutionForTreeUri()
  • handles another Android bug (even exFat got a 2s resolution)
  • caches the calculated time resolutions, this wont change while pftpd is running
• FsFileSystemView, RoSafFileSystemView, SafFileSystemView got a timeResolution argument
  • FsFileSystemView got another safVolumePath argument
  • all got a getCorrectedTime() method
  • but FsFileSystemView uses safVolumePath to use the altered timeResolution only for files under the read-only SAF root path
• FsFile, RoSafFile, SafFile got a fileSystemView attribute, and ctor argument
  • all uses this to get the proper/corrected time when reading/writing lastModifiedTime
  • ...FtpFile, ...SshFile got a private getFileSystemView() method for casting fileSystemView back to the correct type to minimize the casting
• all other modifications are only passing the fileSystemView arguments in the ctor-s

Issues:

• rootFS is not modified, I can't test it, and I assume it is used mainly to access other files, not the SD card
• this time resolution fixing works only, when the user selected something on the SD card for SAF (the root or subfolder; even the subfolder is not an issue for poFS)
  • but if something else (eg. internal Downloads) is selected, roSAF and SAF will not modify anythng, but poFS won't be able to determine the SD card location in the file system and will not modify any timestamp - I think we can live with it, this is still better than the original Android behaviour

Now if I make new pictures with the camera, stored on the SD card, sync it to the laptop (through poFS, roSAF or SAF), the timestamps will be the correct even seconds immediatelly and not only after a few hours (causing another sync or sync conflict if I deleted them meantime on the laptop).

[lmagyar]

Draft again, I will refactor some details (no functional change).

[lmagyar]

It's ready for review again.

Added caching and made some refactor

The long comment above summarizing the changes got updated!

[lmagyar]

Merge conflicts resolved.

[lmagyar]

OK, conflicts are really fixed now, it builds. :D

[wolpi]

This whole topic is still unbelievable. I would have never put so much effort just in timestamps.

One thing I noticed:
Now Fs..File objects have some relation to SAF. I'm going to add comments about that.