Album-Aware Deduplication: Merging Duplicate Files Across Multiple Albums

[jgmarcel]

Consider a situation where I have two identical copies of the same photo or video, such as MOVIE(2).mp4 and MOVIE(3).mp4. These duplicates arose unintentionally, likely from importing a ZIP file from Google Takeout.

Ente’s deduplication feature rightly identifies one of these files as redundant since they are exact matches in size. However, a complication arises because each file is associated with a different album. This duplication happened because the original file was part of two separate albums. With the duplication, one copy ends up in one album, and the other in a second album.

Deleting one of these duplicates creates an issue: the album containing the deleted file loses its association with that image. Therefore, I propose a ”merge” feature. This would involve retaining just one copy of the duplicated file, but ensuring it is linked to all albums where its duplicates were originally placed. Is such a feature technically feasible and practical to implement?

[vishnukvmd]

Adding symlinks instead of removing them altogether does make sense, thanks for the feedback!

Ideally, our desktop app should have deduplicated these items during upload. Expected behavior is that when exact duplicates are encountered, the app will add symlinks into the new album, instead of re-uploading a duplicate file. Curious why that did not happen here.

[jgmarcel]

Adding symlinks instead of removing them altogether does make sense, thanks for the feedback!

Indeed. One copy may be removed, because there are two of them, but the remaining one should be just symlinked within all albums containing one of the copies.

Expected behavior is that when exact duplicates are encountered, the app will add symlinks into the new album, instead of re-uploading a duplicate file.

Even if files have different names across albums? I don’t know why Google Photos did that, but the incremental counter, which is added as a suffix to files with the otherwise-same name, differs from one album to another.

Curious why that did not happen here.

Could I help with the investigation? I may provide you with logs, if there are any, or even a copy of the Takeout files that gave rise to the duplicates.

Last but not least, would you kindly let me know when I may proceed with the deduplication feature, via https://web.ente.io/deduplicate, without having my files removed from the albums where they are currently located? I have a few hundred ones right now, so the earlier, the better!

[ua741]

Even if files have different names across albums?

This explains. To preserve the original file name, if file names are different, we don't consider it as an exact duplicate.

I don’t know why Google Photos did that, but the incremental counter, which is added as a suffix to files with the otherwise-same name, differs from one album to another.

From our understanding, Google Photos export adds a counter suffix if the album contains multiple files with the same name.