Error updating inventory for irregular filenames

[reliztrent]

Two files were deleted, and their removal was logged in json report at /storage/inventory/Drobo5Volume0_reports/2019/04/14/2019-04-14T16:03:13.981114-04:00.json:
XX1.$$$
XX2.$$$

When attempting to update the inventory based on this report, encountered a python error:
Traceback (most recent call last):
File "audit_tool.py", line 265, in
inventory_manager.update_inventory(inventory_report)
File "audit_tool.py", line 59, in update_inventory
timestamp=inventory_diff.timestamp)
File "/opt/audit-tool/inventory.py", line 151, in update
del self.files[file]
KeyError: 'XX2.$$$'

[reliztrent]

Today, a detect_changes chronjob completed.

Expected: The two files above (XX1.$$$ and XX2.$$$) will appear in the chronjob reports, under "files missing from fs."

Actual: Files do not appear in the detect_changes report, suggesting that they were written to the inventory.

Problem: the "update" command run two days ago resulted in an error, and it is unclear which functions of the update command were completed and which were not.

[lwrubel]

I've been able to add and delete files with $$$ extensions, and not trigger the error. However, I was able to trigger a KeyError for a file called 'test.txt' when running an update on a json report file that was not the most recently created one. Here are the steps:

1. python audit_tool.py detect_changes /home/lwrubel/projects/audit-storage/storage/Drobo5Volume0/
   Result: created report /home/lwrubel/projects/audit-storage/storage/inventory/Drobo5Volume0_reports/2019/04/26/2019-04-26T13-16-49-183780-04-00.json)
2. python audit_tool.py update /home/lwrubel/projects/audit-storage/storage/inventory/Drobo5Volume0_reports/2019/04/26/2019-04-26T13-16-49-183780-04-00.json
   Result: No error, completed successfully
3. Running update command on an older report: python audit_tool.py update /home/lwrubel/projects/audit-storage/storage/inventory/Drobo5Volume0_reports/2019/04/26/2019-04-26T13-13-02-818844-04-00.json
   Result: KeyError. This looks to be because the inventory doesn't have that file listed and so it can't compare to update.

Is it possible that in the original scenario, that the update command was used on a report that was not the most recent?

[reliztrent]

Interesting.

In the original scenario, this was the most recent report available that included the XX1.$$$ and XX2.$$$ files. So, no I don't think that the update command would have already been run on a more recent report for these files.

But, it's possible that an update command had been run on the same report twice, and this was the second time that an update command was run on the same report.

In the original scenario, the update report was run on a report that documented only two file changes: removal of XX1.$$$ and XX2.$$$. I am surprised that we got the key error on the second of these files and not on the first. But, otherwise, it seems likely that the error was caused by running update on the same report twice.

Perhaps a future enhancement might be: when the update command is run, the json report is "marked" as having been applied to the inventory. It could be "marked" by appending a string to the report's filename, or by dropping a second json file next to the report, which logs the update.