Fix directory targets with empty subdirs

[ElectreAAS]

Incorporates and replaces #11213, replaces #10931 and #11203, fixes #10609, #11117, #11214.
Work mostly done by myself, with various helpful insights by @maiste and @art-w.

The core of the changes are in dune_targets.ml, where I changed the representation of targets (Targets.Produced.t) from a flat structure to a hierarchical one, mirroring the file system. This makes all traversals easier to reason about, and prevents problems of hitting dirA/dirB/fileB before dirA/fileA, along with the intended feature of actually processing directories properly.

Other noteworthy changes are:

• Local.Target.create now also creates directories, not just files
• Build_system.file_exists now also checks for directories, not just files
• Dune_cache_storage.Artifacts.Metadata_entry.{to,of}_sexp have slightly changed semantics, I added the possibilty to have no digest but instead a simple "<dir>" to indicate a directory. I wouldn't be surprised to hear this has long ramifications. Same concern for Targets.Produced.{digest,to_dyn}.

I'll add for the record that a lot of functions named do_something_file also work on directories, but not all, and that is rather confusing. If I'm not the only one with this sentiment I could do a refactor to make this clear everywhere.

[ElectreAAS]

Below are the initial text of the PR, and a comment I made after working on this for a week. They are now outdated, I'm just keeping them here for posterity.

This PR is related to... a lot of things actually.

For some context, let's take this test file from #11116 & #11117:
Why isn't the empty directory output/child restored from cache like output/file?

• is it because a directory has a name that dune doesn't like, as in [regression] promotion of directory targets has trouble with directories starting in @ #10609?
• is it because it's too deep in a directory target, as in Failure in promote mode for directory targets with subdirectories #11214?
• or the same thing, but different, as in [fix] [directory targets] Fix promotion of directory targets in some cases #10931?
• also why doesn't Fix promotion of directory targets without files in them #11203 fix that too?

I'm not really sure, and the original author of this piece of code (rleshchinskiy) doesn't seem very active in this project at this time.
What I am sure of is that breaking the logic of collect is easy to do accidentally, but that at least fixes one problem.

Debugging this is tricky, as a lot of tests fail with a cryptic

Error: Is a directory
-> required by _build/default/a
[1]

I'll note however that the problems likely originates in either #9407, #9470, or #9535.

---

Belated update before the holidays:
I changed the internal representation of targets (Targets.Produced.t) to be hierarchical - it felt like the right move to detect broken logic, and was mentioned in the comments as something to be done at some point.
From that change I saw that the 'empty-dir' test passed, but a whole lot of other tests didn't. I thought I could work backwards from that to a point where all would pass, but the explanation was harsh: I had broken the caching mechanism, and the only reason the 'empty-dir' test passed was because nothing was cached and everything was rerun 🤦.
When I finally fixed everything else, obviously the 'empty-dir' test didn't pass (the cache worked like it did before my changes, wow!).
At this point in time I have pushed my changes, and they don't break anything, but also don't fix anything...
My hunch now is that this traversal only stores files as metadata, which means that this entries list only contains metadata about files, which means they're the only things getting restored. I tried making a of_files_plus_dirs to be used at that call site, but couldn't get it to work so far. Also I don't know how to generate digests of empty directories to actually store them in metadata...

Hope this progress report isn't too arcanic for reviewers 🙂

[Leonidas-from-XIV]

I forgot I had some review half-started so let me just submit what's there.

[Leonidas-from-XIV]

Maybe while at it, you could rename it to be something neutral? If something is called "file" I would not expect it to be a directory and it something is called a "directory" I would not expect it to be a file.

Maybe "directory entry"/"dirent"? Or "inode", though that might imply a bit too much. Or something else.

[Leonidas-from-XIV]

I would suggest renaming this.

[ElectreAAS]

Agreed, that's what I did initially, but this functions is used in a whole lot of places, and that made the diff rather heavy.
Since this is unrelated to this PR, maybe I'll put this in another one? Or do you think this is relevant?

[snowleopard]

Please avoid renaming things, as this will complicate upstreaming of our internal changes.

Note that build_file and build_dir functions in our internal code base do indeed build files and directories, respectively, so there won't be any such confusion.

[Leonidas-from-XIV]

Also renaming this. It would be very strange if I can call file_exists, get true and then call read_file and it fails since it is not a file.

[ElectreAAS]

Agreed. My initial rename was path_exists since that's the entry type. read_path would also be nice but obviously more involved