Gc-based snapshots

[icristescu]

After discussions with @vect0r-vicall, I'm working on a new snapshot export that:

• calls gc on the commit for which we want to create the snapshot;
• copies the resulting prefix, mapping (and dict is needed too) to a new directory;
• that directory is the new snapshot format

The import:

• opens a new store from a prefix, mapping and a dict, by creating a new control file;
• add the commit to a fresh index.

This is WIP, but I would appreciate any feedback, in case I missed something. Things that I see to improve are:

• instead of manipulating the type snap in the code, we could write to a file;
• the control file after a snapshot import looks a lot like after a gc - we could introduce an extra status to differentiate the two
• a better management for dict?

[Ngoguey42]

Deadly simple!

[Ngoguey42]

Following an offline discussion with @icristescu, a good story for that export function could be that export takes the path where a directory will be created containing a new fully functional irmin-pack store which contains only the commit (and it's tree).

This way, no import function is needed.

[art-w]

Would it make sense to separate the export from running the GC? (= copy the existing prefix/mapping with first_commit as commit_key, without running a (destructive) GC to build the snapshot)
Nevermind I missread the finalize!

[Ngoguey42]

IIUC, this code runs the worker part of the normal GC but doesn't run the finalise part of the normal GC.

The worker part only creates new files, it doesn't modify the existing files in any way.

Does this answer your question?

Sure

[vect0r-vicall]

If I understand correctly, the start_exn function checks whether or not the GC is already running when called. This may trigger two corner cases, as a snapshot export can be called while the context is opened in RW:

• snapshot export is cancelled as a "real context GC" is already running -> not that bad,
• a "real context GC" is cancelled as a snapshot export is running in background -> this is quite bad.

How could we avoid that?
For the snapshot export cancellation, I guess that we could easily check the status of the GC is_finished to prevent the snapshot export or delay it until the GC completion.
Regarding the context GC cancellation, if a "GC export" is already running in background, maybe we could delay it as well and resume it as soon as the preceding snapshot export finishes. However, it means that we may need to distinguish "GC for export" and "real GC", as the delayed behaviour is not desired when "real GC" calls are overlapping.
What do you think?

[icristescu]

as a snapshot export can be called while the context is opened in RW

Indeed, I have not though about this case. Isn't it the case that currently we don't run a node and do a snapshot export at the same time?

To allow a concurrent snapshot export I think it could work as follow (I'm rephrasing a bit your suggestions):

• snapshot export is not launched if a gc is running - already the case with this PR.
• before launching a "real gc", check if one is running, cancel it and then run the "real gc"
  • right now, a "real gc" is skipped if a "real gc" is running. If we want to keep this behaviour, we need to distinguish between export and gc.
  • if we are ok with always cancelling previous gc (if the "real gcs" are separated enough that they don't interleave for example) then the current API allows us to do this from lib_context.

So either way, cancelling or skipping is maybe not too complicated. What is less trivial I think is the "postponing" part.

[vect0r-vicall]

Indeed, I have not though about this case. Isn't it the case that currently we don't run a node and do a snapshot export at the same time?

No. Snapshots can be exported while the octez node is running. We want to keep that feature and it should not interfere too much with the node's life.

if we are ok with always cancelling previous gc (if the "real gcs" are separated enough that they don't interleave for example) then the current API allows us to do this from lib_context.

There should be no interliving, indeed. However, it seems a good property not to cancel a running GC. Thus you will always perform a GC as some point. That is not guaranteed if the GC may be cancelled.

To me, skipping the "real GC" if a "snapshot GC" is running is not fine. That is the main issue.
Can we consider a "?(force = false)" for the gc function such as we chose whether or not the call must:

• true - cancel the running gc and start a new one : "real GC" use case
• false - fails/skip if a gc is running (the current behaviour) : "snapshot GC" use case

[icristescu]

this is ready for another round of reviews

[metanivek]

I really like this solution to updating snapshot as a simple export of a "one commit store". I left some thoughts on naming/code organization.

[codecov-commenter]

Codecov Report

Merging #2125 (ba7ff96) into main (f40ce19) will increase coverage by 0.04%.
The diff coverage is 80.85%.

@@            Coverage Diff             @@
##             main    #2125      +/-   ##
==========================================
+ Coverage   68.17%   68.21%   +0.04%     
==========================================
  Files         134      134              
  Lines       16042    16085      +43     
==========================================
+ Hits        10937    10973      +36     
- Misses       5105     5112       +7     

Impacted Files	Coverage Δ
src/irmin-pack/unix/errors.ml	28.88% <ø> (ø)
src/irmin-pack/unix/gc_worker.ml	6.06% <0.00%> (ø)
src/irmin-pack/unix/io_errors.ml	58.33% <ø> (ø)
src/irmin-pack/unix/gc.ml	77.88% <62.50%> (-1.29%)	⬇️
src/irmin-pack/unix/io.ml	65.21% <75.00%> (+0.35%)	⬆️
src/irmin-pack/unix/ext.ml	63.97% <81.25%> (+1.56%)	⬆️
src/irmin-pack/unix/file_manager.ml	84.75% <100.00%> (+0.83%)	⬆️
src/irmin-fs/irmin_fs.ml	81.21% <0.00%> (-1.22%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more