Oras cache saves the blob, updated the index.json, but fails to retrieve from cache in a second pull attempt

[flavioschuindt]

Hi, everyone.

I am facing a weird behavior with OCI container pulling and would like to start a discussion to have more insights. I am not sure if this is a bug or not hence opening as a discussion instead of a bug to have an initial opinion here.

So, to give some context, I have a typescript application that does many things, including acting as a wrapper for some oras commands, i.e., imagine a typescript application that has a oras.client.ts and that module is responsible for all oras related commands. The way it implements these commands is by calling the oras binary as a subprocess (see spawn). This has been working great for me and I can easily push and pull OCI images using this module capabilities from within my code. This is deployed on AWS in a lambda funtion and all disk storage is backed by EFS.

Now, to the problem: Recently, I came to know about the oras cache and thought to add so that I can save time from oras pulling OCI images. Initially, I tested locally by setting up the ORAS_CACHE env var and doing some push and pull and I can see it working perfectly. Then, I went ahead, added the ORAS_CACHE as env var for my lambda and tried to pull. The first pull of course took time as the cache had zero entries. Image is pulled by oras and I can see oras honoring the ORAS_CACHE and saving the blobs in the cache:

The index.json is populated as well:

Then, I would expect the second pull to be blazing fast, but it is not. It seems that oras is not using the previous content saved in the cache which is weird. I came to some questions/conclusions:

1. The cache is being seen by oras, otherwise the first attempt to pull the OCI image wouldn't even save the content in the cache.
2. Why it save the blobs fine, but doesn't retrieve the blobs in a second pull? Is there something missing in the way oras is indexing it?
3. As mentioned above, this is backend by EFS which in the end is NFS. I know that sometimes NFS has issues with caching, i.e., something you write now and is only available some time later. One thing you can do to "refresh" the cache is to force a "ls" in the file system. I tried this as well just to make sure the content is available, but no luck, oras still ignoring the cache while pulling.
4. Another thought I had was that somehow the node process inside lambda function opens a oras process using spawn and this might not be picking up the ORAS_CACHE env var. However I ruled this out because of these things: (a) If this was the case, (1) wouldn't even save the blobs in the cache folder; (b) as per node docs, the spawn function automatically pass the env vars from the parent process to the child one and (c) I did a printenv inside the child process and I can see the ORAS_CACHE there.

So, I am not sure why oras is saving, but refusing to read from cache here. Thank you in advance!

[qweeah]

@flavioschuindt can you kind share the version of ORAS CLI you used as well as the pull command run log with --debug flag set? By design manifest won't be cached but layers should be cached.

[Wwwsylvia]

Hi @flavioschuindt, this does sound strange.

It seems that oras is not using the previous content saved in the cache

To confirm this, could you collect the debug logs via --debug as @qweeah has mentioned? If the blobs are not fetched from cache, we should be able to see HTTP roundtrips in the logs.

[flavioschuindt]

Hi @Wwwsylvia and @qweeah, apologies for the delay here, but would like to give a quick update. Thanks for mentioning the -debug flag, I was able to enable it inside my TypeScript oras module and I found the root cause: This is not related to oras download at all. Instead, what I learned is that oras does two things: download and extract into the file system. The download part is ok, but what actually takes time is the extraction and this makes sense because my OCI package is a TS application with bunch of node modules backed by NFS. This means that oras is extracting into NFS file by file and this is an expensive operation. There are more than 12k files in the node_modules folder:

We can close this ticket and thank you once again for the support!