When data are available in the volume

[vschoener]

Hello guys,

Not sure it's the right place but I'm wondering when the data is getting synced in the dataset and available as the volume?

Is it done at the moment we mount the volume in a container or has it been done before?

Thank you !

[srikumar003]

@vschoener If you are referring to S3-based datasets (i.e., datasets that point to buckets), the data in the buckets is available whenever the process inside the container does a filesystem operation (read or write). It is only at that point that Goofys makes a call to the object storage provider to fetch or write the data. Mounting the volume does not create any local storage space in the cluster.

(Apologies for the late reply. For some strange reason, the discussions tab in my browser never showed there was an open question)

[vschoener]

@srikumar003 no worry :)

I asked about that cause we have some issues where the data are truncated/not complete when we mount the volume in the container.
We tried to use the initContainer with another emptyDir volume to avoid performance issues but we are still facing data not yet available.

There is no way to make the data available without having to wait for a file operation? Cause if we need to wait for a user to request a file and wait the file being synced, it will cause major availability issues at the end :(

Thank you