Corrupted protected FS after the app crashed

[dzobbe]

Hi all,

Once again I am here to ask your support. We have a gramine-vault solution running on SGX v1 machines, configured with HA using the Raft storage, which is storing data with encrypted file system.
Everything seemed to run smoothly. But at a certain point the application started to crash and after redeploying the FS is not accessible anymore. I am sure it was the same version of the application. I read in the discussions here all the conditions that could lead to the unaccessible file system and none applies to our case.
I have redeployed the application in debug mode and I need to wait for another "corruption of data". So, for the moment I cannot give you relevant logs.
The usage of Vault Raft storage requires flock to be enabled. I am wondering if that could be the source of the problem given that is it still not production-ready.
I know it can be hard for you to help for now without logs, but do you have any idea on what could be wrong or any action I could take to prevent this?

Thanks a lot.

[dimakuv]

Hm, our FS encryption doesn't support file recovery after crashes. In other words, our FS implementation is not crash-consistent (or at least, not by design).

This means that if your app crashes unexpectedly in the middle of file encryption (and subsequent flushing to hard disk), this file probably cannot be recovered at all (as it contains e.g. partially overwritten file chunks, one part with the new encryption key, another part with the old encryption key, which confuses the decryption logic and makes Gramine stop working on the file).

So two questions here:

1. Does your Gramine-vault solution have crash consistency built-in? Or does the app rely on the underlying file system to recover such files? Or maybe the app doesn't take into account crashes in its normal flows at all?
2. Why does your app crash? At which point? Is it a Gramine bug, or an external event (e.g. power outage)?