Storage issue when insecure__keys.default is disabled

[dzobbe]

Hi All,

Once again I am here to ask about an issue I have when running Vault inside Gramine. So, thanks in advance and sorry for bothering.
I have this config for the path where Vault stores its data.

{ path = "/data", uri = "file:/data", type = "encrypted", key_name = "default", allow_empty_files = true },

If I use insecure__keys.default I am able to normally run the vault operator init command.
If that config is disabled (for a prod-oriented deployment), the vault operator init returns this error:

 debug: epoll: added 7 (0xfce66b38) to epoll handle 0xfce664a8
gramine_vault  | (libos_init.c:561:create_pipe) [P3:T38:vault] debug: Creating pipe: pipe.srv:c62c1ea36eb2bdfae4a88cd8530fe3be4b26ac29027a3e5ee6fcd20bd70d3ff6
gramine_vault  | (libos_fs_encrypted.c:194:encrypted_file_internal_open) [P3:T35:vault] warning: key 'default' is not set
gramine_vault  | 2023-10-24T22:12:33.096Z [INFO]  core: security barrier not initialized
gramine_vault  | 2023-10-24T22:12:33.096Z [INFO]  core: seal configuration missing, not initialized
gramine_vault  | (libos_fs_encrypted.c:194:encrypted_file_internal_open) [P3:T38:vault] warning: key 'default' is not set
gramine_vault  | 2023-10-24T22:12:33.098Z [INFO]  core: security barrier not initialized
gramine_vault  | (libos_fs_encrypted.c:194:encrypted_file_internal_open) [P3:T38:vault] warning: key 'default' is not set
gramine_vault  | (libos_fs_encrypted.c:194:encrypted_file_internal_open) [P3:T38:vault] warning: key 'default' is not set
gramine_vault  | 2023-10-24T22:12:33.099Z [ERROR] core: failed to initialize barrier: error="failed to persist keyring: open /data/core/_keyring.temp: permission denied"

If I enable all debug info, besides a lot of noisy things I found this:

trace: ---- openat(AT_FDCWD, "/data/core/_keyring.temp", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0600) = -13

Note: I am using Gramine v1.5

Am I doing something wrong in terms of configuration?

Thanks guys.

[mkow]

Did you send ("provision") the key to the enclave? From the logs it seems that you didn't (a lot of warning: key 'default' is not set ), so there's just no key to encrypt the files with and that's why it fails.

Some reading about this topic: https://gramine.readthedocs.io/en/latest/attestation.html.

[dzobbe]

Thanks @mkow for the reply. I think I have missed something. I thought the encrypted file system was using by default the SGX sealed key and one could configure the key if needed. So it is not like that? I need to perform this secret provisioning to make the encrypted file system work?

Thanks.

[kailun-qin]

I thought the encrypted file system was using by default the SGX sealed key and one could configure the key if needed. So it is not like that?

@dzobbe Yes, using sealing key is one option for a secure version. You can set in the manifest w/ sth like { path = "/data", uri = "file:/data", type = "encrypted", key_name = "_sgx_mrsigner" or "_sgx_mrenclave" } for it to work. For more details, pls see https://gramine.readthedocs.io/en/stable/manifest-syntax.html#encrypted-files.

What @mkow suggested was another option, that is to set key_name = "<provisioned key name>" in the FS mount point in the manifest, then use a key provisioning flow with SGX remote attestation so that the encrypted file is later decrypted using that same provisioned key.

[dzobbe]

I would love to do that provisioning flow but for now the encrypted file system is not working as expected in my case XD

[dimakuv]

Please see replies from @mkow and @kailun-qin, but from my side just a note: allow_empty_files is not a thing in Gramine. This keyword is not implemented (at least yet). It was only discussed once in this issue: #1349. But this keyword doesn't exist in the codebase, and it's not really needed (see the linked issue).

[dzobbe]

Thanks all. I tried to use a different key_name but still the same issue.

d) [P3:T35:vault] warning: EOF
gramine_vault  | (libos_fs_encrypted.c:203:encrypted_file_internal_open) [P3:T35:vault] warning: pf_open failed: Callback failed
gramine_vault  | 2023-10-25T17:02:19.530Z [INFO]  core: security barrier not initialized
gramine_vault  | (libos_fs_encrypted.c:41:cb_read) [P3:T35:vault] warning: EOF
gramine_vault  | (libos_fs_encrypted.c:203:encrypted_file_internal_open) [P3:T35:vault] warning: pf_open failed: Callback failed
gramine_vault  | (libos_fs_encrypted.c:41:cb_read) [P3:T35:vault] warning: EOF
gramine_vault  | (libos_fs_encrypted.c:203:encrypted_file_internal_open) [P3:T35:vault] warning: pf_open failed: Callback failed
gramine_vault  | 2023-10-25T17:02:19.532Z [ERROR] core: failed to initialize barrier: error="failed to persist keyring: open /data/core/_keyring.temp: permission denied"
gramine_vault  | (libos_epoll.c:432:do_epoll_del) [P3:T40:vault] debug: epoll: deleted 7 (0xfce66b38) from epoll handle 0xfce664a8
trace: ---- openat(AT_FDCWD, "/data/core/_keyring.temp", O_WRONLY|O_CREAT|O_TRUNC|0x80000, 0600) = -13

I don't get why it works if I pass the key content with insecure__keys.default.

[kailun-qin]

@dzobbe Was the above log showing a clean run? Could you pls try cleaning up /data and give another try?

[dzobbe]

Wooo it worked. Thanks a lot for the help.