How Gramine can management memory space larger than 128 MB?

[ZinuoCai]

Hello, everyone! After I reading the official documents of Intel SGX, I get it that the performance of SGX applications will degrade if the enclave size is larger than 128 MB. I wonder whether the feature will affect Gramine's performance, and how Gramine overcomes it if not.

[boryspoplawski]

I get it that the performance of SGX applications will degrade if the enclave size is larger than 128 MB

No idea where you got that from, but sounds plain incorrect.

Maybe you confused it with EPC size? It varies from system to system, but on newer ones could be a couple of GB.
EPC size is indeed a SGX limitation and using more memory (than EPC size) will degrade performance, but there is nothing that Gramine can do about that.

[ZinuoCai]

Thanks for your reply.

Maybe you confused it with EPC size?

Sorry for the mistake. I review the materials that I have read and clarify that the PRM size is usually 128 MB, and about 94 MB are allocated to EPC. I found it here and other papers.

on newer ones could be a couple of GB

Would you please provide some evidence about the increasing amount of EPC size? I did not found it before. Thanks again for your reply.

[dimakuv]

@ZinuoCai First, a couple quick answers to your questions.

1. Enclave size may be arbitrary, specified in the Gramine manifest file as sgx.enclave_size.
2. Enclave memory is backed by EPC (Enclave Page Cache), which is a chunk of physical memory allocated by the platform specifically for SGX-enclaves needs.
3. If the working set size (people also call it resident set size) of the enclave (virtual memory) exceeds the EPC size (physical memory), then the process called "EPC page swapping" happens -- this process is very slow, so it degrades performance of the enclave.
4. EPC size was around 128MB on older Intel CPUs (e.g. Skylakes from 2015), but now it is anywhere from 256MB to 128GB on newer Intel CPUs (e.g. Icelake Servers).

Some references that provide more technical info:

• https://gramine.readthedocs.io/en/latest/performance.html#choice-of-sgx-machine
• https://eprint.iacr.org/2016/086.pdf
• https://www.usenix.org/system/files/conference/osdi16/osdi16-arnautov.pdf (first half of the paper)
• https://www.intel.com/content/www/us/en/support/articles/000059614/software/intel-security-products.html
• https://www.fortanix.com/blog/2021/04/how-will-intels-ice-lake-redefine-the-scope-of-data-security

[ZinuoCai]

Thanks all!