Has anyone attempted to use Ray (ray start) through gramine-sgx? Am running into mmap failed with error: Permission denied error.

[gapisback]

Hello,

Has anyone attempted to start a Ray cluster from inside gramine-sgx?

Problem Statement

Using sample template manifest files from this Gramine-examples repo (and a few other Gramine manifest doc sources), I am using a basic python.manifest.template file.

The command I'm using to start head-node of a Ray cluster is like so:

$ gramine-sgx ./python /home/sgx/.local/bin/ray --logging-level debug start --head --port=6379 --object-store-memory=78643200 --object-manager-port=8076 --dashboard-host=0.0.0.0

Btw, I should mention that on this same SGX-enabled machine, Ray cluster-start ray start --head (head-node) and starting / stopping worker nodes ... ray status -- all commands work just fine OUTSIDE Gramine. So, there is NO issue with the ray installation I have done on this machine.

With Gramine, here is what happens: Ray startup goes through for a file, creates the required ray-session-dirs in /tmp/ray and so on.

The startup process eventually fails with these errors:

sgx-server:[54] $ m /tmp/ray/session_2023-12-01_01-44-30_466913_1/logs/raylet.out
[2023-12-01 01:46:22,064 I 15 15] (raylet) main.cc:174: Setting cluster ID to: f15d17b55973d25152b4b7c41090c5549a0ef99ac7fa80dc20ec434e
[2023-12-01 01:46:22,149 I 15 15] (raylet) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2023-12-01 01:46:22,214 I 15 15] (raylet) store_runner.cc:32: Allowing the Plasma store to use up to 0.0786432GB of memory.
[2023-12-01 01:46:22,214 I 15 15] (raylet) store_runner.cc:48: Starting object store with directory /dev/shm, fallback /tmp/ray, and hug
e page support disabled
[2023-12-01 01:46:22,330 I 15 171] (raylet) dlmalloc.cc:154: create_and_mmap_buffer(78643208, /dev/shm/plasmaXXXXXX)
[2023-12-01 01:46:22,331 E 15 171] (raylet) dlmalloc.cc:213: mmap failed with error: Permission denied
[2023-12-01 01:46:23,279 I 15 15] (raylet) client_connection.cc:56: Retrying to connect to socket for endpoint /tmp/ray/session_2023-12-
01_01-44-30_466913_1/sockets/plasma_store (num_attempts = 1, num_retries = 10)
[2023-12-01 01:46:23,316 C 15 171] (raylet) plasma_allocator.cc:76:  Check failed: allocation.has_value() PlasmaAllocator initialization
 failed. It's likely we don't have enought space in /dev/shm

Triage Notes

From my analysis, the error is not really to-do with we don't have enought space in /dev/shm, but possibly due to some interaction (or lack of support) for accessing /dev/shm and how Ray's code does mmap() on a file created in this /dev/shmem directory.

I have chased this down somewhat through Gramine code changes and have found that shared-memory support was added to Gramine under PR #827.

I have checked that this feature support is there in the installed version of gramine-sgx I am using.

The relevant snippets from my python.manifest.template are the following: (See Appendix for the full template file.)

 66 fs.mounts = [
 67   # Mount on-host directory to Gramine glibc/runtime libraries (in 'uri') into
 68   # # in-Gramine visible directory /lib (in 'path').
 69   { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
 70

[...]

 83   { type = "tmpfs", path = "/tmp" },
 84
 85   # raypath supplied by Makefile -D flag
 86   { path = "{{ raypath }}", uri = "file:{{ raypath }}" },
 87
 88   # -------------------------------------------------------------------------------
 89   # This appears suspicious too. Find out if there is a way to define the
 90   # shm dir inside Gramine for Ray to create the shmem.
 91   # Ref: https://github.com/gramineproject/gramine/issues/757
 92   #      https://github.com/gramineproject/gramine/pull/827
 93   # You need this, otherwise, 'ray start' will fail with this error:
 94   #   File "/home/sgx/.local/lib/python3.8/site-packages/ray/_private/services.py", line 1985, in determine_plasma_store_config
 95   #  shm_avail = ray._private.utils.get_shared_memory_bytes()
 96   # File "/home/sgx/.local/lib/python3.8/site-packages/ray/_private/utils.py", line 700, in get_shared_memory_bytes
 97   #  shm_fd = os.open("/dev/shm", os.O_RDONLY)
 98   # FileNotFoundError: [Errno 2] No such file or directory: '/dev/shm'
 99   # -------------------------------------------------------------------------------
100   { path = "/dev/shm", uri = "file:/dev/shm" },
101
102   # These 2 lines _should_ be ok, but if Ray is assuming that /tmp is tmpfs
103   # then L30 and L37 may be a problem if used in conjunction.
104   { path = "/tmp", uri = "file:/tmp" },
105   { path = "/tmp/ray", uri = "file:/tmp/ray" },
106 ]

[...]

111 # As per https://github.com/gramineproject/gramine/pull/827
112 sys.insecure__shared_memory = "passthrough"

[...]

174 sgx.trusted_files = [
175   "file:{{ gramine.libos }}",
176   "file:{{ entrypoint }}",
177   "file:{{ gramine.runtimedir() }}/",
178   "file:{{ arch_libdir }}/",
179   "file:/usr/{{ arch_libdir }}/",
180
181   {% for path in python.get_sys_path(entrypoint) %}
182     "file:{{ path }}{{ '/' if path.is_dir() else '' }}",
183   {% endfor %}
184
185   # "file:src/",
186   "file:scripts/",
187   "file:helper-files/",
188   "file:{{ raypath }}",
189   "file:/tmp/ray/",
190 ]
191
192 # Obtained syntax from: https://github.com/gramineproject/gramine/issues/1445
193 sgx.allowed_files = [
194   "file:/dev/shm/"
195 ]

All the right incantations seem to be in-place in above template specification.

But I'm still running into the error reported above.

Relevant code from Ray Sources:

Ray's code that runs during ray start is doing this in ray/object_manager/plasma/dlmalloc.cc @ L211:

153   file_template += "/plasmaXXXXXX";
154   RAY_LOG(INFO) << "create_and_mmap_buffer(" << size << ", " << file_template << ")";
155   std::vector<char> file_name(file_template.begin(), file_template.end());
156   file_name.push_back('\0');
157   *fd = mkstemp(&file_name[0]);
158   if (*fd < 0) {

[...]

177   // MAP_POPULATE can be used to pre-populate the page tables for this memory region
178   // which avoids work when accessing the pages later. However it causes long pauses
179   // when mmapping the files. Only supported on Linux.
180   auto flags = MAP_SHARED;
181   if (RayConfig::instance().preallocate_plasma_memory()) {
182     if (!MAP_POPULATE) {
183       RAY_LOG(FATAL) << "MAP_POPULATE is not supported on this platform.";
184     }
185     RAY_LOG(INFO) << "Preallocating all plasma memory using MAP_POPULATE.";
186     flags |= MAP_POPULATE;
187   }

[...]

211   *pointer = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, *fd, 0);
212   if (*pointer == MAP_FAILED) {
213     RAY_LOG(ERROR) << "mmap failed with error: " << std::strerror(errno);
214     if (errno == ENOMEM && dlmalloc_config.hugepages_enabled) {
215       RAY_LOG(ERROR)
216           << "  (this probably means you have to increase /proc/sys/vm/nr_hugepages)";
217     }

The failure is coming from the call to mmap() on L 211 above.

Question:

Questions, to the general members of Gramine and specifically to the implementers / reviewers of PR #827, are:

1. Has anyone attempted this integration of running Ray from inside gramine-sgx?

2. Am I missing something in my Gramine template specifications?

3. With Gramine's support for shared-memory (PR [LibOS,PAL/Linux-SGX] Add shared untrusted memory support #827), is this usage of doing mmap(PROT_WRITE) on a randomly-named file from /dev/shm supposed to work? Or, am I simply running into some feature incompatibility here?

Acknowledgements

Thanks for reading this far! I'm stuck ... so any guidance to get me unblocked will be most appreciated!

Appendix

Contents of python.manifest.template

  1 # -------------------------------------------------------------------------
  2 # Python3 manifest example
  3 # Ref: https://gramine.readthedocs.io/en/stable/manifest-syntax.html
  4 # -------------------------------------------------------------------------
  5
  6 # Copious comments copied from online README for Redis manifest
  7 # https://github.com/gramineproject/gramine/blob/master/CI-Examples/redis/redis-server.manifest.template
  8 # PAL entrypoint (points to the LibOS layer library of Gramine). There is
  9 # currently only one implementation, so it is always set to libsysdb.so.
 10 loader.entrypoint = "file:{{ gramine.libos }}"
 11
 12 # Entrypoint binary which Gramine invokes. This is supplied by the makefile
 13 # -D<arg>, pointing to the location of the python3 binary.
 14 libos.entrypoint = "{{ entrypoint }}"
 15
 16 # Supplied by Makefile; should be one of:
 17 # loader.log_level = "[none|error|warning|debug|trace|all]"
 18 loader.log_level = "{{ log_level }}"
 19
 20 ################################# ENV VARS ####################################
 21
 22 # Specify paths to search for libraries. The usual LD_LIBRARY_PATH syntax
 23 # applies. Paths must be in-Gramine visible paths, not host paths (i.e.,
 24 # paths must be taken from fs.mounts[...].path, not fs.mounts[...].uri).
 25 #
 26 loader.env.LD_LIBRARY_PATH = "/lib:/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"
 27
 28 loader.env.PYTHONPATH="/home/sgx/.local/lib/python3.8/site-packages"
 29
 30 # Python's NumPy spawns as many threads as there are CPU cores, and each thread
 31 # consumes a chunk of memory, so on large machines 1G enclave size may be not enough.
 32 # We limit the number of spawned threads via OMP_NUM_THREADS env variable.
 33 loader.env.OMP_NUM_THREADS = "4"
 34
 35 loader.insecure__use_cmdline_argv = true
 36
 37 ################################## SIGNALS ####################################
 38
 39 # Allow for injecting SIGTERM signal from the host. Without this option,
 40 # pressing `Ctrl + C` wouldn't terminate the app (Ray) being run under Gramine.
 41 sys.enable_sigterm_injection = true
 42
 43 # -------------------------------------------------------------------------
 44 # Turn this ON to see how subprocess-related specifications work.
 45 # This will cause 'ray start' to fail with this error:
 46 #   File "/home/sgx/.local/lib/python3.8/site-packages/ray/_private/services.py", line 1003, in start_ray_process
 47 #    process = ConsolePopen(
 48 #  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
 49 #    self._execute_child(args, executable, preexec_fn, close_fds,
 50 #  File "/usr/lib/python3.8/subprocess.py", line 1639, in _execute_child
 51 #    self.pid = _posixsubprocess.fork_exec(
 52 # BlockingIOError: [Errno 11] Resource temporarily unavailable
 53 # -------------------------------------------------------------------------
 54 # sys.disallow_subprocesses = true
 55
 56 ################################# MOUNT FS ####################################
 57
 58 # General notes:
 59 # - All mount points are mounted using the default 'chroot' type.
 60 # - `path`: names of directories and files in Gramine environment; they may be
 61 #           arbitrary but here we mostly reuse host URIs for simplicity (except
 62 #           for the first `/lib` mount point).
 63 # - `uri`:  names of directories and files on the host, somewhat confusingly
 64 #           prepended by the 'file:' keyword.
 65
 66 fs.mounts = [
 67   # Mount on-host directory to Gramine glibc/runtime libraries (in 'uri') into
 68   # # in-Gramine visible directory /lib (in 'path').
 69   { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
 70
 71   { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
 72   { path = "/usr/{{ arch_libdir }}", uri = "file:/usr/{{ arch_libdir }}" },
 73
 74   {% for path in python.get_sys_path(entrypoint) %}
 75     { path = "{{ path }}", uri = "file:{{ path }}" },
 76   {% endfor %}
 77
 78   # entrypoint supplied by Makefile -D flag
 79   { path = "{{ entrypoint }}", uri = "file:{{ entrypoint }}" },
 80
 81   { path = "/etc/hosts", uri = "file:helper-files/hosts" },
 82
 83   { type = "tmpfs", path = "/tmp" },
 84
 85   # raypath supplied by Makefile -D flag
 86   { path = "{{ raypath }}", uri = "file:{{ raypath }}" },
 87
 88   # -------------------------------------------------------------------------------
 89   # This appears suspicious too. Find out if there is a way to define the
 90   # shm dir inside Gramine for Ray to create the shmem.
 91   # Ref: https://github.com/gramineproject/gramine/issues/757
 92   #      https://github.com/gramineproject/gramine/pull/827
 93   # You need this, otherwise, 'ray start' will fail with this error:
 94   #   File "/home/sgx/.local/lib/python3.8/site-packages/ray/_private/services.py", line 1985, in determine_plasma_store_config
 95   #  shm_avail = ray._private.utils.get_shared_memory_bytes()
 96   # File "/home/sgx/.local/lib/python3.8/site-packages/ray/_private/utils.py", line 700, in get_shared_memory_bytes
 97   #  shm_fd = os.open("/dev/shm", os.O_RDONLY)
 98   # FileNotFoundError: [Errno 2] No such file or directory: '/dev/shm'
 99   # -------------------------------------------------------------------------------
100   { path = "/dev/shm", uri = "file:/dev/shm" },
101
102   # These 2 lines _should_ be ok, but if Ray is assuming that /tmp is tmpfs
103   # then L30 and L37 may be a problem if used in conjunction.
104   { path = "/tmp", uri = "file:/tmp" },
105   { path = "/tmp/ray", uri = "file:/tmp/ray" },
106 ]
107
108 sys.stack.size = "2M"
109 sys.enable_extra_runtime_domain_names_conf = true
110
111 # As per https://github.com/gramineproject/gramine/pull/827
112 sys.insecure__shared_memory = "passthrough"
113
114 # Supposedly recognized in libos/src/sys/libos_fcntl.c, but isn't being
115 # recognized by Gramine/SGX installed on our SGX machine.
116 # sys.experimental__enable_flock = true
117
118 ############################### SGX: GENERAL ##################################
119
120 # Create a debug SGX enclave (with SIGSTRUCT.ATTRIBUTES.DEBUG bit set to 1).
121 # This allows to debug Gramine with the application using GDB, read perf
122 # counters and enable SGX statistics. Note that this option is *insecure*!
123 sgx.debug = true
124
125 # Set enclave size (somewhat arbitrarily) to 1 GiB. Recall that SGX v1 requires
126 # to specify enclave size at enclave creation time. If Ray exhausts this, it
127 # will start failing. Greater enclave sizes result in longer startup times,
128 # smaller enclave sizes are not enough for typical Ray workloads.
129 sgx.enclave_size = "1G"
130
131 # Enable Enclave Dynamic Memory Management (EDMM) feature based on EDMM
132 # environment variable. This allows for addition of pages to enclave at runtime,
133 # instead of allocating them upfront at startup. If this feature is enabled,
134 # `sgx.enclave_size` above describes a maximal enclave size and can usually be
135 # increased without negative consequences (it does not impact startup time).
136 sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
137
138 # Set maximum number of in-enclave threads (somewhat arbitrarily) to 32. Recall
139 # that SGX v1 requires to specify the maximum number of simultaneous threads at
140 # enclave creation time.
141 #
142 # Should be at least 4, even for single-threaded applications (to accommodate
143 # for the main thread, the IPC thread, the Async thread and one TLS-handshake thread).
144 sgx.max_threads = 32
145
146 sgx.remote_attestation = "{{ ra_type }}"
147 sgx.ra_client_spid = "{{ ra_client_spid }}"
148 sgx.ra_client_linkable = {{ 'true' if ra_client_linkable == '1' else 'false' }}
149
150 # Temp w/a; Real fix may be to use sgx.allowed_files specification.
151 sgx.file_check_policy = "allow_all_but_log"
152
153 ############################# SGX: TRUSTED FILES ###############################
154
155 # Specify all files used by Ray and its dependencies (including all libraries
156 # which can be loaded at runtime via dlopen), as well as other static read-only
157 # files (like configuration files).
158 #
159 # The paths to files are on-host paths. These files will be searched for in
160 # the in-Gramine visible paths according to the mount points specified above.
161 #
162 # NOTE: The following happens on the ** developer machine ** or a build server.
163 # As part of the build process, Gramine-SGX script (`gramine-sgx-sign`) finds
164 # each specified file, measures its hash, and adds it to the manifest entry for
165 # that file (converting each entry to a table with "uri" and "sha256" keys).
166 #
167 # If a directory is specified in the list below, then this directory is
168 # recursively traversed and each found file is processed as described above.
169 #
170 # NOTE: The following happens at run-time on the ** deployment ** machine:
171 # At runtime, during loading of each "trusted file", Gramine-SGX measures its
172 # hash and compares with the "sha256" value in the corresponding manifest entry.
173 # If hashes match, this file is trusted and allowed to be loaded and used.
174 sgx.trusted_files = [
175   "file:{{ gramine.libos }}",
176   "file:{{ entrypoint }}",
177   "file:{{ gramine.runtimedir() }}/",
178   "file:{{ arch_libdir }}/",
179   "file:/usr/{{ arch_libdir }}/",
180
181   {% for path in python.get_sys_path(entrypoint) %}
182     "file:{{ path }}{{ '/' if path.is_dir() else '' }}",
183   {% endfor %}
184
185   # "file:src/",
186   "file:scripts/",
187   "file:helper-files/",
188   "file:{{ raypath }}",
189   "file:/tmp/ray/",
190 ]
191
192 # Obtained syntax from: https://github.com/gramineproject/gramine/issues/1445
193 sgx.allowed_files = [
194   "file:/dev/shm/"
195 ]

[kailun-qin]

Hi @gapisback, thanks for the question and detailed information!

Has anyone attempted this integration of running Ray from inside gramine-sgx?

Unfortunately no, I'm not aware of such attempt at this moment.

Am I missing something in my Gramine template specifications?

Yes, from a quick look, I noticed below configs in your template:

{ path = "/dev/shm", uri = "file:/dev/shm" },
...
sgx.allowed_files = [
    "file:/dev/shm/"
]

Instead, I think you would actually need { type = "untrusted_shm", path = "/dev/shm", uri = "dev:/dev/shm" } (i.e., w/ type untrusted_shm specified in the fs.mounts) for this to work.

I also noticed:

sys.insecure__shared_memory = "passthrough"

This looks stale. Pls use the latest Gramine master branch (w/ the feature included) and see https://gramine.readthedocs.io/en/latest/manifest-syntax.html#untrusted-shared-memory for more details.

Would you pls give the above another try? Thanks!

is this usage of doing mmap(PROT_WRITE) on a randomly-named file from /dev/shm supposed to work? Or, am I simply running into some feature incompatibility here?

No, files in a shared memory mount need to be explicitly listed as allowed_files to be accessed (pls take a look at our shm regression test and manifest).

[gapisback]

Thanks, @kailun-qin -- For your quick and clear response.

We rebased to the latest commit off master @ SHA

sgx-server:[6] $ gramine-sgx --version
Gramine 1.5post~UNRELEASED (8dad2ecd50d7a4baffdb81193ea25cfe4325d513)

And we made the following changes to my template file: (Brief diffs of changes shown below.)

-  { path = "/dev/shm", uri = "file:/dev/shm" },
+  { type = "untrusted_shm", path = "/dev/shm", uri = "dev:/dev/shm" },
+
+  { path = "/dev/vfio", uri = "dev:/dev/vfio" },

-sys.insecure__shared_memory = "passthrough"
+# sys.insecure__shared_memory = "passthrough".  (Commented this out.)

 sgx.allowed_files = [
-  "file:/dev/shm/"
+    "dev:/dev/shm/",
 ]

And am now able to get past the bootstrap issues relating to /dev/shm while starting Ray from inside Gramine.

The startup messages from Ray code look like so: /tmp/ray/session_2023-12-02_01-11-31_422392_1/logs/raylet.out:

[2023-12-02 01:13:21,950 I 15 15] (raylet) main.cc:174: Setting cluster ID to: 7d4888d404a3217dbae06ef2268dfd7192da36a3191947351d87e486
[2023-12-02 01:13:22,150 I 15 15] (raylet) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2023-12-02 01:13:22,546 I 15 15] (raylet) store_runner.cc:32: Allowing the Plasma store to use up to 0.0873628GB of memory.
[2023-12-02 01:13:22,547 I 15 15] (raylet) store_runner.cc:48: Starting object store with directory /dev/shm, fallback /tmp/ray, and huge page support disabled
[2023-12-02 01:13:22,691 I 15 172] (raylet) dlmalloc.cc:154: create_and_mmap_buffer(87425032, /dev/shm/plasmaXXXXXX)
[2023-12-02 01:13:22,692 W 15 172] (raylet) dlmalloc.cc:227: madvise(MADV_DONTDUMP) call failed: -1, Function not implemented
[2023-12-02 01:13:22,860 I 15 172] (raylet) store.cc:535: ========== Plasma store: =================
Current usage: 0 / 0.0873628 GB
[...]

This is looking better.

But we still have other issues while trying to bring up Ray under Gramine, which I will describe below.

These need troubleshooting on the Gramine side, and hopefully you / someone on Gramine-dev can help.

[gapisback]

The basic quest still remains:

What configuration do we need in the template file to allow Ray cluster's ray start to successfully create a cluster from inside Gramine-sgx.

Subsequent to above re-configuration, we played around with these configs in the template:

 # We limit the number of spawned threads via OMP_NUM_THREADS env variable.
-loader.env.OMP_NUM_THREADS = "4"
+loader.env.OMP_NUM_THREADS = "32"

-sys.stack.size = "2M"
+sys.stack.size = "8M"

Now gramine-sgx ./python /home/sgx/.local/bin/ray --logging-level debug start --head ... seems to go through for a (long ...) while to print out this 'seemingly-successful' message:

Dbg-Ray-CC: wait_for_node()  Found node_plasma_store_socket_name=/tmp/ray/session_2023-12-02_01-11-31_422392_1/sockets/plasma_store

/tmp/ray/session_2023-12-02_01-11-31_422392_1/sockets/plasma_store2023-12-02 01:13:58,726	SUCC scripts.py:781 -- --------------------
2023-12-02 01:13:58,745	SUCC scripts.py:782 -- Ray runtime started.
2023-12-02 01:13:58,745	SUCC scripts.py:783 -- --------------------
2023-12-02 01:13:58,746	INFO scripts.py:785 -- Next steps
2023-12-02 01:13:58,752	INFO scripts.py:788 -- To add another node to this Ray cluster, run
2023-12-02 01:13:58,753	INFO scripts.py:791 --   ray start --address='10.40.200.186:6379'
2023-12-02 01:13:58,753	INFO scripts.py:800 -- To connect to this Ray cluster:
2023-12-02 01:13:58,754	INFO scripts.py:802 -- import ray
2023-12-02 01:13:58,755	INFO scripts.py:803 -- ray.init()
2023-12-02 01:13:58,755	INFO scripts.py:834 -- To terminate the Ray runtime, run
2023-12-02 01:13:58,755	INFO scripts.py:835 --   ray stop
2023-12-02 01:13:58,755	INFO scripts.py:838 -- To view the status of the cluster, use
2023-12-02 01:13:58,755	INFO scripts.py:839 --   ray status
sgx-server:[110] $

But few seconds later, we get this very cryptic message from Gramine:

sgx-server:[110] $ [P13:T189:python3.8] error: process creation failed

From our triaging, it appears that some startup process coming up while the Ray cluster (and its sub-processes) are coming up is being killed.

But the message above does not indicate which process is being killed. We are unable to connect to this Ray cluster via ray status either outside Gramine or from inside gramine-sgx.

From outside Gramine, we get:

sgx-server:[110] $ ray status --address='10.40.200.186:6379'
No cluster status. It may take a few seconds for the Ray internal services to start up.

From inside gramine-sgx, I get:

sgx-server:[111] $ gramine-sgx ./python /home/sgx/.local/bin/ray status --address='10.40.200.186:6379'
Gramine is starting. Parsing TOML manifest file, this may take some time...
-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.file_check_policy = allow_all_but_log  (all files are passed through from untrusted host without verification)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Traceback (most recent call last):
  File "python/ray/_raylet.pyx", line 3022, in ray._raylet.check_health
  File "python/ray/_raylet.pyx", line 459, in ray._raylet.check_status
ray.exceptions.RpcError: failed to connect to all addresses; last error: UNKNOWN: ipv4:10.40.200.186:6379: connection attempt timed out before receiving SETTINGS frame
Ray cluster is not found at 10.40.200.186:6379

Would you / anyone else be able to help with tips to troubleshoot this further?

Thanks, again!

[dimakuv]

sgx-server:[110] $ [P13:T189:python3.8] error: process creation failed

This feels very much like the classic SGX case of "Out Of Memory, so the Linux kernel decides to kill SGX processes".

See discussions on this topic:

• https://gramine.readthedocs.io/en/stable/manifest-syntax.html#enclave-size
• https://gramine.readthedocs.io/en/stable/performance.html#multi-process-workloads
• Error in Pytorch.manifest: toml.decoder.TomlDecodeError: Duplicate keys! #233
• Can't create child processes with node.js inside gramine #600
• OOM with javascript + WebAssembly by Gramine in SGX mode #927
• 23. August 2022 #848
• Got error when use fork and exec to start up python executable inside gramine #1595

TLDR: You need to increase the amount of system memory (RAM + swap file size), or you need to decrease the amount of created sub-processes in your SGX application.