Ray cluster `ray start` fails to come up under gramine-sgx / gramine-direct. Fails with: thread_exit(): process 1 exited with status 0 #1680

gapisback · 2023-12-12T20:03:04Z

gapisback
Dec 12, 2023

HI, folks,

This post is a follow-on to the discussion thread #1664, to which I got some help / answers from @kailun-qin and @dimakuv . Thanks! for that. [ I had posted this to the gramine user's group but upon request by Dimitrii, I'm starting a new discussion thread here. ]

This post is about using Ray under gramine-sgx / gramine-direct.

I'm on gramine-SGX-V1: Gramine was built from commit: 4212a2525efffecbc787419ccf349299957b679f

Thanks to the help received from Kailun-Qin and Dmitrii, I am now past the basic config / template / memory resources issues. I am still searching for someone else out there who has successfully started a Ray Cluster under gramine-sgx.

The problem:

The Ray cluster never seems to come up successfully under gramine-sgx or gramine-direct. 'ray start' does seem to go through successfully [1] but soon immediately thereafter, the bootstrap process fails with this cryptic message:

[P1:T1:python3.8] libos_syscall_exit_group() -> do_process_exit: First time=1
[P1:T1:python3.8] do_process_exit() -> do_thread_exit(): process 1 exited with status 0
vsgx-vm:[43] $ [P12:T187:python3.8] libos_syscall_exit_group() -> do_process_exit: First time=1
[P13:T189:python3.8] libos_syscall_exit_group() -> do_process_exit: First time=1

The brief function names in above output is from debug instrumentation I added to figure out what's going on with the Gramine / libOS execution, leading to process[es] exiting.

In [2] below I have shown snippets of the Ray dashboard_agent.log file showing more diagnostic messages leading to the failure. These messages simply give more info about the retry attempts, and show that eventually the node crashes.

The thing relevant to Gramine in that log is this brief message: FileNotFoundError: [Errno 2] No such file or directory: '/proc/net/dev'.

Questions to Gramine-devs:

Do you know if /proc/net/dev is supported under gramine-sgx or gramine-dev? I did go thru the online docs, but cant' recall if this dev is supported.
On the same SGX-enabled box, I am able to bring up Ray cluster (ray start, ray status) directly without using gramine-sgx. Question is: has anyone in this group tried this exercise of integrating Ray under Gramine-sgx or gramine-direct?
Would it be possible for someone in your dev-/QA-team to try this integration out? And let me know whether you are able to get 'ray start' to work under gramine-sgx? it should be a fairly simple install of Ray s/w to get this working on some Linux box.

Digging further on the Ray-side:

I found these two threads. Some of this may be useful to Gramine-devs to help triage / troubleshoot the problems I am seeing. The signature of the problem I am seeing is exactly the same as the issues reported here:

Ray Issue-29412: [Ray Core] Ray agent getting killed unexpectedly

Which lead to a tentative code-fix in ray Python libraries,
Ray PR-29540: [Agent] Make agent shutdown more informative and graceful

The point of these two threads is that: Seems like there might have been some issue with Python library, psutil.Process.parent() misreporting that parent node is down, causing some cascading shutdowns on the Ray-side.

Question(s) are:

Can Gramine-devs speculate if such issues with node patrolling on Gramine-side, induced by some Python library hiccups could lead to 'ray start' totally aborting?
I was hoping someone would have tested out this integration on your end and put-up a nice tutorial on this page (Several other interesting integrations have been tried out.)

Given that Ray / ML/ Python workloads are becoming so very popular, I would have thought it would get some push from Gramine-CI/QA folks to try out this integration. And give us a helpful tutorial on how-to get this to work.

Thanks in advance, and thanks for reading this far. Any help / tips will be most graciously accepted.

--AdityA>

References

Messages seen booting up 'ray start' from under gramine-direct:

[P16:T257:python3.8] libos_init():511: Process ID=16: LibOS initialized
[P16:T257:python3.8] libos_syscall_exit_group() -> do_process_exit: First time=1
[P16:T257:python3.8] do_process_exit() -> do_thread_exit(): process 16 exited with status 255
2023-12-12 02:09:15,068	SUCC scripts.py:781 -- --------------------
2023-12-12 02:09:15,069	SUCC scripts.py:782 -- Ray runtime started.
2023-12-12 02:09:15,069	SUCC scripts.py:783 -- --------------------
2023-12-12 02:09:15,069	INFO scripts.py:785 -- Next steps
2023-12-12 02:09:15,069	INFO scripts.py:788 -- To add another node to this Ray cluster, run
2023-12-12 02:09:15,070	INFO scripts.py:791 --   ray start --address='[10.208.196.155:6379](http://10.208.196.155:6379/)'
2023-12-12 02:09:15,070	INFO scripts.py:800 -- To connect to this Ray cluster:
2023-12-12 02:09:15,071	INFO scripts.py:802 -- import ray
2023-12-12 02:09:15,071	INFO scripts.py:803 -- ray.init()
2023-12-12 02:09:15,071	INFO scripts.py:834 -- To terminate the Ray runtime, run
2023-12-12 02:09:15,071	INFO scripts.py:835 --   ray stop
2023-12-12 02:09:15,071	INFO scripts.py:838 -- To view the status of the cluster, use
2023-12-12 02:09:15,071	INFO scripts.py:839 --   ray status

The above lines indicate that 'ray start' did go through cleanly ... albeit very briefly. Immediately after these 'successful' start-up messages, we get that message from Ray about 'process exiting' (presumably because either it received some kill, or there was some faulty logic in detecting if the parent process is up).

Snippets of messages from Ray's dashboard_agent.log:

These messages show "the issue" with Python psutil.Process.parent() not being able to 'locate' a parent node.

43 2023-12-11 22:22:20,007»INFO http_server_agent.py:78 -- <ResourceRoute [OPTIONS] <StaticResource  /logs -> PosixPath('/tmp/ray/session_2023-12-11_22-22-10_412945_1/logs')> -> <bound method _PreflightHandler._preflight_handler of <aiohttp_cors.cors_c    onfig._CorsConfigImpl object at 0x2d4742bbf670>>
 44 2023-12-11 22:22:20,007»INFO http_server_agent.py:79 -- Registered 30 routes.
 45 2023-12-11 22:22:20,012»INFO process_watcher.py:44 -- raylet pid is 15
 46 2023-12-11 22:22:20,012»WARNING process_watcher.py:89 -- Raylet is considered dead 1 X. If it reaches to 5, the agent will kill itself. Parent: None, parent_gone: True, init_assigned_for_parent: False, parent_changed: False.
 47 2023-12-11 22:22:20,016»INFO event_agent.py:56 -- Report events to [10.208.196.155:45899](http://10.208.196.155:45899/)
 48 2023-12-11 22:22:20,017»INFO event_utils.py:132 -- Monitor events logs modified after 1702331539.842946 on /tmp/ray/session_2023-12-11_22-22-10_412945_1/logs/events, the source types are all.
 49 2023-12-11 22:22:20,019»ERROR reporter_agent.py:1149 -- Error publishing node physical stats.
 50 Traceback (most recent call last):
 51   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1132, in _perform_iteration
 52     stats = self._get_all_stats()
 53   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 630, in _get_all_stats
 54     network_stats = self._get_network_stats()
 55   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 434, in _get_network_stats
 56     v for k, v in psutil.net_io_counters(pernic=True).items() if k[0] == "e"
 57   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/__init__.py", line 2122, in net_io_counters
 58     rawdict = _psplatform.net_io_counters()
 59   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/_pslinux.py", line 1023, in net_io_counters
 60     with open_text("%s/net/dev" % get_procfs_path()) as f:
 61   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/_common.py", line 786, in open_text
 62     fobj = open(fname, buffering=FILE_READ_BUFFER_SIZE,
 63 FileNotFoundError: [Errno 2] No such file or directory: '/proc/net/dev'
 64 2023-12-11 22:22:20,414»WARNING process_watcher.py:89 -- Raylet is considered dead 2 X. If it reaches to 5, the agent will kill itself. Parent: None, parent_gone: True, init_assigned_for_parent: False, parent_changed: False.
 65 2023-12-11 22:22:20,816»WARNING process_watcher.py:89 -- Raylet is considered dead 3 X. If it reaches to 5, the agent will kill itself. Parent: None, parent_gone: True, init_assigned_for_parent: False, parent_changed: False.
 66 2023-12-11 22:22:21,217»WARNING process_watcher.py:89 -- Raylet is considered dead 4 X. If it reaches to 5, the agent will kill itself. Parent: None, parent_gone: True, init_assigned_for_parent: False, parent_changed: False.
 67 2023-12-11 22:22:21,618»WARNING process_watcher.py:89 -- Raylet is considered dead 5 X. If it reaches to 5, the agent will kill itself. Parent: None, parent_gone: True, init_assigned_for_parent: False, parent_changed: False.
 68 2023-12-11 22:22:21,618»INFO agent.py:227 -- Terminated Raylet: ip=10.208.196.155, node_id=5ea8e24111ca76d5135365a6bea0e7da046378d411339af896a80d4f.·
 69 2023-12-11 22:22:21,619»ERROR process_watcher.py:142 -- Raylet is terminated. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Oth    er termination signals. Last 20 lines of the Raylet logs:
 70     [state-dump] Event stats:
 71     [state-dump] »······PeriodicalRunner.RunFnPeriodically - 11 total (2 active, 1 running), CPU time: mean = 599.819 us, total = 6.598 ms
 72     [state-dump] »······NodeManager.ScheduleAndDispatchTasks - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
 73     [state-dump] »······NodeManager.deadline_timer.record_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
 74     [state-dump] »······NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
 75     [state-dump] »······NodeManager.GCTaskFailureReason - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s

kailun-qin · 2023-12-13T05:34:48Z

kailun-qin
Dec 13, 2023
Maintainer

The thing relevant to Gramine in that log is this brief message: FileNotFoundError: [Errno 2] No such file or directory: '/proc/net/dev'.
Do you know if /proc/net/dev is supported under gramine-sgx or gramine-dev? I did go thru the online docs, but cant' recall if this dev is supported.
Can Gramine-devs speculate if such issues with node patrolling on Gramine-side, induced by some Python library hiccups could lead to 'ray start' totally aborting?

First, I think the issue is not directly related to /proc/net/dev, though Gramine indeed does not emulate it. The shown /proc/net/dev error was generated by reporter_agent.py, which was acutally a result of psutil.Process.parent() not being able to "locate" a parent node by process_watcher.py, as you noticed. And this was clearly shown in the log: "Parent: None, parent_gone: True, init_assigned_for_parent: False, parent_changed: False". The corresponding logic in process_watcher.py is here.

For Ray to (temporarily) work, have you tried the workaround proposed here by setting RAY_DASHBOARD_AGENT_CHECK_PARENT_INTERVAL_S=100000000 which disables the healthcheck? Note that from the provided discussions, psutil.Process.parent() may also work unexpectedly in other environment and the Ray community is also considering an alternative healthcheck mechanism.

To analyze the issue, I think we could debug into psutil.Process.parent() (maybe only using some simple standalone examples first, i.e., w/o Ray) and see what really happens there.

Would it be possible for someone in your dev-/QA-team to try this integration out? And let me know whether you are able to get 'ray start' to work under gramine-sgx? it should be a fairly simple install of Ray s/w to get this working on some Linux box.
I was hoping someone would have tested out this integration on your end and put-up a nice tutorial on this page (Several other interesting integrations have been tried out.)

Maybe you could share a little more about your use case and/or motivation (i.e., why Ray in SGX LibOS, as a random try out or any specific scenario)? I'm personally interested in having this integrated, though not in CI-Examples, maybe in our examples repo or at least in contrib repo.

5 replies

gapisback Dec 13, 2023
Author

Thank you, @kailun-qin -- for troubleshooting this alongside me.

I have also seen these code fragments you point out. I have a post into Ray support group asking for knobs to turn OFF health-check monitoring.

I am not entirely sure whether this is the right sequence of events:

The shown /proc/net/dev error was generated by reporter_agent.py, which was acutally a result of psutil.Process.parent() not being able to "locate" a parent node by process_watcher.py, as you noticed.

Am investigating into Ray's machinery to figure out how process_watcher.py works and if it is independent of reporter_agent.py.

Btw, I did add some patch code to reporter PY module and once I suppressed the network IO checks, I next ran into this limitation. (Brief snippets of errors seen on stdout while Ray is booting up under Gramine ...)

 47 2023-12-13 03:08:54,755»INFO event_agent.py:56 -- Report events to 10.208.196.155:44593
 48 2023-12-13 03:08:54,755»INFO event_utils.py:132 -- Monitor events logs modified after 1702435134.608772 on /tmp/ray/session_2023-12-13_03-08-42_327199_1/logs/events, the source types     are all.
 49 2023-12-13 03:08:54,757»ERROR reporter_agent.py:1149 -- Error publishing node physical stats.
 50 Traceback (most recent call last):
 51   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 1132, in _perform_iteration
 52     stats = self._get_all_stats()
 53   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 634, in _get_all_stats
 54     disk_stats = self._get_disk_io_stats()
 55   File "/home/sgx/.local/lib/python3.8/site-packages/ray/dashboard/modules/reporter/reporter_agent.py", line 468, in _get_disk_io_stats
 56     stats = psutil.disk_io_counters()
 57   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/__init__.py", line 2072, in disk_io_counters
 58     rawdict = _psplatform.disk_io_counters(**kwargs)
 59   File "/home/sgx/.local/lib/python3.8/site-packages/ray/thirdparty_files/psutil/_pslinux.py", line 1161, in disk_io_counters
 60     raise NotImplementedError(
 61 NotImplementedError: /proc/diskstats nor /sys/block filesystem are available on this system

I think Ray's monitoring sub-systems are tapping into metrics from /proc/diskstats, /sys/block filesystem.

Can you confirm if these devices are supported through gramine-sgx? From above messages, it seems these are not supported through gramine-sgx.

Still troubleshooting ...

dimakuv Dec 14, 2023

NotImplementedError: /proc/diskstats nor /sys/block filesystem are available on this system

Yes, this devices are not supported in Gramine.

We could add dummy implementations for one of them, hopefully this can unblock Ray to proceed further.

gapisback Dec 15, 2023
Author

HI, @dimakuv -- From my investigation, Ray's monitoring module is tapping into metrics from [ at least ] 3 end-points: /proc/net/dev, /proc/diskstats and /sys/block.

(For now) I patched the Ray's monitoring package to only invoke these monitoring if an env-var, GRAMINE_GET_STATS is true. And I futzed my python.manifest.template to temporarily set this env-var to "0", which turns OFF this bit of Ray's monitoring code.

It helps me get past current issues at the cost of disabling device / network stats -- which is ok for now, in my integration-POC.

I suspect your suggestion of dummy implementations may only go so far to make things somewhat functional. I think other Python packages (like Ray's monitoring) would naturally want to tap into run-time device / network metrics, in which case, seems like a fully functional interface to return stats from these device handles would become necessary.

I think Gramine's dev team will have to re-consider what the "right" approach would be in the longer-term for these monitoring interfaces.

gapisback Dec 15, 2023
Author

HI, @kailun-qin : Re: Maybe you could share a little more about your use case and/or motivation (i.e., why Ray in SGX LibOS, as a random try out or any specific scenario)?

I am trying to assess the technical challenges to integrate Ray (which is an open-source AI/ML package) with Gramine-SGX secure enclaves.

The objective is to eventually understand how we can further the use of Ray, running under Gramine-SGX, integrated with the Certifier Framework, an Open-Source software package that supports multi-vendor Confidential Computing platforms.

kailun-qin Dec 18, 2023
Maintainer

Great! Thanks for all the context info! @gapisback

gapisback · 2023-12-15T12:59:35Z

gapisback
Dec 15, 2023
Author

HI, @kailun-qin , @dimakuv : Re: To analyze the issue, I think we could debug into psutil.Process.parent() (maybe only using some giampaolo/psutil#1905 first, i.e., w/o Ray) and see what really happens there.

I did a small experiment to see if even basic use of psutil.Process() will work correctly under Gramine-SGX.

Here is a very small repro script: ./test_sgx_psutil_simple.py

#!/usr/bin/env python3
import sys
import psutil

curr_proc = psutil.Process()
curr_pid = curr_proc.pid
print(f'curr_proc: {curr_proc}, curr_proc: pid = {curr_proc.pid}')

parent_proc = curr_proc.parent()
if parent_proc is None:
    print('Warning! parent_proc handle not found. Exiting!')
    sys.exit(1)
else:
    parent_pid = parent_proc.pid
    print(f'curr_proc: parent_pid = {parent_pid}, parent_proc = {parent_proc}')
    sys.exit(0)

Standalone output: (Without Gramine-SGX)

vsgx-vm:[99] $ ./test_sgx_psutil_simple.py
curr_proc: psutil.Process(pid=898456, name='python3', status='running', started='07:50:31'), curr_proc: pid = 898456
curr_proc: parent_pid = 880575, parent_proc = psutil.Process(pid=880575, name='bash', status='sleeping', started='20:48:44')

vsgx-vm:[100] $ echo $?
0

Output when run under gramine-sgx (same behaviour seen under gramine-direct)

vsgx-vm:[27] $ gramine-sgx ./python scripts/test_sgx_psutil_simple.py
Gramine is starting. Parsing TOML manifest file, this may take some time...

-----------------------------------------------------------------------------------------------------------------------
Gramine detected the following insecure configurations:

  - sgx.debug = true                           (this is a debug enclave)
  - loader.insecure__use_cmdline_argv = true   (forwarding command-line args from untrusted host to the app)
  - sgx.file_check_policy = allow_all_but_log  (all files are passed through from untrusted host without verification)
  - sgx.allowed_files = [ ... ]                (some files are passed through from untrusted host without verification)

Gramine will continue application execution, but this configuration must not be used in production!
-----------------------------------------------------------------------------------------------------------------------

Gramine was built from commit: 4212a2525efffecbc787419ccf349299957b679f
[...]
[P1:T1:python3.8] libos_init():511: Process ID=1: LibOS initialized
curr_proc: psutil.Process(pid=1, name='python3.8', status='running', started='1970-01-01 00:00:00'), curr_proc: pid = 1
Warning! parent_proc handle not found. Exiting!

vsgx-vm:[28] $ echo $?
1

What I am finding is that basic usage of curr_proc = psutil.Process() followed by parent_proc = curr_proc.parent() fails to return a proc-handle for the parent-proc, when this is run under gramine-sgx/-direct.

I may be making very silly pilot errors in my test repro, but I think it is a fair expectation that Python's curr_proc.parent() should return something valid even under Gramine. No?

Kindly take a look at this repro test-case to see if my expectations of a valid return from parent_proc = curr_proc.parent() are incorrect.

This may, well-be, the issue that Ray's code here (ray/_private/process_watcher.py) is running into when Ray is run under Gramine.

It may be that we have to 'handle' this on Ray's-side by not falling over and dying immediately. But that would mask real errors that Ray's code is written to handle the case of not-finding-parent-proc scenario.

Your thoughts?

5 replies

dimakuv Dec 18, 2023

@gapisback Thanks for the reproducer and very good descriptions!

I'd like us (the Gramine team) to look into these issues. To not forget about this, I added the bug and Priority 1 labels to this discussion.

@kailun-qin You mentioned that you're interested in Ray integration. Is this something (the above issues detected by @gapisback) that you want to investigate?

kailun-qin Dec 18, 2023
Maintainer

You mentioned that you're interested in Ray integration. Is this something that you want to investigate?

Sure, I'll take some time to look into this.

kailun-qin Dec 19, 2023
Maintainer

Ok, I looked a bit closer into the related code snippets.

First of all, I think the basic usage of curr_proc = psutil.Process() followed by parent_proc = curr_proc.parent() (as shown in the repro script) may not directly reflect the problem met in Ray. The failed case in Ray happened as follows: in every Ray node, there's a Raylet; and Raylet spawns a child process called "agent" which will check if parent (Raylet) is dead via psutil.parent() and fate shares (i.e., when one of the agent or Raylet is dead, the other is dead) -- this was described in ray-project/ray#29412. So the failed parent checking logic is against Raylet (which should indeed exists) whereas the repro script is checking against the parent of Gramine first/main/PID 1 process (which actually does not exist).

Second, concerning the issue itself, note that psutil.parent() has the following logic which will check the calling process's PID against the lowest PID obtained from /proc (see here and here) -- and if they're the same -- None is returned (i.e., psutil assmues in such case it has no parent). However in Gramine, we have the below limitation in procfs:

gramine/libos/src/fs/proc/fs.c

Lines 95 to 96 in 5d38bb1

    
           /* Listing remote processes is not currently supported. */ 
        
           ipc_thread_pid->list_names = NULL;

.
This means when a child process does sth. like getdents64() on /proc, this will not list the parent's PID:

gramine/libos/src/fs/libos_fs_pseudo.c

Lines 232 to 243 in 5d38bb1

    
           LISTP_FOR_EACH_ENTRY(child_node, &node->dir.children, siblings) { 
        
               if (child_node->name) { 
        
                   /* If `name_exists` callback is provided, check it. */ 
        
                   if (!child_node->name_exists || node->name_exists(dent, child_node->name)) 
        
                       nlink++; 
        
               } 
        
               if (child_node->list_names) { 
        
                   int ret = child_node->list_names(dent, &count_nlink, &nlink); 
        
                   if (ret < 0) 
        
                       return ret; 
        
               } 
        
           }

. Thus using psutil.parent() in child will probably get the lowest PID equal to its PID (as the parent PID is skipped) and no proc-handle of the parent will be returned.

gapisback Dec 19, 2023
Author

Thank you both, @kailun-qin and @dimakuv.

After some further internal brainstorming on these use-cases, our suspicion is that it's probably an expected behaviour that the use of curr_proc = psutil.Process() followed by parent_proc = curr_proc.parent() fails to return a proc-handle for the parent-proc, when this is run under gramine-sgx/-direct.

Our theory is that likely gramine-sgx libOS is starting this process as-if it's being started under init() process (under Linux OS). That's why, probably, there is no parent found for this process.

There is no further parent process, unlike the bash process that shows up as the parent when this repro is run outside gramine-sgx.

[P1:T1:python3.8] libos_init():511: Process ID=1: LibOS initialized
curr_proc: psutil.Process(pid=1, name='python3.8', status='running', started='1970-01-01 00:00:00'), curr_proc: pid = 1
Warning! parent_proc handle not found. Exiting!

Does this theory make sense; i.e., does gramine-sgx libOS start the python3 process like as-if it were started under Linux' init process?

gapisback Dec 19, 2023
Author

Thanks, @kailun-qin -- for your analysis. Our posts, coincidentally, crossed.

I will re-look at building a repro closer to how Ray / raylet / agent sub-process does this polling using psutil() interfaces. It may take a few days to turn this around, due to the holidays and some time-off.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ray cluster `ray start` fails to come up under gramine-sgx / gramine-direct. Fails with: thread_exit(): process 1 exited with status 0 #1680

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Ray cluster ray start fails to come up under gramine-sgx / gramine-direct. Fails with: thread_exit(): process 1 exited with status 0 #1680

gapisback Dec 12, 2023

The problem:

Digging further on the Ray-side:

References

Replies: 2 comments · 10 replies

kailun-qin Dec 13, 2023 Maintainer

gapisback Dec 13, 2023 Author

dimakuv Dec 14, 2023

gapisback Dec 15, 2023 Author

gapisback Dec 15, 2023 Author

kailun-qin Dec 18, 2023 Maintainer

gapisback Dec 15, 2023 Author

Here is a very small repro script: ./test_sgx_psutil_simple.py

Standalone output: (Without Gramine-SGX)

Output when run under gramine-sgx (same behaviour seen under gramine-direct)

dimakuv Dec 18, 2023

kailun-qin Dec 18, 2023 Maintainer

kailun-qin Dec 19, 2023 Maintainer

gapisback Dec 19, 2023 Author

gapisback Dec 19, 2023 Author

Ray cluster `ray start` fails to come up under gramine-sgx / gramine-direct. Fails with: thread_exit(): process 1 exited with status 0 #1680

gapisback
Dec 12, 2023

Replies: 2 comments 10 replies

kailun-qin
Dec 13, 2023
Maintainer

gapisback Dec 13, 2023
Author

gapisback Dec 15, 2023
Author

gapisback Dec 15, 2023
Author

kailun-qin Dec 18, 2023
Maintainer

gapisback
Dec 15, 2023
Author

kailun-qin Dec 18, 2023
Maintainer

kailun-qin Dec 19, 2023
Maintainer

gapisback Dec 19, 2023
Author

gapisback Dec 19, 2023
Author