Realm: Optimize gather copies in Moya/FleCSI #1733

apryakhin · 2024-07-30T15:56:41Z

We have been working with the LANL team on Moay, which is an unstructured multimaterial Lagrangian hydrodynamics application. The application performs a number of gather operations of various types (to be determined), where most of them involve gathering data into framebuffer memory. It has been observed that the performance of these operations is sub-optimal

I am filing an issue to track the progress of this work. The application is run in settings of 1, 2, 4, and 8 nodes, with 4 GPUs per node. In addition, a hang is observed when running the application with 8 nodes.

The expectation is that it should be possible to leverage faster cuda-dma gather path in Realm to improve the timing.

@jpietarilagraham please fill-on more details. @lightsighter for visibility Thanks

jpietarilagraham · 2024-08-08T19:43:57Z

Backtrace for rank 0

(gdb) thread apply * bt

Thread 1 (Thread 0x14c5934be000 (LWP 95177) "moya_3d"):
#0  0x000014c59e32570c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014c59efae3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014c59efa592a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014c59efa571c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c5a02d12c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014c5a16ef590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7fffffcde418) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14c58c07b000 (LWP 95223) "cuda-EvtHandlr"):
#0  0x000014c5984ef1c9 in poll () from /lib64/libc.so.6
#1  0x000014c599964b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014c599a0bd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014c59995fb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014c59e31e6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014c5984fba6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14c568361000 (LWP 95235) "cuda-EvtHandlr"):
#0  0x000014c5984ef1c9 in poll () from /lib64/libc.so.6
#1  0x000014c599964b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014c599a0bd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014c59995fb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014c59e31e6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014c5984fba6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14c56dbf9000 (LWP 95317) "moya_3d"):
#0  0x000014c5984f4ea9 in syscall () from /lib64/libc.so.6
#1  0x000014c59f0f5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014c59efa0faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014c59f06b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c59e31e6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014c5984fba6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14c568160000 (LWP 95323) "moya_3d"):
#0  ofi_spin_unlock_op (lock=0x10b0e48) at ./include/ofi_lock.h:187
#1  0x000014c59e9e9107 in ofi_genlock_unlock (lock=0x10b0e40) at ./include/ofi_lock.h:364
#2  ofi_cq_readfrom (cq_fid=0x10b0dd0, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:231
#3  0x000014c59f504a5a in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c59f504d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014c59f4f9f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014c59f0d776e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014c59efa14e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014c59efa0e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014c59f06b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x000014c59e31e6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#11 0x000014c5984fba6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14c58801b000 (LWP 95332) "moya_3d"):
#0  0x000014c5984f4ea9 in syscall () from /lib64/libc.so.6
#1  0x000014c59f0f5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014c59f06333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014c59f062e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c59f06d73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014c598445600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14c58269f000 (LWP 95334) "moya_3d"):
#0  0x000014c5984f4ea9 in syscall () from /lib64/libc.so.6
#1  0x000014c59f0f5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014c59f06333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014c59f062e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c59f06d73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014c598445600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14c560dfd000 (LWP 95336) "moya_3d"):
#0  0x000014c5984f4ea9 in syscall () from /lib64/libc.so.6
#1  0x000014c59f0f5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x000014c59f06333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014c59f062e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c59f067857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014c59f06b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014c59e31e6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014c5984fba6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14c5605e9000 (LWP 95340) "moya_3d"):
#0  0x000014c5984f4ea9 in syscall () from /lib64/libc.so.6
#1  0x000014c59f0f5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014c59f0f6986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014c59f07a7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014c59f06b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014c59e31e6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014c5984fba6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T19:44:39Z

Backtrace for rank1

(gdb) thread apply * bt

Thread 1 (Thread 0x14d940af1000 (LWP 95178) "moya_3d"):
#0  0x000014d94b95870c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014d94c5e13e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d94c5d892a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d94c5d871c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94d9042c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014d94ed22590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffd194e1f68) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14d9396ae000 (LWP 95220) "cuda-EvtHandlr"):
#0  0x000014d945b221c9 in poll () from /lib64/libc.so.6
#1  0x000014d946f97b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014d94703ed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014d946f92b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014d94b9516ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014d945b2ea6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14d915864000 (LWP 95233) "cuda-EvtHandlr"):
#0  0x000014d945b221c9 in poll () from /lib64/libc.so.6
#1  0x000014d946f97b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014d94703ed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014d946f92b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014d94b9516ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014d945b2ea6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14d91b5f9000 (LWP 95319) "moya_3d"):
#0  0x000014d945b27ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d94c728875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d94c5d3faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d94c69e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94b9516ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014d945b2ea6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14d915463000 (LWP 95321) "moya_3d"):
#0  0x000014d94b9599b3 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x000014d94c01c11b in ofi_genlock_lock (lock=0x10b7400) at ./include/ofi_lock.h:359
#2  ofi_cq_readfrom (cq_fid=0x10b7390, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:233
#3  0x000014d94cb37b36 in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94cb37d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d94cb2cf9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014d94c70a76e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014d94c5d44e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014d94c5d3e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014d94c69e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x000014d94b9516ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#11 0x000014d945b2ea6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14d9300f2000 (LWP 95327) "moya_3d"):
#0  0x000014d945b27ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d94c728875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d94c69633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d94c695e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94c6a073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d945a78600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14d9300d7000 (LWP 95328) "moya_3d"):
#0  0x000014d945b27ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d94c728875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d94c69633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d94c695e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94c6a073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d945a78600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14d91434e000 (LWP 95329) "moya_3d"):
#0  0x000014d945b27ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d94c728875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x000014d94c69633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d94c695e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94c69a857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d94c69e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014d94b9516ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014d945b2ea6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14d91413b000 (LWP 95341) "moya_3d"):
#0  0x000014d945b27ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d94c728875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d94c729986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d94c6ad7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d94c69e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d94b9516ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014d945b2ea6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T19:45:45Z

rank 2 backtrace

(gdb) thread apply * bt

Thread 1 (Thread 0x147d7f7a1000 (LWP 95179) "moya_3d"):
#0  0x0000147d8a60870c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000147d8b2913e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000147d8b28892a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000147d8b28871c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000147d8c5b42c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x0000147d8d9d2590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7fffeb6a7e68) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x147d7835e000 (LWP 95213) "cuda-EvtHandlr"):
#0  0x0000147d847d21c9 in poll () from /lib64/libc.so.6
#1  0x0000147d85c47b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000147d85ceed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000147d85c42b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000147d8a6016ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000147d847dea6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x147d54562000 (LWP 95232) "cuda-EvtHandlr"):
#0  0x0000147d847d21c9 in poll () from /lib64/libc.so.6
#1  0x0000147d85c47b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000147d85ceed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000147d85c42b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000147d8a6016ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000147d847dea6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x147d5a3f9000 (LWP 95320) "moya_3d"):
#0  0x0000147d8aca9c30 in pthread_spin_lock@plt () from /opt/cray/libfabric/1.15.2.0/lib64/libfabric.so.1
#1  0x0000147d8acf04d0 in cxip_cq_progress (cq=0x10b95d0) at prov/cxi/src/cxip_cq.c:545
#2  0x0000147d8acf0c79 in cxip_util_cq_progress (util_cq=0x10b95d0) at prov/cxi/src/cxip_cq.c:563
#3  0x0000147d8accc111 in ofi_cq_readfrom (cq_fid=0x10b95d0, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:232
#4  0x0000147d8b7e7c16 in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000147d8b7e7d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x0000147d8b7dcf9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x0000147d8b3ba76e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x0000147d8b2844e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x0000147d8b283e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x0000147d8b34e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#11 0x0000147d8a6016ea in start_thread () from /lib64/libpthread.so.0
#12 0x0000147d847dea6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x147d54161000 (LWP 95324) "moya_3d"):
#0  0x0000147d847d7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000147d8b3d8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000147d8b283faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000147d8b34e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024---Type <RET> for more, q to quit, c to continue without paging--
04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000147d8a6016ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000147d847dea6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x147d700b6000 (LWP 95335) "moya_3d"):
#0  0x0000147d847d7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000147d8b3d8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000147d8b34633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000147d8b345e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000147d8b35073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000147d84728600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x147d7009b000 (LWP 95337) "moya_3d"):
#0  0x0000147d847d7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000147d8b3d8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000147d8b34633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000147d8b345e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000147d8b35073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000147d84728600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x146b3cc7c000 (LWP 95338) "moya_3d"):
#0  0x0000147d847d7ea9 in syscall () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--
#1  0x0000147d8b3d8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000147d8b34633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000147d8b345e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000147d8b34a857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000147d8b34e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x0000147d8a6016ea in start_thread () from /lib64/libpthread.so.0
#7  0x0000147d847dea6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x146b3c468000 (LWP 95342) "moya_3d"):
#0  0x0000147d847d7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000147d8b3d8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000147d8b3d9986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000147d8b35d7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000147d8b34e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000147d8a6016ea in start_thread () from /lib64/libpthread.so.0
#6  0x0000147d847dea6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T19:46:24Z

rank 3 backtrace

(gdb) thread apply * bt

Thread 1 (Thread 0x149b5f671000 (LWP 95180) "moya_3d"):
#0  0x0000149b6a4d870c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000149b6b1613e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b15892a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6b15871c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000149b6c4842c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x0000149b6d8a2590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffe7dbbba78) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x149b5822e000 (LWP 95216) "cuda-EvtHandlr"):
#0  0x0000149b646a21c9 in poll () from /lib64/libc.so.6
#1  0x0000149b65b17b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000149b65bbed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000149b65b12b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000149b6a4d16ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000149b646aea6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x149b34361000 (LWP 95234) "cuda-EvtHandlr"):
#0  0x0000149b646a21c9 in poll () from /lib64/libc.so.6
#1  0x0000149b65b17b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000149b65bbed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000149b65b12b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000149b6a4d16ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000149b646aea6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x149b3e7fb000 (LWP 95318) "moya_3d"):
#0  0x0000149b6b154518 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x0000149b6b153e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b21e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6a4d16ea in start_thread () from /lib64/libpthread.so.0
#4  0x0000149b646aea6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x149b3a3f9000 (LWP 95322) "moya_3d"):
#0  0x0000149b646a7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000149b6b2a8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b153faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6b21e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000149b6a4d16ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000149b646aea6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x149b5005a000 (LWP 95330) "moya_3d"):
#0  0x0000149b646a7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000149b6b2a8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b21633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6b215e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000149b6b22073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/s--Type <RET> for more, q to quit, c to continue without paging--
ystems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000149b645f8600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x149b5003f000 (LWP 95331) "moya_3d"):
#0  0x0000149b646a7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000149b6b2a8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b21633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6b215e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000149b6b22073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000149b645f8600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x149b2cdfd000 (LWP 95333) "moya_3d"):
#0  0x0000149b646a7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000149b6b2a8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b21633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6b215e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000149b6b21a857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000149b6b21e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x0000149b6a4d16ea in start_thread () from /lib64/libpthread.so.0
#7  0x0000149b646aea6f in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 9 (Thread 0x149b34160000 (LWP 95339) "moya_3d"):
#0  0x0000149b646a7ea9 in syscall () from /lib64/libc.so.6
#1  0x0000149b6b2a8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000149b6b2a9986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000149b6b22d7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000149b6b21e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000149b6a4d16ea in start_thread () from /lib64/libpthread.so.0
#6  0x0000149b646aea6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T21:50:15Z

rank 4 backtrace

(gdb) thread apply * bt

Thread 1 (Thread 0x150af9fbe000 (LWP 84536) "moya_3d"):
#0  0x0000150b04e2570c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000150b05aae3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05aa592a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05aa571c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b06dd12c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x0000150b081ef590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7fff807989e8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x150af2b7b000 (LWP 84565) "cuda-EvtHandlr"):
#0  0x0000150afefef1c9 in poll () from /lib64/libc.so.6
#1  0x0000150b00464b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000150b0050bd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000150b0045fb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000150b04e1e6ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000150afeffba6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x150acec65000 (LWP 84601) "cuda-EvtHandlr"):
#0  0x0000150afefef1c9 in poll () from /lib64/libc.so.6
#1  0x0000150b00464b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000150b0050bd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000150b0045fb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000150b04e1e6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000150afeffba6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x150ad47f9000 (LWP 84664) "moya_3d"):
#0  0x0000150afeff4ea9 in syscall () from /lib64/libc.so.6
#1  0x0000150b05bf5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05aa0faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05b6b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b04e1e6ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000150afeffba6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x150ace864000 (LWP 84669) "moya_3d"):
#0  0x0000150b06004e1a in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x0000150b05ff9f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05bd776e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05aa14e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b05aa0e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150b05b6b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x0000150b04e1e6ea in start_thread () from /lib64/libpthread.so.0
#7  0x0000150afeffba6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x150af00de000 (LWP 84671) "moya_3d"):
#0  0x0000150afeff4ea9 in syscall () from /lib64/libc.so.6
#1  0x0000150b05bf5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems--Type <RET> for more, q to quit, c to continue without paging--
/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05b6333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05b62e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b05b6d73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150afef45600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x150af00c3000 (LWP 84674) "moya_3d"):
#0  0x0000150afeff4ea9 in syscall () from /lib64/libc.so.6
#1  0x0000150b05bf5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05b6333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05b62e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b05b6d73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150afef45600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x150acd74f000 (LWP 84678) "moya_3d"):
#0  0x0000150afeff4ea9 in syscall () from /lib64/libc.so.6
#1  0x0000150b05bf5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05b6333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05b62e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b05b67857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTas--Type <RET> for more, q to quit, c to continue without paging--
kScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150b05b6b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x0000150b04e1e6ea in start_thread () from /lib64/libpthread.so.0
#7  0x0000150afeffba6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x150accf3b000 (LWP 84683) "moya_3d"):
#0  0x0000150afeff4ea9 in syscall () from /lib64/libc.so.6
#1  0x0000150b05bf5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000150b05bf6986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000150b05b7a7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000150b05b6b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150b04e1e6ea in start_thread () from /lib64/libpthread.so.0
#6  0x0000150afeffba6f in clone () from /lib64/libc.so.6
(gdb) 
(gdb)

jpietarilagraham · 2024-08-08T21:51:36Z

backtrace 5

(gdb) thread apply * bt

Thread 1 (Thread 0x14b69e02b000 (LWP 84537) "moya_3d"):
#0  0x000014b6a8e9270c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014b6a9b1b3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9b1292a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9b1271c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6aae3e2c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014b6ac25c590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffd1a4bdfd8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14b696be8000 (LWP 84554) "cuda-EvtHandlr"):
#0  0x000014b6a305c1c9 in poll () from /lib64/libc.so.6
#1  0x000014b6a44d1b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014b6a4578d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014b6a44ccb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014b6a8e8b6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014b6a3068a6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14b673073000 (LWP 84603) "cuda-EvtHandlr"):
#0  0x000014b6a305c1c9 in poll () from /lib64/libc.so.6
#1  0x000014b6a44d1b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014b6a4578d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014b6a44ccb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014b6a8e8b6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014b6a3068a6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14b672c72000 (LWP 84665) "moya_3d"):
#0  0x000014b6aa073233 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x000014b6aa066f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9c4476e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9b0e4e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6a9b0de98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014b6a9bd886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014b6a8e8b6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014b6a3068a6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14b672b5f000 (LWP 84670) "moya_3d"):
#0  0x000014b6a3061ea9 in syscall () from /lib64/libc.so.6
#1  0x000014b6a9c62875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9b0dfaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9bd886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6a8e8b6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014b6a3068a6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14b691228000 (LWP 84680) "moya_3d"):
#0  0x000014b6a3061ea9 in syscall () from /lib64/libc.so.6
#1  0x000014b6a9c62875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems--Type <RET> for more, q to quit, c to continue without paging--
/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9bd033c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9bcfe2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6a9bda73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014b6a2fb2600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14b67c0e7000 (LWP 84681) "moya_3d"):
#0  0x000014b6a3061ea9 in syscall () from /lib64/libc.so.6
#1  0x000014b6a9c62875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9bd033c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9bcfe2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6a9bda73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014b6a2fb2600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14b671a4a000 (LWP 84682) "moya_3d"):
#0  0x000014b6a3061ea9 in syscall () from /lib64/libc.so.6
#1  0x000014b6a9c62875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9bd033c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9bcfe2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6a9bd4857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTas--Type <RET> for more, q to quit, c to continue without paging--
kScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014b6a9bd886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014b6a8e8b6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014b6a3068a6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14b671236000 (LWP 84686) "moya_3d"):
#0  0x000014b6a3061ea9 in syscall () from /lib64/libc.so.6
#1  0x000014b6a9c62875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014b6a9c63986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014b6a9be77d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014b6a9bd886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014b6a8e8b6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014b6a3068a6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T21:54:53Z

backtrace 6

(gdb) thread apply * bt

Thread 1 (Thread 0x14d31b7dd000 (LWP 84538) "moya_3d"):
#0  0x000014d32664470c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014d3272cd3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d3272c492a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d3272c471c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d3285f02c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014d329a0e590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffdb2acd4b8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14d31439a000 (LWP 84568) "cuda-EvtHandlr"):
#0  0x000014d32080e1c9 in poll () from /lib64/libc.so.6
#1  0x000014d321c83b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014d321d2ad7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014d321c7eb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014d32663d6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014d32081aa6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14d2f0664000 (LWP 84600) "cuda-EvtHandlr"):
#0  0x000014d32080e1c9 in poll () from /lib64/libc.so.6
#1  0x000014d321c83b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014d321d2ad7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014d321c7eb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014d32663d6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014d32081aa6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14d2f5bf9000 (LWP 84663) "moya_3d"):
#0  0x000014d3266459b5 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x000014d326d0811b in ofi_genlock_lock (lock=0x10b2cb0) at ./include/ofi_lock.h:359
#2  ofi_cq_readfrom (cq_fid=0x10b2c40, buf=<optimized out>, count=1, src_addr=0x0) at prov/util/src/util_cq.c:233
#3  0x000014d327823e46 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d327818f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d3273f676e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014d3272c04e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014d3272bfe98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014d32738a86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014d32663d6ea in start_thread () from /lib64/libpthread.so.0
#10 0x000014d32081aa6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14d2f0263000 (LWP 84667) "moya_3d"):
#0  0x000014d320813ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d327414875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d3272bffaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d32738a86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d32663d6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014d32081aa6f in clone () from /lib64/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
Thread 6 (Thread 0x14d2fe6d7000 (LWP 84673) "moya_3d"):
#0  0x000014d320813ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d327414875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d32738233c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d327381e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d32738c73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d320764600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14d2fe6bc000 (LWP 84677) "moya_3d"):
#0  0x000014d320813ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d327414875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d32738233c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d327381e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d32738c73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d320764600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14c0d8c7c000 (LWP 84679) "moya_3d"):
#0  0x000014d320813ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d327414875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d32738233c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x000014d327381e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d327386857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d32738a86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014d32663d6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014d32081aa6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14d2f0150000 (LWP 84685) "moya_3d"):
#0  0x000014d320813ea9 in syscall () from /lib64/libc.so.6
#1  0x000014d327414875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014d327415986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014d3273997d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014d32738a86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014d32663d6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014d32081aa6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T22:02:53Z

backtrace 7

(gdb) thread apply * bt

Thread 1 (Thread 0x1493d65cb000 (LWP 84539) "moya_3d"):
#0  0x00001493e143270c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001493e20bb3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e20b292a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e20b271c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e33de2c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x00001493e47fc590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffe0dd527f8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x1493cf188000 (LWP 84557) "cuda-EvtHandlr"):
#0  0x00001493db5fc1c9 in poll () from /lib64/libc.so.6
#1  0x00001493dca71b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001493dcb18d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001493dca6cb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001493e142b6ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001493db608a6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x1493ab171000 (LWP 84602) "cuda-EvtHandlr"):
#0  0x00001493db5fc1c9 in poll () from /lib64/libc.so.6
#1  0x00001493dca71b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001493dcb18d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001493dca6cb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001493e142b6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00001493db608a6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x1493c11f9000 (LWP 84666) "moya_3d"):
#0  0x00001493e2613233 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x00001493e2606f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e21e476e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e20ae4e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e20ade98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001493e217886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001493e142b6ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001493db608a6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x1493c01f0000 (LWP 84668) "moya_3d"):
#0  0x00001493db601ea9 in syscall () from /lib64/libc.so.6
#1  0x00001493e2202875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e20adfaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e217886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e142b6ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001493db608a6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x1493cc0ea000 (LWP 84672) "moya_3d"):
#0  0x00001493db601ea9 in syscall () from /lib64/libc.so.6
#1  0x00001493e2202875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems--Type <RET> for more, q to quit, c to continue without paging--
/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e217033c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e216fe2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e217a73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001493db552600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x1493cc0cf000 (LWP 84675) "moya_3d"):
#0  0x00001493db601ea9 in syscall () from /lib64/libc.so.6
#1  0x00001493e2202875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e217033c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e216fe2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e217a73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001493db552600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x1493a9d6e000 (LWP 84676) "moya_3d"):
#0  0x00001493db601ea9 in syscall () from /lib64/libc.so.6
#1  0x00001493e2202875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e217033c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e216fe2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e2174857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTas--Type <RET> for more, q to quit, c to continue without paging--
kScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001493e217886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001493e142b6ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001493db608a6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x1493a955a000 (LWP 84684) "moya_3d"):
#0  0x00001493db601ea9 in syscall () from /lib64/libc.so.6
#1  0x00001493e2202875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001493e2203986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001493e21877d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001493e217886e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001493e142b6ea in start_thread () from /lib64/libpthread.so.0
#6  0x00001493db608a6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T22:09:02Z

backtrace 8

(gdb) thread apply * bt

Thread 1 (Thread 0x145b3ea5c000 (LWP 41073) "moya_3d"):
#0  0x0000145b498c970c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000145b4a5523e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a54992a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b4a54971c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000145b4b8752c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x0000145b4cc8d590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7fff02df0a38) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x145b37619000 (LWP 41122) "cuda-EvtHandlr"):
#0  0x0000145b43a931c9 in poll () from /lib64/libc.so.6
#1  0x0000145b44f08b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000145b44fafd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000145b44f03b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000145b498c26ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000145b43a9fa6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x145b13770000 (LWP 41133) "cuda-EvtHandlr"):
#0  0x0000145b43a931c9 in poll () from /lib64/libc.so.6
#1  0x0000145b44f08b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000145b44fafd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000145b44f03b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x0000145b498c26ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000145b43a9fa6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x145b1d1f9000 (LWP 41221) "moya_3d"):
#0  0x0000145b4a545518 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x0000145b4a544e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a60f86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b498c26ea in start_thread () from /lib64/libpthread.so.0
#4  0x0000145b43a9fa6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x145b1c1f0000 (LWP 41225) "moya_3d"):
#0  0x0000145b43a98ea9 in syscall () from /lib64/libc.so.6
#1  0x0000145b4a699875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a544faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b4a60f86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000145b498c26ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000145b43a9fa6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x145b31c56000 (LWP 41233) "moya_3d"):
#0  0x0000145b43a98ea9 in syscall () from /lib64/libc.so.6
#1  0x0000145b4a699875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a60733c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b4a606e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000145b4a61173d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/s--Type <RET> for more, q to quit, c to continue without paging--
ystems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000145b439e9600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x145b31c3b000 (LWP 41236) "moya_3d"):
#0  0x0000145b43a98ea9 in syscall () from /lib64/libc.so.6
#1  0x0000145b4a699875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a60733c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b4a606e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000145b4a61173d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000145b439e9600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x145b1236d000 (LWP 41237) "moya_3d"):
#0  0x0000145b43a98ea9 in syscall () from /lib64/libc.so.6
#1  0x0000145b4a699875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a60733c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b4a606e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000145b4a60b857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000145b4a60f86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x0000145b498c26ea in start_thread () from /lib64/libpthread.so.0
#7  0x0000145b43a9fa6f in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 9 (Thread 0x145b11b59000 (LWP 41241) "moya_3d"):
#0  0x0000145b43a98ea9 in syscall () from /lib64/libc.so.6
#1  0x0000145b4a699875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x0000145b4a69a986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x0000145b4a61e7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000145b4a60f86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000145b498c26ea in start_thread () from /lib64/libpthread.so.0
#6  0x0000145b43a9fa6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T22:13:58Z

backtrace 9

(gdb) thread apply * bt

Thread 1 (Thread 0x14e5efbcf000 (LWP 41074) "moya_3d"):
#0  0x000014e5faa3670c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014e5fb6bf3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014e5fb6b692a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014e5fb6b671c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014e5fc9e22c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014e5fde00590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffeae73bb08) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14e5e878c000 (LWP 41119) "cuda-EvtHandlr"):
#0  0x000014e5f4c001c9 in poll () from /lib64/libc.so.6
#1  0x000014e5f6075b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014e5f611cd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014e5f6070b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014e5faa2f6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014e5f4c0ca6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14e5c4664000 (LWP 41134) "cuda-EvtHandlr"):
#0  0x000014e5f4c001c9 in poll () from /lib64/libc.so.6
#1  0x000014e5f6075b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014e5f611cd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014e5f6070b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014e5faa2f6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014e5f4c0ca6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14e5ce7fb000 (LWP 41222) "moya_3d"):
#0  0x000014e5fb13eb12 in cxi_eq_peek_event (eq=0x10bb888) at /usr/include/cxi_prov_hw.h:1540
#1  cxip_ep_ctrl_eq_progress (ep_obj=0x109d2e0, ctrl_evtq=0x10bb888, tx_evtq=false, ep_obj_locked=false) at prov/cxi/src/cxip_ctrl.c:326
#2  0x000014e5fb13f1a4 in cxip_ep_ctrl_progress (ep_obj=0x109d2e0) at prov/cxi/src/cxip_ctrl.c:372
#3  0x000014e5fb0fa111 in ofi_cq_readfrom (cq_fid=0x10b7300, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:232
#4  0x000014e5fbc15b36 in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014e5fbc15d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014e5fbc0af9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014e5fb7e876e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014e5fb6b24e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014e5fb6b1e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x000014e5fb77c86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#11 0x000014e5faa2f6ea in start_thread () from /lib64/libpthread.so.0
#12 0x000014e5f4c0ca6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14e5ca3f9000 (LWP 41226) "moya_3d"):
#0  0x000014e5f4c05ea9 in syscall () from /lib64/libc.so.6
#1  0x000014e5fb806875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014e5fb6b1faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x000014e5fb77c86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014e5faa2f6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014e5f4c0ca6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14e5e80f4000 (LWP 41229) "moya_3d"):
#0  0x000014e5f4c05ea9 in syscall () from /lib64/libc.so.6
#1  0x000014e5fb806875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014e5fb77433c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014e5fb773e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014e5fb77e73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014e5f4b56600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14e5e80d9000 (LWP 41230) "moya_3d"):
#0  0x000014e5f4c05ea9 in syscall () from /lib64/libc.so.6
#1  0x000014e5fb806875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014e5fb77433c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014e5fb773e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014e5fb77e73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014e5f4b56600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14e5c4263000 (LWP 41231) "moya_3d"):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x000014e5f4c05ea9 in syscall () from /lib64/libc.so.6
#1  0x000014e5fb806875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014e5fb77433c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014e5fb773e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014e5fb778857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014e5fb77c86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014e5faa2f6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014e5f4c0ca6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14d3ac67b000 (LWP 41243) "moya_3d"):
#0  0x000014e5f4c05ea9 in syscall () from /lib64/libc.so.6
#1  0x000014e5fb806875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014e5fb807986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014e5fb78b7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014e5fb77c86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014e5faa2f6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014e5f4c0ca6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T22:17:13Z

backtrace 10

(gdb) thread apply * bt

Thread 1 (Thread 0x151264181000 (LWP 41075) "moya_3d"):
#0  0x000015126efe870c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000015126fc713e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fc6892a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fc6871c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x0000151270f942c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x00001512723b2590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffc78a0a9f8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x15125cd3e000 (LWP 41116) "cuda-EvtHandlr"):
#0  0x00001512691b21c9 in poll () from /lib64/libc.so.6
#1  0x000015126a627b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000015126a6ced7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000015126a622b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000015126efe16ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001512691bea6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x151238e73000 (LWP 41132) "cuda-EvtHandlr"):
#0  0x00001512691b21c9 in poll () from /lib64/libc.so.6
#1  0x000015126a627b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000015126a6ced7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000015126a622b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000015126efe16ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00001512691bea6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x15123e7fb000 (LWP 41223) "moya_3d"):
#0  0x00001512691b7ea9 in syscall () from /lib64/libc.so.6
#1  0x000015126fdb8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fc63faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fd2e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015126efe16ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001512691bea6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x151238a72000 (LWP 41227) "moya_3d"):
#0  0x00001512701c9233 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x00001512701bcf9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fd9a76e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fc644e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015126fc63e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015126fd2e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000015126efe16ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001512691bea6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x15125c0a5000 (LWP 41238) "moya_3d"):
#0  0x00001512691b7ea9 in syscall () from /lib64/libc.so.6
#1  0x000015126fdb8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems--Type <RET> for more, q to quit, c to continue without paging--
/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fd2633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fd25e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015126fd3073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000151269108600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x15125c08a000 (LWP 41239) "moya_3d"):
#0  0x00001512691b7ea9 in syscall () from /lib64/libc.so.6
#1  0x000015126fdb8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fd2633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fd25e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015126fd3073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000151269108600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x15123895f000 (LWP 41240) "moya_3d"):
#0  0x00001512691b7ea9 in syscall () from /lib64/libc.so.6
#1  0x000015126fdb8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fd2633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fd25e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015126fd2a857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTas--Type <RET> for more, q to quit, c to continue without paging--
kScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015126fd2e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000015126efe16ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001512691bea6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x15123814b000 (LWP 41242) "moya_3d"):
#0  0x00001512691b7ea9 in syscall () from /lib64/libc.so.6
#1  0x000015126fdb8875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015126fdb9986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015126fd3d7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015126fd2e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015126efe16ea in start_thread () from /lib64/libpthread.so.0
#6  0x00001512691bea6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T22:20:07Z

backtrace 11

(gdb) thread apply * bt

Thread 1 (Thread 0x1485b557a000 (LWP 41076) "moya_3d"):
#0  0x00001485c03e770c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001485c10703e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001485c106792a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001485c106771c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c23932c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x00001485c37ab590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffe9e5714b8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x1485ae137000 (LWP 41113) "cuda-EvtHandlr"):
#0  0x00001485ba5b11c9 in poll () from /lib64/libc.so.6
#1  0x00001485bba26b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001485bbacdd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001485bba21b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001485c03e06ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001485ba5bda6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14858a464000 (LWP 41135) "cuda-EvtHandlr"):
#0  0x00001485ba5b11c9 in poll () from /lib64/libc.so.6
#1  0x00001485bba26b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001485bbacdd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001485bba21b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001485c03e06ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00001485ba5bda6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14858a063000 (LWP 41224) "moya_3d"):
#0  0x00001485ba5b6ea9 in syscall () from /lib64/libc.so.6
#1  0x00001485c11b7875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001485c1062faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001485c112d86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c03e06ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001485ba5bda6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x148589f50000 (LWP 41228) "moya_3d"):
#0  cxip_cq_progress (cq=0x10b9530) at prov/cxi/src/cxip_cq.c:544
#1  0x00001485c0acfc79 in cxip_util_cq_progress (util_cq=0x10b9530) at prov/cxi/src/cxip_cq.c:563
#2  0x00001485c0aab111 in ofi_cq_readfrom (cq_fid=0x10b9530, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:232
#3  0x00001485c15c6c16 in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c15c6d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001485c15bbf9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001485c119976e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x00001485c10634e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x00001485c1062e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x00001485c112d86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#10 0x00001485c03e06ea in start_thread () from /lib64/libpthread.so.0
#11 0x00001485ba5bda6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x1485ac09e000 (LWP 41232) "moya_3d"):
#0  0x00001485ba5b6ea9 in syscall () from /lib64/libc.so.6
#1  0x00001485c11b7875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001485c112533c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001485c1124e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c112f73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001485ba507600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x1485ac083000 (LWP 41234) "moya_3d"):
#0  0x00001485ba5b6ea9 in syscall () from /lib64/libc.so.6
#1  0x00001485c11b7875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001485c112533c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001485c1124e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c112f73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001485ba507600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x148588e3b000 (LWP 41235) "moya_3d"):
#0  0x00001485ba5b6ea9 in syscall () from /lib64/libc.so.6
#1  0x00001485c11b7875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems--Type <RET> for more, q to quit, c to continue without paging--
/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001485c112533c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001485c1124e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c1129857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001485c112d86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001485c03e06ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001485ba5bda6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x148588627000 (LWP 41244) "moya_3d"):
#0  0x00001485ba5b6ea9 in syscall () from /lib64/libc.so.6
#1  0x00001485c11b7875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001485c11b8986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001485c113c7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001485c112d86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001485c03e06ea in start_thread () from /lib64/libpthread.so.0
#6  0x00001485ba5bda6f in clone () from /lib64/libc.so.6
(gdb) q
A debugging session is active.

	Inferior 1 [process 41076] will be detached.

Quit anyway? (y or n)

jpietarilagraham · 2024-08-08T22:26:06Z

backtrace 12

(gdb) thread apply * bt

Thread 1 (Thread 0x14cd766e1000 (LWP 118044) "moya_3d"):
#0  0x000014cd8154870c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014cd821d13e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014cd821c892a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014cd821c871c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014cd834f42c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014cd84912590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffc161e8f98) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14cd6f29e000 (LWP 118085) "cuda-EvtHandlr"):
#0  0x000014cd7b7121c9 in poll () from /lib64/libc.so.6
#1  0x000014cd7cb87b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014cd7cc2ed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014cd7cb82b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014cd815416ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014cd7b71ea6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14cd4b474000 (LWP 118104) "cuda-EvtHandlr"):
#0  0x000014cd7b7121c9 in poll () from /lib64/libc.so.6
#1  0x000014cd7cb87b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014cd7cc2ed7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014cd7cb82b98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014cd815416ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014cd7b71ea6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14cd611f9000 (LWP 118185) "moya_3d"):
#0  0x000014cd81c30539 in cxip_cq_eq_progress (eq=0x10b91a0, cq=0x10b9080) at prov/cxi/src/cxip_cq.c:508
#1  cxip_cq_progress (cq=0x10b9080) at prov/cxi/src/cxip_cq.c:550
#2  0x000014cd81c30c79 in cxip_util_cq_progress (util_cq=0x10b9080) at prov/cxi/src/cxip_cq.c:563
#3  0x000014cd81c0c111 in ofi_cq_readfrom (cq_fid=0x10b9080, buf=<optimized out>, count=1, src_addr=0x0) at prov/util/src/util_cq.c:232
#4  0x000014cd82727e46 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014cd8271cf9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014cd822fa76e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014cd821c44e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014cd821c3e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014cd8228e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x000014cd815416ea in start_thread () from /lib64/libpthread.so.0
#11 0x000014cd7b71ea6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14cd4b073000 (LWP 118189) "moya_3d"):
#0  0x000014cd7b717ea9 in syscall () from /lib64/libc.so.6
#1  0x000014cd82318875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014cd821c3faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014cd8228e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014cd815416ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014cd7b71ea6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14cd654cf000 (LWP 118191) "moya_3d"):
#0  0x000014cd7b717ea9 in syscall () from /lib64/libc.so.6
#1  0x000014cd82318875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014cd8228633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014cd82285e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014cd8229073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014cd7b668600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14cd654b4000 (LWP 118193) "moya_3d"):
#0  0x000014cd7b717ea9 in syscall () from /lib64/libc.so.6
#1  0x000014cd82318875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014cd8228633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014cd82285e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014cd8229073d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014cd7b668600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14cd49f5e000 (LWP 118194) "moya_3d"):
#0  0x000014cd7b717ea9 in syscall () from /lib64/libc.so.6
#1  0x000014cd82318875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x000014cd8228633c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014cd82285e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014cd8228a857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014cd8228e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014cd815416ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014cd7b71ea6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14cd4974a000 (LWP 118203) "moya_3d"):
#0  0x000014cd7b717ea9 in syscall () from /lib64/libc.so.6
#1  0x000014cd82318875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014cd82319986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014cd8229d7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014cd8228e86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014cd815416ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014cd7b71ea6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-08T22:35:24Z

backtrace 13

(gdb) thread apply * bt

Thread 1 (Thread 0x1520bc41e000 (LWP 118045) "moya_3d"):
#0  0x00001520c728570c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001520c7f0e3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c7f0592a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c7f0571c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c92312c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x00001520ca64f590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffd4e6ad7f8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x1520b4fdb000 (LWP 118094) "cuda-EvtHandlr"):
#0  0x00001520c144f1c9 in poll () from /lib64/libc.so.6
#1  0x00001520c28c4b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001520c296bd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001520c28bfb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001520c727e6ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001520c145ba6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x152091074000 (LWP 118103) "cuda-EvtHandlr"):
#0  0x00001520c144f1c9 in poll () from /lib64/libc.so.6
#1  0x00001520c28c4b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001520c296bd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001520c28bfb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001520c727e6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00001520c145ba6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x15209c1ed000 (LWP 118183) "moya_3d"):
#0  0x00001520c79492e2 in ofi_cq_read (cq_fid=0x10b2f00, buf=0x15209c1dbcf0, count=1) at prov/util/src/util_cq.c:286
#1  0x00001520c8464e46 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c8459f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c803776e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c7f014e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001520c7f00e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001520c7fcb86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x00001520c727e6ea in start_thread () from /lib64/libpthread.so.0
#8  0x00001520c145ba6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x1520967fb000 (LWP 118186) "moya_3d"):
#0  0x00001520c1454ea9 in syscall () from /lib64/libc.so.6
#1  0x00001520c8055875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c7f00faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c7fcb86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c727e6ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001520c145ba6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x1520b0031000 (LWP 118197) "moya_3d"):
--Type <RET> for more, q to quit, c to continue without paging--
#0  0x00001520c1454ea9 in syscall () from /lib64/libc.so.6
#1  0x00001520c8055875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c7fc333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c7fc2e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c7fcd73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001520c13a5600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x15209f2e1000 (LWP 118199) "moya_3d"):
#0  0x00001520c1454ea9 in syscall () from /lib64/libc.so.6
#1  0x00001520c8055875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c7fc333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c7fc2e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c7fcd73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001520c13a5600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x152090c73000 (LWP 118201) "moya_3d"):
#0  0x00001520c1454ea9 in syscall () from /lib64/libc.so.6
#1  0x00001520c8055875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c7fc333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c7fc2e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack---Type <RET> for more, q to quit, c to continue without paging--
2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c7fc7857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001520c7fcb86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001520c727e6ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001520c145ba6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x15209045f000 (LWP 118204) "moya_3d"):
#0  0x00001520c1454ea9 in syscall () from /lib64/libc.so.6
#1  0x00001520c8055875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001520c8056986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001520c7fda7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001520c7fcb86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001520c727e6ea in start_thread () from /lib64/libpthread.so.0
#6  0x00001520c145ba6f in clone () from /lib64/libc.so.6
(gdb) 
(gdb)

jpietarilagraham · 2024-08-08T22:39:41Z

backtrace 14

(gdb) thread apply * bt

Thread 1 (Thread 0x150700a7c000 (LWP 118046) "moya_3d"):
#0  0x000015070b8e370c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000015070c56c3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c56392a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c56371c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070d88f2c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000015070ecad590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffe9ba941e8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x1506f9639000 (LWP 118091) "cuda-EvtHandlr"):
#0  0x0000150705aad1c9 in poll () from /lib64/libc.so.6
#1  0x0000150706f22b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000150706fc9d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000150706f1db98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000150705ab9a6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x1506d5762000 (LWP 118106) "cuda-EvtHandlr"):
#0  0x0000150705aad1c9 in poll () from /lib64/libc.so.6
#1  0x0000150706f22b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x0000150706fc9d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x0000150706f1db98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000150705ab9a6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x1506db5f9000 (LWP 118184) "moya_3d"):
#0  0x0000150705ab2ea9 in syscall () from /lib64/libc.so.6
#1  0x000015070c6b3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c55efaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c62986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
#5  0x0000150705ab9a6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x1506d5361000 (LWP 118187) "moya_3d"):
#0  0x000015070bfa70c0 in ofi_cq_readfrom (cq_fid=0x10b75f0, buf=0x1506d534fa70, count=64, src_addr=0x0) at prov/util/src/util_cq.c:221
#1  0x000015070cac2b36 in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070cac2d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070cab7f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070c69576e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015070c55f4e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000015070c55ee98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000015070c62986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
#9  0x0000150705ab9a6f in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 6 (Thread 0x1506f80a1000 (LWP 118192) "moya_3d"):
#0  0x0000150705ab2ea9 in syscall () from /lib64/libc.so.6
#1  0x000015070c6b3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c62133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c620e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070c62b73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150705a03600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x1506f8086000 (LWP 118195) "moya_3d"):
#0  0x0000150705ab2ea9 in syscall () from /lib64/libc.so.6
#1  0x000015070c6b3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c62133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c620e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070c62b73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x0000150705a03600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x1506d424c000 (LWP 118196) "moya_3d"):
#0  0x0000150705ab2ea9 in syscall () from /lib64/libc.so.6
#1  0x000015070c6b3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c62133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/r--Type <RET> for more, q to quit, c to continue without paging--
istra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c620e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070c625857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015070c62986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
#7  0x0000150705ab9a6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14bfe167d000 (LWP 118206) "moya_3d"):
#0  0x0000150705ab2ea9 in syscall () from /lib64/libc.so.6
#1  0x000015070c6b3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c6b4986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c6387d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000015070c62986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
#6  0x0000150705ab9a6f in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x14bfe156a000 (LWP 118260) "moya_3d"):
#0  0x0000150705ab2ea9 in syscall () from /lib64/libc.so.6
#1  0x000015070c6b3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000015070c6b4986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000015070c6387d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x000015070c62986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000015070b8dc6ea in start_thread () from /lib64/libpthread.so.0
#6  0x0000150705ab9a6f in clone () from /lib64/libc.so.6
(gdb) 
(gdb)

jpietarilagraham · 2024-08-08T22:40:16Z

backtrace 15

(gdb) thread apply * bt

Thread 1 (Thread 0x14fed60a6000 (LWP 118047) "moya_3d"):
#0  0x000014fee0f1370c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014fee1b9c3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014fee1b9392a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014fee1b9371c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014fee2ebf2c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014fee42d7590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7fff9aa3f128) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14fecec63000 (LWP 118088) "cuda-EvtHandlr"):
#0  0x000014fedb0dd1c9 in poll () from /lib64/libc.so.6
#1  0x000014fedc552b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014fedc5f9d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014fedc54db98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014fee0f0c6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014fedb0e9a6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14feaae64000 (LWP 118105) "cuda-EvtHandlr"):
#0  0x000014fedb0dd1c9 in poll () from /lib64/libc.so.6
#1  0x000014fedc552b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014fedc5f9d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014fedc54db98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014fee0f0c6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014fedb0e9a6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14fec0ff9000 (LWP 118188) "moya_3d"):
#0  0x000014fee161bb12 in cxi_eq_peek_event (eq=0x10fed38) at /usr/include/cxi_prov_hw.h:1540
#1  cxip_ep_ctrl_eq_progress (ep_obj=0x10a71b0, ctrl_evtq=0x10fed38, tx_evtq=false, ep_obj_locked=false) at prov/cxi/src/cxip_ctrl.c:326
#2  0x000014fee161c1a4 in cxip_ep_ctrl_progress (ep_obj=0x10a71b0) at prov/cxi/src/cxip_ctrl.c:372
#3  0x000014fee15d7111 in ofi_cq_readfrom (cq_fid=0x10b52c0, buf=<optimized out>, count=1, src_addr=0x0) at prov/util/src/util_cq.c:232
#4  0x000014fee20f4278 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014fee20e7f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014fee1cc56b4 in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014fee1b8f4e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014fee1b8ee98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014fee1c5986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x000014fee0f0c6ea in start_thread () from /lib64/libpthread.so.0
#11 0x000014fedb0e9a6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14feaaa63000 (LWP 118190) "moya_3d"):
#0  0x000014fedb0e2ea9 in syscall () from /lib64/libc.so.6
#1  0x000014fee1ce3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014fee1b8efaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014fee1c5986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#4  0x000014fee0f0c6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014fedb0e9a6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14fec529a000 (LWP 118198) "moya_3d"):
#0  0x000014fedb0e2ea9 in syscall () from /lib64/libc.so.6
#1  0x000014fee1ce3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014fee1c5133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014fee1c50e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014fee1c5b73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014fedb033600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14fec527f000 (LWP 118200) "moya_3d"):
#0  0x000014fedb0e2ea9 in syscall () from /lib64/libc.so.6
#1  0x000014fee1ce3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014fee1c5133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014fee1c50e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014fee1c5b73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014fedb033600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14fea994e000 (LWP 118202) "moya_3d"):
#0  0x000014fedb0e2ea9 in syscall () from /lib64/libc.so.6
#1  0x000014fee1ce3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems--Type <RET> for more, q to quit, c to continue without paging--
/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014fee1c5133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014fee1c50e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014fee1c55857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014fee1c5986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014fee0f0c6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014fedb0e9a6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14fea913a000 (LWP 118205) "moya_3d"):
#0  0x000014fedb0e2ea9 in syscall () from /lib64/libc.so.6
#1  0x000014fee1ce3875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014fee1ce4986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014fee1c687d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014fee1c5986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014fee0f0c6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014fedb0e9a6f in clone () from /lib64/libc.so.6
(gdb) 
(gdb) 
(gdb)

elliottslaughter · 2024-08-08T23:33:45Z

Just FYI, when you have a bunch of files like this you can just attach them directly by clicking and dragging into the comment text box. (Technically the file extension needs to be .txt or .log or something else that GitHub recognizes, but that's usually easy.)

jpietarilagraham · 2024-08-09T00:06:36Z

thread 16

(gdb) thread apply * bt

Thread 1 (Thread 0x14f4eca2a000 (LWP 55102) "moya_3d"):
#0  0x000014f4f789170c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014f4f851a3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f851192a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014f4f851171c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f983d2c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014f4fac5b590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7fffcb4d16a8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14f4e55e7000 (LWP 55155) "cuda-EvtHandlr"):
#0  0x000014f4f1a5b1c9 in poll () from /lib64/libc.so.6
#1  0x000014f4f2ed0b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014f4f2f77d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014f4f2ecbb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014f4f1a67a6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14f4c1664000 (LWP 55167) "cuda-EvtHandlr"):
#0  0x000014f4f1a5b1c9 in poll () from /lib64/libc.so.6
#1  0x000014f4f2ed0b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014f4f2f77d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014f4f2ecbb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014f4f1a67a6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14f4cc1fb000 (LWP 55249) "moya_3d"):
#0  ofi_spin_lock_op (lock=0x10bb168) at ./include/ofi_lock.h:182
#1  0x000014f4f7f550df in ofi_genlock_lock (lock=0x10bb160) at ./include/ofi_lock.h:359
#2  ofi_cq_readfrom (cq_fid=0x10bb0f0, buf=<optimized out>, count=1, src_addr=0x0) at prov/util/src/util_cq.c:229
#3  0x000014f4f8a72278 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f8a65f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014f4f864376e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014f4f850d4e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x000014f4f850ce98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x000014f4f85d786e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
#10 0x000014f4f1a67a6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14f4c73fd000 (LWP 55255) "moya_3d"):
#0  0x000014f4f1a60ea9 in syscall () from /lib64/libc.so.6
#1  0x000014f4f8661875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f850cfaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014f4f85d786e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014f4f1a67a6f in clone () from /lib64/libc.so.6

--Type <RET> for more, q to quit, c to continue without paging--
Thread 6 (Thread 0x14f4e0041000 (LWP 55267) "moya_3d"):
#0  0x000014f4f1a60ea9 in syscall () from /lib64/libc.so.6
#1  0x000014f4f8661875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f85cf33c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014f4f85cee2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f85d973d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014f4f19b1600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14f4e0026000 (LWP 55268) "moya_3d"):
#0  0x000014f4f1a60ea9 in syscall () from /lib64/libc.so.6
#1  0x000014f4f8661875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f85cf33c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014f4f85cee2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f85d973d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014f4f19b1600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14f4c0261000 (LWP 55269) "moya_3d"):
#0  0x000014f4f1a60ea9 in syscall () from /lib64/libc.so.6
#1  0x000014f4f8661875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f85cf33c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x000014f4f85cee2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f85d3857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014f4f85d786e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014f4f1a67a6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14adcd67d000 (LWP 55273) "moya_3d"):
#0  0x000014f4f1a60ea9 in syscall () from /lib64/libc.so.6
#1  0x000014f4f8661875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f8662986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014f4f85e67d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f85d786e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014f4f1a67a6f in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x14adcd56a000 (LWP 55346) "moya_3d"):
#0  0x000014f4f1a60ea9 in syscall () from /lib64/libc.so.6
#1  0x000014f4f8661875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014f4f8662986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014f4f85e67d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014f4f85d786e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024---Type <RET> for more, q to quit, c to continue without paging--
04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014f4f788a6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014f4f1a67a6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-09T00:07:21Z

Just FYI, when you have a bunch of files like this you can just attach them directly by clicking and dragging into the comment text box. (Technically the file extension needs to be .txt or .log or something else that GitHub recognizes, but that's usually easy.)

I'm just cutting and pasting from my terminal.

jpietarilagraham · 2024-08-09T00:13:03Z

17


(gdb) thread apply * bt

Thread 1 (Thread 0x1479c246e000 (LWP 55103) "moya_3d"):
#0  0x00001479cd2d570c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00001479cdf5e3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001479cdf5592a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001479cdf5571c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479cf2812c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x00001479d069f590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffe9b07caf8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x1479bb02b000 (LWP 55146) "cuda-EvtHandlr"):
#0  0x00001479c749f1c9 in poll () from /lib64/libc.so.6
#1  0x00001479c8914b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001479c89bbd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001479c890fb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001479cd2ce6ea in start_thread () from /lib64/libpthread.so.0
#5  0x00001479c74aba6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x147996f61000 (LWP 55165) "cuda-EvtHandlr"):
#0  0x00001479c749f1c9 in poll () from /lib64/libc.so.6
#1  0x00001479c8914b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00001479c89bbd7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x00001479c890fb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00001479cd2ce6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00001479c74aba6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x1479b15f8000 (LWP 55248) "moya_3d"):
#0  0x00001479cd2d69b3 in pthread_spin_lock () from /lib64/libpthread.so.0
#1  0x00001479cd99911b in ofi_genlock_lock (lock=0x10b97c0) at ./include/ofi_lock.h:359
#2  ofi_cq_readfrom (cq_fid=0x10b9750, buf=<optimized out>, count=64, src_addr=0x0) at prov/util/src/util_cq.c:233
#3  0x00001479ce4b4c16 in gasnetc_ofi_tx_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479ce4b4d96 in gasnetc_ofi_poll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001479ce4a9f9d in gasnetc_AMPoll () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001479ce08776e in Realm::GASNetEXPoller::do_work(Realm::TimeLimit) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#7  0x00001479cdf514e1 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#8  0x00001479cdf50e98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#9  0x00001479ce01b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#10 0x00001479cd2ce6ea in start_thread () from /lib64/libpthread.so.0
#11 0x00001479c74aba6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14799c7f9000 (LWP 55252) "moya_3d"):
#0  0x00001479c74a4ea9 in syscall () from /lib64/libc.so.6
#1  0x00001479ce0a5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001479cdf50faa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001479ce01b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479cd2ce6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x00001479c74aba6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x1479b805d000 (LWP 55262) "moya_3d"):
#0  0x00001479c74a4ea9 in syscall () from /lib64/libc.so.6
#1  0x00001479ce0a5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001479ce01333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001479ce012e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479ce01d73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001479c73f5600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x1479b8042000 (LWP 55265) "moya_3d"):
#0  0x00001479c74a4ea9 in syscall () from /lib64/libc.so.6
#1  0x00001479ce0a5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001479ce01333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001479ce012e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479ce01d73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001479c73f5600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x147995d5e000 (LWP 55266) "moya_3d"):
#0  0x00001479c74a4ea9 in syscall () from /lib64/libc.so.6
#1  0x00001479ce0a5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
--Type <RET> for more, q to quit, c to continue without paging--
#2  0x00001479ce01333c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001479ce012e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479ce017857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001479ce01b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x00001479cd2ce6ea in start_thread () from /lib64/libpthread.so.0
#7  0x00001479c74aba6f in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x14799554a000 (LWP 55272) "moya_3d"):
#0  0x00001479c74a4ea9 in syscall () from /lib64/libc.so.6
#1  0x00001479ce0a5875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x00001479ce0a6986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x00001479ce02a7d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x00001479ce01b86e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x00001479cd2ce6ea in start_thread () from /lib64/libpthread.so.0
#6  0x00001479c74aba6f in clone () from /lib64/libc.so.6
(gdb)

elliottslaughter · 2024-08-09T00:13:47Z

Not that it matters to me, since I'm not the one debugging it, but if I were to do this, I'd open a text editor like Emacs or Vim and paste the backtraces into it, then save the files out as .txt. You can do <details> too, but this way whoever is going to debug them can download them easily into their own text editor.

jpietarilagraham · 2024-08-09T00:14:28Z

18

(gdb) thread apply * bt

Thread 1 (Thread 0x14db6b446000 (LWP 55104) "moya_3d"):
#0  0x000014db762b370c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000014db76f3c3e6 in Realm::GenEventImpl::external_wait(unsigned int, bool&) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db76f3392a in Realm::Event::external_wait_faultaware(bool&) const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db76f3371c in Realm::Event::external_wait() const () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014db7825f2c6 in Legion::Internal::LegionHandshakeImpl::ext_wait_on_legion() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/liblegion.so.1
#5  0x000014db79677590 in flecsi::run::context_t::start(std::function<int ()> const&, bool) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/libFleCSI.so.2.255.0
#6  0x0000000000598a5f in flecsi::runtime::control<flecsi::run::control<flecsi::run::call_policy>, int (&)()> (this=<optimized out>, aa=<optimized out>) at /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/include/flecsi/runtime.hh:48
#7  main (argc=<optimized out>, argv=0x7ffc5173c0a8) at /users/jgraham/RISTRA/re-git/moya/app/moya.cc:17

Thread 2 (Thread 0x14db64003000 (LWP 55149) "cuda-EvtHandlr"):
#0  0x000014db7047d1c9 in poll () from /lib64/libc.so.6
#1  0x000014db718f2b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014db71999d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014db718edb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014db762ac6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014db70489a6f in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x14db3ff70000 (LWP 55166) "cuda-EvtHandlr"):
#0  0x000014db7047d1c9 in poll () from /lib64/libc.so.6
#1  0x000014db718f2b89 in ?? () from /usr/lib64/libcuda.so.1
#2  0x000014db71999d7b in ?? () from /usr/lib64/libcuda.so.1
#3  0x000014db718edb98 in ?? () from /usr/lib64/libcuda.so.1
#4  0x000014db762ac6ea in start_thread () from /lib64/libpthread.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x000014db70489a6f in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x14db4e1fa000 (LWP 55250) "moya_3d"):
#0  0x000014db70482ea9 in syscall () from /lib64/libc.so.6
#1  0x000014db77083875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db76f2efaa in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db76ff986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014db762ac6ea in start_thread () from /lib64/libpthread.so.0
#5  0x000014db70489a6f in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x14db45bf9000 (LWP 55257) "moya_3d"):
#0  0x000014db76f2f522 in Realm::BackgroundWorkManager::Worker::do_work(long long, Realm::atomic<bool>*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#1  0x000014db76f2ee98 in Realm::BackgroundWorkThread::main_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db76ff986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db762ac6ea in start_thread () from /lib64/libpthread.so.0
#4  0x000014db70489a6f in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x14db60039000 (LWP 55258) "moya_3d"):
#0  0x000014db70482ea9 in syscall () from /lib64/libc.so.6
#1  0x000014db77083875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db76ff133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db76ff0e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014db76ffb73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/s--Type <RET> for more, q to quit, c to continue without paging--
ystems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014db703d3600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x14db6001e000 (LWP 55259) "moya_3d"):
#0  0x000014db70482ea9 in syscall () from /lib64/libc.so.6
#1  0x000014db77083875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db76ff133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db76ff0e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014db76ffb73d in Realm::UserThread::uthread_entry() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014db703d3600 in ?? () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x14db3c9ec000 (LWP 55260) "moya_3d"):
#0  0x000014db70482ea9 in syscall () from /lib64/libc.so.6
#1  0x000014db77083875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db76ff133c in Realm::ThreadedTaskScheduler::wait_for_work(unsigned long) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db76ff0e2b in Realm::ThreadedTaskScheduler::scheduler_loop() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014db76ff5857 in void Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock>(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014db76ff986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#6  0x000014db762ac6ea in start_thread () from /lib64/libpthread.so.0
#7  0x000014db70489a6f in clone () from /lib64/libc.so.6
--Type <RET> for more, q to quit, c to continue without paging--

Thread 9 (Thread 0x14db3c1d8000 (LWP 55271) "moya_3d"):
#0  0x000014db70482ea9 in syscall () from /lib64/libc.so.6
#1  0x000014db77083875 in Realm::Doorbell::wait_slow() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#2  0x000014db77084986 in Realm::UnfairCondVar::wait() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#3  0x000014db770087d5 in Realm::Cuda::ContextSynchronizer::thread_main() () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#4  0x000014db76ff986e in Realm::KernelThread::pthread_entry(void*) () from /users/jgraham/RISTRA/ristra-spack-2024-04-10/systems/chicoma/cuda_release/.spack-env/view/lib64/librealm.so.1
#5  0x000014db762ac6ea in start_thread () from /lib64/libpthread.so.0
#6  0x000014db70489a6f in clone () from /lib64/libc.so.6
(gdb)

jpietarilagraham · 2024-08-09T00:16:40Z

Well the job went down now. I hope there is enough info in the first 19 of 32 ranks.

lightsighter · 2024-08-09T08:49:17Z

Are these backtraces from a run that is hanging or running slowly?

apryakhin · 2024-08-09T22:26:02Z

I have an impression that this is coming from the "hang" but @jpietarilagraham would be best to confirm. I think what Elliott had suggested perhaps just making it a single out.txt file out of the backtraces would be good.

apryakhin changed the title ~~Optimize gather copies in Moya/FleCSI~~ Realm: Optimize gather copies in Moya/FleCSI Sep 17, 2024

apryakhin added this to the realm-24.11 milestone Sep 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Realm: Optimize gather copies in Moya/FleCSI #1733

Realm: Optimize gather copies in Moya/FleCSI #1733

apryakhin commented Jul 30, 2024 •

edited

Loading

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

elliottslaughter commented Aug 8, 2024

jpietarilagraham commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

elliottslaughter commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

lightsighter commented Aug 9, 2024

apryakhin commented Aug 9, 2024 •

edited

Loading

Realm: Optimize gather copies in Moya/FleCSI #1733

Realm: Optimize gather copies in Moya/FleCSI #1733

Comments

apryakhin commented Jul 30, 2024 • edited Loading

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

jpietarilagraham commented Aug 8, 2024

elliottslaughter commented Aug 8, 2024

jpietarilagraham commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

elliottslaughter commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

jpietarilagraham commented Aug 9, 2024

lightsighter commented Aug 9, 2024

apryakhin commented Aug 9, 2024 • edited Loading

apryakhin commented Jul 30, 2024 •

edited

Loading

apryakhin commented Aug 9, 2024 •

edited

Loading