[PPC] Segfault via cling::IncrementalExecutor::runStaticInitializersOnce

[aloeliger]

What

Taken from cms-sw#41658

Many workflows failed in CMSSW_13_2_X_2023-05-11-2300 el8_ppc64le_gcc11 with

Fri May 12 01:48:41 CEST 2023
Thread 5 (Thread 0x10086aa09210 (LWP 127950) "cmsRun"):
#2  0x0000100009cb05fc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002c8274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#5  0x000010000292a53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x0000100000f49198 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so
#9  0x0000100000f5484c in ROOT::TRWMutexImp<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so
#10 0x0000100001aec050 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#11 0x0000100001aeda94 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#12 0x0000100000a8cf28 in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so
#13 0x0000100835b7a21c in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#14 0x0000100835b37050 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#15 0x0000100835b38b78 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#16 0x0000100835b72360 in boost::spirit::classic::parser_result<boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >::type boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >::parse<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
<snip>
#37 0x0000100835b6a700 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#38 0x0000100835b61488 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#39 0x000010086ee52f40 in StringCutObjectSelector<trigger::TriggerObject, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#40 0x000010086ee4f078 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#41 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#42 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#43 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#44 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#45 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so
#46 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so

Thread 4 (Thread 0x100866609210 (LWP 127949) "cmsRun"):
#2  0x0000100009cb05fc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002c8274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#5  0x000010000292a53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x0000100000f46bb8 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::ReadLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so
#9  0x0000100000f547cc in ROOT::TRWMutexImp<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::ReadLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so
#10 0x0000100001afbc08 in TClass::GetClass(std::type_info const&, bool, bool, unsigned long, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#11 0x0000100000a89d90 in edm::TypeWithDict::TypeWithDict(std::type_info const&, long) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so
#12 0x0000100000a8bf7c in edm::TypeWithDict::TypeWithDict(std::type_info const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so
#13 0x000010086ee52ce4 in StringCutObjectSelector<reco::Muon, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#14 0x000010086ee4ee90 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#15 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#16 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#17 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#18 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#19 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so
#20 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so

Thread 3 (Thread 0x100865bf9210 (LWP 127948) "cmsRun"):
#3  0x0000100009cb5e9c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00001008c3642b40 in ?? ()
#6  0x00001008c3641bcc in ?? ()
#7  0x00001008c36434b0 in ?? ()
#8  0x00001008c36432fc in ?? ()
#9  0x00001008c363bfe8 in ?? ()
#10 0x00001008c363be90 in ?? ()
#11 0x00001008c363d7c4 in ?? ()
#12 0x00001008c363e8c0 in ?? ()
#13 0x00001008c363e820 in ?? ()
#14 0x00001008c3630188 in ?? ()
#15 0x0000100814722ac8 in cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#16 0x000010081468810c in cling::Interpreter::executeTransaction(cling::Transaction&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#17 0x000010081473a154 in cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#18 0x000010081473b930 in cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#19 0x00001008146837b0 in cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#20 0x000010081457937c in ExecAutoParse(char const*, bool, cling::Interpreter*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#21 0x0000100814589b98 in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#22 0x000010081458f934 in TCling::AutoParse(char const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCling.so
#23 0x0000100001aec1d0 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#24 0x0000100001aeda94 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#25 0x0000100000a8cf28 in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so
#26 0x0000100835b7a21c in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#27 0x0000100835b37050 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#28 0x0000100835b38b78 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#29 0x0000100835b72360 in boost::spirit::classic::parser_result<boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >::type boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >::parse<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
<snip>
#50 0x0000100835b6a700 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#51 0x0000100835b61488 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#52 0x000010086ee52f40 in StringCutObjectSelector<trigger::TriggerObject, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#53 0x000010086ee4f078 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#54 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#55 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#56 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#57 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#58 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so
#59 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so

Thread 1 (Thread 0x10000277d0b0 (LWP 127467) "cmsRun"):
#2  0x0000100009cb05fc in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002c8274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#5  0x000010000292a53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865
#6  std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/gcc-11.2.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#7  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#8  0x0000100000f49198 in ROOT::TReentrantRWLock<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so
#9  0x0000100000f5484c in ROOT::TRWMutexImp<std::mutex, ROOT::Internal::RecurseCountsTBBUnique>::WriteLock() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libThread.so
#10 0x0000100001aec050 in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#11 0x0000100001aeda94 in TClass::GetMethodWithPrototype(char const*, char const*, bool, ROOT::EFunctionMatchMode) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/external/el8_ppc64le_gcc11/lib/libCore.so
#12 0x0000100000a8cf28 in edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreReflection.so
#13 0x0000100835b7a21c in reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#14 0x0000100835b37050 in reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#15 0x0000100835b38b78 in reco::parser::MethodSetter::operator()(char const*, char const*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#16 0x0000100835b72360 in boost::spirit::classic::parser_result<boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >::type boost::spirit::classic::sequence<boost::spirit::classic::action<boost::spirit::classic::contiguous<boost::spirit::classic::sequence<boost::spirit::classic::alpha_parser, boost::spirit::classic::kleene_star<boost::spirit::classic::chset<char> > > >, reco::parser::MethodSetter>, boost::spirit::classic::optional<boost::spirit::classic::sequence<boost::spirit::classic::chlit<char>, boost::spirit::classic::chlit<char> > > >::parse<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > >(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
<snip>
#37 0x0000100835b6a700 in boost::spirit::classic::impl::concrete_parser<boost::spirit::classic::action<boost::spirit::classic::rule<boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t, boost::spirit::classic::nil_t>, reco::parser::CutSetter>, boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> >, boost::spirit::classic::nil_t>::do_parse_virtual(boost::spirit::classic::scanner<char const*, boost::spirit::classic::scanner_policies<boost::spirit::classic::skipper_iteration_policy<boost::spirit::classic::iteration_policy>, boost::spirit::classic::match_policy, boost::spirit::classic::action_policy> > const&) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#38 0x0000100835b61488 in reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libCommonToolsUtils.so
#39 0x000010086ee52f40 in StringCutObjectSelector<trigger::TriggerObject, false>::StringCutObjectSelector(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#40 0x000010086ee4f078 in HLTMuonMatchAndPlot::HLTMuonMatchAndPlot(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#41 0x000010086ee5eabc in void std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> >::_M_realloc_insert<edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool&>(__gnu_cxx::__normal_iterator<HLTMuonMatchAndPlot*, std::vector<HLTMuonMatchAndPlot, std::allocator<HLTMuonMatchAndPlot> > >, edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, bool&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#42 0x000010086ee5d1e8 in HLTMuonMatchAndPlotContainer::addPlotter(edm::ParameterSet const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/libDQMOfflineTrigger.so
#43 0x00001008732284f8 in HLTMuonOfflineAnalyzer::dqmBeginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#44 0x00001008731164b4 in DQMEDAnalyzer::beginRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw-patch/CMSSW_13_2_X_2023-05-11-2300/lib/el8_ppc64le_gcc11/pluginDQMOfflineTriggerPlugins.so
#45 0x0000100000440fa4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamBeginRun(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so
#46 0x000010000040223c in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamBegin(edm::StreamID, edm::RunTransitionInfo const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02784/el8_ppc64le_gcc11/cms/cmssw/CMSSW_13_2_X_2023-05-09-2300/lib/el8_ppc64le_gcc11/libFWCoreFramework.so

Current Modules:
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef (crashed)
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef
Module: HLTMuonOfflineAnalyzer:globalAnalyzerRef

...

To quote Andrea Valenzuela

I have been looking further into this issue and it seems it is caused by cms-sw#41590. If I use the latest IBs and revert the changes made in this PR, relval workflows run fine. I see @thesps is the original author of the PR. Could you please have a look at it?

In any case, I am not sure still why it is only failing on the ppc64le architecture.

Recipe to recreate

TBD

@thesps could I ask you to take a look at this? This issue seems to go away when your PR is reverted.