harmony crash vmState=0x0005ff0a #20561

pshipton · 2024-11-11T17:17:25Z

Internal build
[Linux S390 64bit] 80 Load_Level_2.harmony.5mins.Mode187
-Xaggressive -Xgcpolicy:gencon -Xjit -Xnocompressedrefs
rtv-rhel8s390x-svl-test-11xv9-1

vmState [0x5ff0a]: {J9VMSTATE_JIT} {EmitSnippets}

30x grinder passed

j> 07:51:56 20241111 07:51:56 Runtime State Reporter IMPORTANT: 7394 tests complete, 12 currently running
j> 07:51:57 #0: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x98d946) [0x3ff9988d946]
j> 07:51:57 #1: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x99b828) [0x3ff9989b828]
j> 07:51:57 #2: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x148bec) [0x3ff99048bec]
j> 07:51:57 #3: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9prt29.so(+0x2c450) [0x3ff9a72c450]
j> 07:51:57 #4: [0x3ff62de6bd8]
j> 07:51:57 #5: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x1adc96) [0x3ff990adc96]
j> 07:51:57 #6: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x33a334) [0x3ff9923a334]
j> 07:51:57 #7: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x9e6a26) [0x3ff998e6a26]
j> 07:51:57 #8: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x53a340) [0x3ff9943a340]
j> 07:51:57 #9: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x542cbc) [0x3ff99442cbc]
j> 07:51:57 #10: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x544e1e) [0x3ff99444e1e]
j> 07:51:57 #11: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x53fd94) [0x3ff9943fd94]
j> 07:51:57 #12: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x55f144) [0x3ff9945f144]
j> 07:51:57 #13: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x15aa06) [0x3ff9905aa06]
j> 07:51:57 #14: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x15b920) [0x3ff9905b920]
j> 07:51:57 #15: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9prt29.so(+0x2d7a0) [0x3ff9a72d7a0]
j> 07:51:57 #16: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x15954e) [0x3ff9905954e]
j> 07:51:57 #17: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x159b3a) [0x3ff99059b3a]
j> 07:51:57 #18: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x158730) [0x3ff99058730]
j> 07:51:57 #19: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x158a16) [0x3ff99058a16]
j> 07:51:57 #20: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x158ab4) [0x3ff99058ab4]
j> 07:51:57 #21: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9prt29.so(+0x2d7a0) [0x3ff9a72d7a0]
j> 07:51:57 #22: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so(+0x158f36) [0x3ff99058f36]
j> 07:51:57 #23: /j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9thr29.so(+0x923a) [0x3ff9a88923a]
j> 07:51:57 #24: /lib64/libpthread.so.0(+0x80de) [0x3ff9b8880de]
j> 07:51:57 #25: /lib64/libc.so.6(+0x28b42) [0x3ff9b5a8b42]
j> 07:51:57 #26: [(nil)]
j> 07:51:57 Unhandled exception
j> 07:51:57 Type=Segmentation error vmState=0x0005ff0a
j> 07:51:57 J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
j> 07:51:57 Handler1=000003FF9A9DCC20 Handler2=000003FF9A72C180 InaccessibleAddress=FFFFFFFFFFFFF000
j> 07:51:57 gpr0=0000000000000000 gpr1=000003FF943186F0 gpr2=FFFFFFFFFFFFFFFB gpr3=000003FF00000000
j> 07:51:57 gpr4=000003FF9923A2F8 gpr5=000003FF00000000 gpr6=000003FEE69C7440 gpr7=000003FF99097428
j> 07:51:57 gpr8=000003FF99C5DDB0 gpr9=000003FEE58E4000 gpr10=000003FEDF41F0B8 gpr11=000003FF64160090
j> 07:51:57 gpr12=000003FF9B9AAF70 gpr13=000003FF99C5DDB0 gpr14=000003FF9923A334 gpr15=000003FF62DE7060
j> 07:51:57 psw=000003FF990ADC96 mask=0705000180000000 fpc=00880000 bea=000003FF9923A332
j> 07:51:57 fpr0=000003fedf3a2540 (f: 3745129728.000000, d: 2.170530e-311)
j> 07:51:57 fpr1=3f1fbc93c82cc903 (f: 3358378240.000000, d: 1.210656e-04)
j> 07:51:57 fpr2=45ae3000cea4cded (f: 3466907136.000000, d: 4.671291e+27)
j> 07:51:57 fpr3=3fbada67d508f378 (f: 3574133504.000000, d: 1.048951e-01)
j> 07:51:57 fpr4=3fcd0e0968f0518e (f: 1760579968.000000, d: 2.269909e-01)
j> 07:51:57 fpr5=3fd999999997fa04 (f: 2576873984.000000, d: 4.000000e-01)
j> 07:51:57 fpr6=3f7e0c0aaf43f3a9 (f: 2940466176.000000, d: 7.335703e-03)
j> 07:51:57 fpr7=3fcc71c51d8e78af (f: 495876256.000000, d: 2.222220e-01)
j> 07:51:57 fpr8=000003ff9432fee8 (f: 2486370048.000000, d: 2.172030e-311)
j> 07:51:57 fpr9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
j> 07:51:57 fpr10=000003ff5414ec60 (f: 1410657408.000000, d: 2.171499e-311)
j> 07:51:57 fpr11=000003ffda77e2f8 (f: 3665289984.000000, d: 2.172613e-311)
j> 07:51:57 fpr12=000003ff62ded910 (f: 1658771712.000000, d: 2.171621e-311)
j> 07:51:57 fpr13=000002aa10b626f0 (f: 280372992.000000, d: 1.447340e-311)
j> 07:51:57 fpr14=0000000000101000 (f: 1052672.000000, d: 5.200891e-318)
j> 07:51:57 fpr15=000003ffda77d230 (f: 3665285632.000000, d: 2.172613e-311)
j> 07:51:57 Module=/j9vm/ascii/builds/bld_81406/sdk/xz6480/jre/lib/s390x/default/libj9jit29.so
j> 07:51:57 Module_base_address=000003FF98F00000
j> 07:51:57 
j> 07:51:57 Method_being_compiled=com/ibm/jtc/test/jltf/logging/LoggingSystem.getLogger(Lcom/ibm/jtc/test/jltf/logging/LoggerType;Lcom/ibm/jtc/test/jltf/WorkUnitContext;Ljava/lang/String;)Lcom/ibm/jtc/test/jltf/logging/Logger;
j> 07:51:57 Target=2_90_20241111_81406 (Linux 4.18.0-553.27.1.el8_10.s390x)
j> 07:51:57 CPU=s390x (4 logical CPUs) (0x1f6614000 RAM)
j> 07:51:57 ----------- Stack Backtrace -----------
j> 07:51:59 _ZThn88_N19TR_ResolvedJ9Method27startAddressForJittedMethodEv+0x3e (0x000003FF990ADC96 [libj9jit29.so+0x1adc96])
j> 07:51:59 _ZN2TR30J9S390InterfaceCallDataSnippet15emitSnippetBodyEv+0x3a4 (0x000003FF9923A334 [libj9jit29.so+0x33a334])
j> 07:51:59 _ZN3OMR1Z13CodeGenerator16emitDataSnippetsEv+0x30e (0x000003FF998E6A26 [libj9jit29.so+0x9e6a26])
j> 07:51:59 _ZN3OMR13CodeGenerator12emitSnippetsEv+0x70 (0x000003FF9943A340 [libj9jit29.so+0x53a340])
j> 07:51:59 _ZN3OMR12CodeGenPhase24performEmitSnippetsPhaseEPN2TR13CodeGeneratorEPNS1_12CodeGenPhaseE+0x74 (0x000003FF99442CBC [libj9jit29.so+0x542cbc])
j> 07:51:59 _ZN3OMR12CodeGenPhase10performAllEv+0x136 (0x000003FF99444E1E [libj9jit29.so+0x544e1e])
j> 07:51:59 _ZN3OMR13CodeGenerator12generateCodeEv+0x5c (0x000003FF9943FD94 [libj9jit29.so+0x53fd94])
j> 07:51:59 _ZN3OMR11Compilation7compileEv+0xe1c (0x000003FF9945F144 [libj9jit29.so+0x55f144])
j> 07:51:59 _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadPNS_11CompilationEP17TR_ResolvedMethodR11TR_J9VMBaseP19TR_OptimizationPlanRKNS_16SegmentAllocatorE+0x5ae (0x000003FF9905AA06 [libj9jit29.so+0x15aa06])
j> 07:51:59 _ZN2TR28CompilationInfoPerThreadBase14wrappedCompileEP13J9PortLibraryPv+0x408 (0x000003FF9905B920 [libj9jit29.so+0x15b920])
j> 07:51:59 omrsig_protect+0x3d8 (0x000003FF9A72D7A0 [libj9prt29.so+0x2d7a0])
j> 07:51:59 _ZN2TR28CompilationInfoPerThreadBase7compileEP10J9VMThreadP21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x3f6 (0x000003FF9905954E [libj9jit29.so+0x15954e])
j> 07:51:59 _ZN2TR24CompilationInfoPerThread12processEntryER21TR_MethodToBeCompiledRN2J917J9SegmentProviderE+0x1aa (0x000003FF99059B3A [libj9jit29.so+0x159b3a])
j> 07:51:59 _ZN2TR24CompilationInfoPerThread14processEntriesEv+0x3c8 (0x000003FF99058730 [libj9jit29.so+0x158730])
j> 07:51:59 _ZN2TR24CompilationInfoPerThread3runEv+0x7e (0x000003FF99058A16 [libj9jit29.so+0x158a16])
j> 07:51:59 _Z30protectedCompilationThreadProcP13J9PortLibraryPN2TR24CompilationInfoPerThreadE+0x94 (0x000003FF99058AB4 [libj9jit29.so+0x158ab4])
j> 07:51:59 omrsig_protect+0x3d8 (0x000003FF9A72D7A0 [libj9prt29.so+0x2d7a0])
j> 07:51:59 _Z21compilationThreadProcPv+0x196 (0x000003FF99058F36 [libj9jit29.so+0x158f36])
j> 07:51:59 thread_wrapper+0xfa (0x000003FF9A88923A [libj9thr29.so+0x923a])
j> 07:51:59 start_thread+0xd6 (0x000003FF9B8880DE [libpthread.so.0+0x80de])
j> 07:51:59  (0x000003FF9B5A8B42 [libc.so.6+0x28b42])
j> 07:51:59  (0x0000000000000000 [<unknown>+0x0])
j> 07:51:59 ---------------------------------------

The text was updated successfully, but these errors were encountered:

pshipton · 2024-11-11T17:17:38Z

@hzongaro pls take a look

github-actions · 2024-11-11T17:17:55Z

Issue Number: 20561
Status: Open
Recommended Components: comp:vm, comp:gc, comp:test
Recommended Assignees: gacholio, pshipton, hangshao0

hzongaro · 2024-11-11T18:34:56Z

@r30shah, could you ask someone on your team to take an initial look at this crash? I'm not sure whether it's a problem in code generation or a common optimizer problem. . . .

r30shah · 2024-11-11T20:11:47Z

@VermaSh Can you take a look at this failure ? Looking at the failing stack, it seems to be failing in this loop with bad profiledMethod object [1]. I would load the core-dump and check how the profiledClassList looks and from where we got the profiledMethod.

I launched the grinder build_info.php?build_id=81422 and can not reproduce this in 50x. So let's do the initial analysis from the core-dump and assess the failure first.

[1].

openj9/runtime/compiler/z/codegen/S390J9CallSnippet.cpp

Lines 1484 to 1515 in 83533e1

    
           if (profiledClassesList) 
        
              { 
        
              for (auto valuesIt = profiledClassesList->begin(); valuesIt != profiledClassesList->end(); ++valuesIt) 
        
                 { 
        
                 TR::SymbolReference *methodSymRef = callNode->getSymbolReference(); 
        
                 TR_ResolvedMethod * profiledMethod = methodSymRef->getOwningMethod(comp)->getResolvedInterfaceMethod(comp, 
        
                       (TR_OpaqueClassBlock *)(*valuesIt), methodSymRef->getCPIndex()); 
        
                 numInterfaceCallCacheSlots--; 
        
                 updateField = true; 
        
                 if (comp->target().is64Bit() && TR::Compiler->om.generateCompressedObjectHeaders()) 
        
                    *(uintptr_t *) cursor = (uintptr_t) (*valuesIt) << 32; 
        
                 else 
        
                    *(uintptr_t *) cursor = (uintptr_t) (*valuesIt); 
        
                 if (comp->getOption(TR_EnableHCR)) 
        
                    { 
        
                    cg()->jitAddPicToPatchOnClassRedefinition(*valuesIt, (void *) cursor); 
        
                    } 
        
                 if (cg()->fe()->isUnloadAssumptionRequired((TR_OpaqueClassBlock *)(*valuesIt), comp->getCurrentMethod())) 
        
                    { 
        
                    cg()->jitAddPicToPatchOnClassUnload(*valuesIt, (void *) cursor); 
        
                    } 
        
                 cursor += TR::Compiler->om.sizeofReferenceAddress(); 
        
                 // Method Pointer 
        
                 *(uintptr_t *) (cursor) = (uintptr_t)profiledMethod->startAddressForJittedMethod(); 
        
                 cursor += TR::Compiler->om.sizeofReferenceAddress(); 
        
                 } 
        
              }

r30shah · 2024-11-11T20:14:56Z

I have downloaded the result files on our lnxec dev image to inspect - /home/rahil/temp feel free to copy them from there.

pshipton · 2024-11-15T02:10:22Z

http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96003119
[Linux S390] 80 Load_Level_2.harmony.5mins.Mode110

http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96010585
[Linux S390] 80 Load_Level_2.harmony.5mins.Mode110

pshipton · 2024-11-19T06:16:03Z

http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96247741
[Linux S390 64bit] 80 Load_Level_2.harmony.5mins.Mode187 -Xaggressive -Xgcpolicy:gencon -Xjit -Xnocompressedrefs

50x grinder passed

pshipton · 2024-11-20T18:41:13Z

http://vmfarm.rtp.raleigh.ibm.com/job_output.php?id=96358235
[Linux S390] 80 Load_Level_2.harmony.5mins.Mode110

r30shah · 2024-11-20T20:26:49Z

@VermaSh Can we checkout this one, seems like popping up quite frequently now.

r30shah · 2024-11-21T19:21:09Z

@VermaSh I launched grinder to get the failing logs (http://vmfarm.rtp.raleigh.ibm.com/build_info.php?build_id=82077). I was able to get one failure. We should look into the core-dump to get more idea about the failure.,

VermaSh · 2024-11-21T19:55:23Z

Thank you for launching the grinder @r30shah. Looking into the failure.

VermaSh · 2024-11-28T17:43:01Z

Apologies for the delay as I looking into another zOS jdk21 failure. Having opened a fix pr for that I'll get back to this failure.

pshipton · 2024-12-05T19:51:04Z

I haven't seen this in a couple of week, removing the blocker label.

pshipton added comp:jit test failure segfault Issues that describe segfaults / JVM crashes labels Nov 11, 2024

pshipton added this to the Release 0.49 (Java 8, 11, 17, 21, 23) January refresh milestone Nov 19, 2024

pshipton added the blocker label Nov 20, 2024

pshipton removed the blocker label Dec 5, 2024

hzongaro assigned VermaSh Dec 10, 2024

hzongaro modified the milestones: Release 0.49 (Java 8, 11, 17, 21, 23) January refresh, Release 0.51 (Java 8, 11, 17, 21, 24) Apr refresh Dec 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

harmony crash vmState=0x0005ff0a #20561

harmony crash vmState=0x0005ff0a #20561

pshipton commented Nov 11, 2024 •

edited

Loading

pshipton commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

hzongaro commented Nov 11, 2024

r30shah commented Nov 11, 2024

r30shah commented Nov 11, 2024

pshipton commented Nov 15, 2024 •

edited

Loading

pshipton commented Nov 19, 2024 •

edited

Loading

pshipton commented Nov 20, 2024

r30shah commented Nov 20, 2024 •

edited

Loading

r30shah commented Nov 21, 2024

VermaSh commented Nov 21, 2024

VermaSh commented Nov 28, 2024

pshipton commented Dec 5, 2024

harmony crash vmState=0x0005ff0a #20561

harmony crash vmState=0x0005ff0a #20561

Comments

pshipton commented Nov 11, 2024 • edited Loading

pshipton commented Nov 11, 2024

github-actions bot commented Nov 11, 2024

hzongaro commented Nov 11, 2024

r30shah commented Nov 11, 2024

r30shah commented Nov 11, 2024

pshipton commented Nov 15, 2024 • edited Loading

pshipton commented Nov 19, 2024 • edited Loading

pshipton commented Nov 20, 2024

r30shah commented Nov 20, 2024 • edited Loading

r30shah commented Nov 21, 2024

VermaSh commented Nov 21, 2024

VermaSh commented Nov 28, 2024

pshipton commented Dec 5, 2024

pshipton commented Nov 11, 2024 •

edited

Loading

pshipton commented Nov 15, 2024 •

edited

Loading

pshipton commented Nov 19, 2024 •

edited

Loading

r30shah commented Nov 20, 2024 •

edited

Loading