Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0 #19768

c-koell · 2024-06-27T13:49:25Z

Java -version output

openjdk version "21.0.3" 2024-04-16 LTS
IBM Semeru Runtime Open Edition 21.0.3.0 (build 21.0.3+9-LTS)
Eclipse OpenJ9 VM 21.0.3.0 (build openj9-0.44.0, JRE 21 Windows Server 2022 amd64-64-Bit Compressed References 20240416_179 (JIT enabled, AOT enabled)
OpenJ9 - b069931
OMR - 254af5a04
JCL - ac8f341bc20 based on jdk-21.0.3+9)

Summary of problem

We have a openliberty application running on ibm-semeru-open-jdk_x64_windows_21.0.3_09_openj9-0.44.0.
Today we had jvm crashes with following error in console.log

Unhandled exception
Type=Segmentation error vmState=0x00000000
Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFFD85D42F0 ContextFlags=0010005f
Handler1=00007FFFD856EAA0 Handler2=00007FFFD827AC50 InaccessibleReadAddress=0000000000000003
RDI=0000000000000000 RSI=000000000000000B RAX=0000000000000005 RBX=000001DA0C6058D8
RCX=0000000000000002 RDX=0000000000000000 R8=0000000000000000 R9=0000000000000000
R10=000001DA0C6058D8 R11=00000000000000EB R12=0000000000000008 R13=000001DA0987B9F0
R14=000001DA0C6058D0 R15=00000077E1C7F150
RIP=00007FFFD85D42F0 RSP=00000077E1C7EF28 RBP=0000000000000008 EFLAGS=0000000000010246
FS=0053 ES=002B DS=002B
XMM0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM2 000001da7b4972b4 (f: 2068411008.000000, d: 1.006848e-311)
XMM3 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM4 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
XMM6 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8 00000cb741000000 (f: 1090519040.000000, d: 6.907635e-311)
XMM9 000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
XMM10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=C:\openjdk\bin\default\j9vm29.dll
Module_base_address=00007FFFD84A0000 Offset_in_DLL=00000000001342f0
Target=2_90_20240416_179 (Windows Server 2022 10.0 build 20348)
CPU=amd64 (4 logical CPUs) (0x5fff0a000 RAM)

JavaVMInitArgs.nOptions=69:
    -Xoptionsfile=C:\openjdk\lib\options.default
    -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/PrimitiveHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple
    -XX:+ensurehashed:java/lang/Class,java/lang/Thread
    -Xjcl:jclse29
    -Dcom.ibm.oti.vm.bootstrap.library.path=C:\openjdk\bin\default;C:\openjdk\bin
    -Dsun.boot.library.path=C:\openjdk\bin\default;C:\openjdk\bin
    -Djava.library.path=C:\openjdk\bin\default;C:\openjdk\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Users\ElakWebServerP\AppData\Local\Microsoft\WindowsApps;.
    -Djava.home=C:\openjdk
 
Generated system dump: C:\Users\ElakWebServerP\j9.dmp
----------- Stack Backtrace -----------
J9_GetInterface+0x57ea0 (0x00007FFFD85D42F0 [j9vm29+0x1342f0])
(0x00007FFFD8569F81 [j9vm29+0xc9f81])
JVM_GetExtendedNPEMessage+0x177 (0x00007FFFDB5B8947 [jvm+0x18947])
(0x00007FFFC6BD04A8)
(0x00000007FB82FC80)
(0x00000007FB82E618)
(0x00000007FB82FCA0)
(0x00007FFFC5BE5550)
(0x0000000000000003)
(0x000000000098BD00)
(0x00000007FB6D38E8)
(0x000000007DB0A948)
(0x00000007FB7D8A50)
(0x0000100200000003)
(0x000000000098BD00)
(0x00000077E1C7F430)
(0x00000007FB82E618)
(0x00000007FB82FC80)

Now we are searching for the reason ...
We switched back to to ibm-semeru-open-jdk_x64_windows_21.0.2_13_openj9-0.43.0. I'm not sure the failure comes from the latest jdk but we will see. At the moment the application runs without any crashes :-)

We had the crashes on windows and also on Linux. I will attach the dump file from a windows server.

Diagnostic files

I will attach a j9.dmp file ...

The text was updated successfully, but these errors were encountered:

c-koell · 2024-06-27T13:55:49Z

You can find the dump under following cloud storage https://tbox.tirol.gv.at/index.php/s/GPrM3xrF98pTDWX with "jEzmabtqTR" as password

pshipton · 2024-06-27T21:05:03Z

@tajila @JasonFengJ9 fyi

pshipton · 2024-06-27T21:07:02Z

Perhaps related to #19163 ?

pshipton · 2024-06-27T21:10:43Z

Here is a preliminary Windows jdk21 build of 0.46 you can try, in case it's fixed.
https://ibm.box.com/s/d9jcgx4ggb46mycq2duzvz5hw497kzka

pshipton · 2024-06-27T21:14:03Z

and xLinux as well
https://ibm.box.com/s/cgxoi65pjrustf8f8wi6qyhukg0fqd6b

pshipton · 2024-06-27T22:04:45Z

There is some confusion with the title and description, but the core is from openj9-0.44.0

JRE 21 Windows Server 2022 amd64-64-Bit Compressed References 20240416_179 (JIT enabled, AOT enabled)
OpenJ9   - b0699311c7
OMR      - 254af5a04
JCL      - ac8f341bc20 based on jdk-21.0.3+9

Also I zipped the 3GB file and transferred it to Box.
https://ibm.box.com/s/u56fke31u817a943x2rvsmpx63ocvg7m

c-koell · 2024-06-28T03:48:07Z

There is some confusion with the title and description, but the core is from openj9-0.44.0
JRE 21 Windows Server 2022 amd64-64-Bit Compressed References 20240416_179 (JIT enabled, AOT enabled)
OpenJ9   - b0699311c7
OMR      - 254af5a04
JCL      - ac8f341bc20 based on jdk-21.0.3+9
Also I zipped the 3GB file and transferred it to Box.
https://ibm.box.com/s/u56fke31u817a943x2rvsmpx63ocvg7m

Sorry for the confusion. I have changed the title and description.

JasonFengJ9 · 2024-06-28T11:53:01Z

@c-koell Is this an intermittent failure? Do you happen to have a standalone test case?

c-koell · 2024-06-28T16:29:01Z

Yes it is an intermittent failure.

c-koell · 2024-07-01T05:24:05Z

FYI .. as a fast fix we have tried to set the flag -XX:-ShowCodeDetailsInExceptionMessages as mentioned in #19163

Since that we havn't seen any crashes.

c-koell · 2024-07-12T06:00:50Z

@JasonFengJ9 dou you think our problem is the same as #19163

JasonFengJ9 · 2024-07-15T11:10:50Z

dou you think our problem is the same as #19163
Here is a preliminary Windows jdk21 build of 0.46 you can try, in case it's fixed.
https://ibm.box.com/s/d9jcgx4ggb46mycq2duzvz5hw497kzka

@c-koell did you try the preliminary 0.46 build?

c-koell · 2024-07-15T12:11:20Z

Hi @JasonFengJ9.
Unfortionally not. We have set the flag -XX:-ShowCodeDetailsInExceptionMessages and sinse that we had no crashes.
The application where the crash has occured is a key application. We where happy since that it runs now.

When the official release is out we have planned to test it with that application.

JasonFengJ9 · 2024-07-15T12:25:41Z

When the official release is out we have planned to test it with that application.

@c-koell Thanks for the update. In the meantime, is there any standalone test case we can try?

c-koell · 2024-07-15T12:37:36Z

@JasonFengJ9 unfortunately not. We have no idea why this happens only in our key application :-) We have a lot of other applications running on the latest release without any problems.

JasonFengJ9 · 2024-08-09T14:08:36Z

@c-koell could you re-try with OpenJ9 latest 0.46 release https://github.com/ibmruntimes/semeru21-binaries/releases?

c-koell · 2024-08-09T17:06:18Z

Yes of course! But it will take some time because I'm on vacation

c-koell · 2024-08-27T07:31:52Z

@JasonFengJ9 @pshipton I'm back in office.
I have tried the new release on some local test machines but now we see really often segmentation errors. I don't know if this belongs to this issue but i have attached the Dump files. Can you please have a look .. thanks !

I have uploaded the File to our cloud storage https://tbox.tirol.gv.at/index.php/s/tCHtngz4o6CpCNg with "Rk5ic7RBn9" as password

JasonFengJ9 · 2024-08-27T12:44:40Z

gpInfo:
Windows_ExceptionCode=c0000005 J9Generic_Signal=00000004 ExceptionAddress=00007FFA5C195E0E ContextFlags=0010005f
Handler1=00007FFA5BB9ED30 Handler2=00007FFA9231AC60 InaccessibleReadAddress=000000009966995A
RDI=0000000002CC4100 RSI=0000000000000548 RAX=000000000101ED00 RBX=00000000018CA600
RCX=0000000099669966 RDX=0000000000FE6B00 R8=000000000000003F R9=0000000000000400
R10=00007FFA5C202908 R11=0000004DFD279930 R12=0000000000000001 R13=00007FF47A200000
R14=0000000000000000 R15=0000004DFD27E758
RIP=00007FFA5C195E0E RSP=0000004DFD2798C0 RBP=0000000000000001 EFLAGS=0000000000010206
FS=0053 ES=002B DS=002B
XMM0=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM1=2f6d62692f6d6f63 (f: 795701120.000000, d: 3.097763e-80)
XMM2=000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
XMM3=000000003f400000 (f: 1061158912.000000, d: 5.242822e-315)
XMM4=00007ff47a1bc020 (f: 2048638976.000000, d: 6.950911e-310)
XMM5=6870736265772f6d (f: 1702309760.000000, d: 1.200887e+195)
XMM6=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM7=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM8=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
XMM15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=C:\Program Files\openjdk-21\bin\default\j9jit29.dll
Module_base_address=00007FFA5BB30000 Offset_in_DLL=0000000000665e0e

Method_being_compiled=com/ibm/ws/connectionpool/monitor/ConnectionPoolMonitor.decConnectionHandleCount(Ljava/lang/Object;)V

> !j9vmthread 0x22100 | grep omrVMThread
	0x9b8: class OMR_VMThread* omrVMThread = !omr_vmthread 0x0000000000022B78

> !omr_vmthread 0x0000000000022B78 | grep vmState
	0x40: U64 vmState = 0x00000000000501FF (328191)

vmState [0x501ff]: {J9VMSTATE_JIT} {inlining}

This is a segmentation error at j9jit29.dll which is different from the original JVM_GetExtendedNPEMessage at j9vm29.dll.

@c-koell are all the crashes the same as this one?

FYI @hzongaro

c-koell · 2024-08-27T13:05:26Z

Yes, i think all the crashes are the same ...
I have attached a other dmp file (other_dmp.zip) to the share so you can check it.

Should i open a other issue for that problem ?

JasonFengJ9 · 2024-08-27T13:19:02Z

Yes, i think all the crashes are the same ...
I have attached a other dmp file (other_dmp.zip) to the share so you can check it.

You are right, this dmp file shows same segmentation error.

Should i open a other issue for that problem ?

Yes, please

hzongaro · 2024-08-27T18:26:25Z

Looking at the dump for this most recent problem in windbg, I see this stack trace:

[0xf]   j9jit29!TR_J9VMBase::getResolvedInterfaceMethod+0x98   0x222ca795d0   0x7ffb102e0296
[0x10]   j9jit29!TR_ResolvedJ9Method::getResolvedInterfaceMethod+0x46   0x222ca79620   0x7ffb102d083b
[0x11]   j9jit29!CollectImplementors::visitSubclass+0x8b   0x222ca79680   0x7ffb102cfa84
[0x12]   j9jit29!TR_SubclassVisitor::visitSubclasses+0xd4   0x222ca796c0   0x7ffb102cf936
[0x13]   j9jit29!TR_SubclassVisitor::visit+0x136   0x222ca79710   0x7ffb102cfccb
[0x14]   j9jit29!TR_ClassQueries::collectImplementorsCapped+0x12b   0x222ca79770   0x7ffb102f1901
[0x15]   j9jit29!TR_PersistentCHTable::findSingleImplementer+0xb1   0x222ca79800   0x7ffb103fc982
[0x16]   j9jit29!TR_J9InterfaceCallSite::findCallSiteTargetImpl+0x1b2   0x222ca79870   0x7ffb103fc589
[0x17]   j9jit29!TR_J9InterfaceCallSite::findCallSiteTarget+0x59   0x222ca79930   0x7ffb10607c92
[0x18]   j9jit29!TR_InlinerBase::getSymbolAndFindInlineTargets+0x372   0x222ca799b0   0x7ffb103d4161

This looks similar to the stack trace reported for issue #19765. The fix for #19765 didn't make it into the 0.46 release.

@c-koell, would it be possible for you to try rerunning with the JVM option -XX:+EnableExtendedHCR to see whether that avoids this most recent crash?

pshipton · 2024-08-27T21:06:06Z

The fix #20062 is already added to the v0.46.1-release branch.

c-koell · 2024-08-28T11:34:47Z

@hzongaro @pshipton with the JVM option -XX:+EnableExtendedHCR we see no more crashes.
@JasonFengJ9 i think i can skip to create a extra ticket for the crashes with the latest release.

So we can focus now again to the initial segmenation crashes. We think we will wait now to next stable release to test against that segmentation failure. Is there a plan to release a semeru runtime with a 0.46.1 release ?

I can say that with we cannot use the latest version without the jvm option -XX:+EnableExtendedHCR

pshipton · 2024-08-28T12:21:43Z

Is there a plan to release a semeru runtime with a 0.46.1 release ?

Yes there is, but we don't yet have a timeline.

JasonFengJ9 · 2024-09-10T19:03:03Z

@c-koell OpenJ9 0.46.1 release is available now - https://github.com/ibmruntimes/semeru21-binaries/releases/tag/jdk-21.0.4%2B7_openj9-0.46.1, please give it a try.

c-koell · 2024-09-11T05:21:02Z

@JasonFengJ9 Thanks for the info. Unfortunately i do not see any windows images ?

JasonFengJ9 · 2024-09-11T14:18:31Z

@c-koell The Windows (and Mac) images are on their way, sorry for the confusion. I will let you know when they arrive.

JasonFengJ9 · 2024-09-12T15:24:33Z

@c-koell The JDK21 0.46.1 Windows images are available now at https://github.com/ibmruntimes/semeru21-binaries/releases/tag/jdk-21.0.4%2B7_openj9-0.46.1

c-koell · 2024-10-02T08:43:01Z

@JasonFengJ9 We have now tried the 0.46.1 release. We see no more segmentation errors 👍

github-actions · 2024-10-02T08:43:13Z

Issue Number: 19768
Status: Closed
Actual Components: comp:vm, userRaised
Actual Assignees: No one :(
PR Assignees: No one :(

JasonFengJ9 · 2024-10-02T11:36:16Z

Thanks @c-koell for the verification.

c-koell added the userRaised label Jun 27, 2024

pshipton added the comp:vm label Jun 27, 2024

pshipton added this to the Release 0.48 (Java 8, 11, 17, 21, 23) October refresh milestone Jun 27, 2024

JasonFengJ9 self-assigned this Jun 27, 2024

c-koell changed the title ~~Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.2_13_openj9-0.43.0~~ Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0 Jun 28, 2024

GioChkhaidze mentioned this issue Jul 22, 2024

Final Issue Tigers-X/openj9#19

Closed

pshipton modified the milestones: Release 0.48 (Java 8, 11, 17, 21, 23) October refresh, Release 0.46.1 (Java 8, 11, 17, 21, 22) Sep Sep 16, 2024

c-koell closed this as completed Oct 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0 #19768

Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0 #19768

c-koell commented Jun 27, 2024 •

edited

Loading

c-koell commented Jun 27, 2024 •

edited

Loading

pshipton commented Jun 27, 2024

pshipton commented Jun 27, 2024

pshipton commented Jun 27, 2024 •

edited

Loading

pshipton commented Jun 27, 2024

pshipton commented Jun 27, 2024

c-koell commented Jun 28, 2024

JasonFengJ9 commented Jun 28, 2024

c-koell commented Jun 28, 2024

c-koell commented Jul 1, 2024

c-koell commented Jul 12, 2024

JasonFengJ9 commented Jul 15, 2024

c-koell commented Jul 15, 2024

JasonFengJ9 commented Jul 15, 2024

c-koell commented Jul 15, 2024

JasonFengJ9 commented Aug 9, 2024

c-koell commented Aug 9, 2024

c-koell commented Aug 27, 2024

JasonFengJ9 commented Aug 27, 2024 •

edited by pshipton

Loading

c-koell commented Aug 27, 2024

JasonFengJ9 commented Aug 27, 2024

hzongaro commented Aug 27, 2024

pshipton commented Aug 27, 2024 •

edited

Loading

c-koell commented Aug 28, 2024

pshipton commented Aug 28, 2024

JasonFengJ9 commented Sep 10, 2024

c-koell commented Sep 11, 2024

JasonFengJ9 commented Sep 11, 2024

JasonFengJ9 commented Sep 12, 2024

c-koell commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

JasonFengJ9 commented Oct 2, 2024

Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0 #19768

Segmentation error with ibm-semeru-open-jdk_x64_windows_21.0.3+9_openj9-0.44.0 #19768

Comments

c-koell commented Jun 27, 2024 • edited Loading

Java -version output

Summary of problem

Diagnostic files

c-koell commented Jun 27, 2024 • edited Loading

pshipton commented Jun 27, 2024

pshipton commented Jun 27, 2024

pshipton commented Jun 27, 2024 • edited Loading

pshipton commented Jun 27, 2024

pshipton commented Jun 27, 2024

c-koell commented Jun 28, 2024

JasonFengJ9 commented Jun 28, 2024

c-koell commented Jun 28, 2024

c-koell commented Jul 1, 2024

c-koell commented Jul 12, 2024

JasonFengJ9 commented Jul 15, 2024

c-koell commented Jul 15, 2024

JasonFengJ9 commented Jul 15, 2024

c-koell commented Jul 15, 2024

JasonFengJ9 commented Aug 9, 2024

c-koell commented Aug 9, 2024

c-koell commented Aug 27, 2024

JasonFengJ9 commented Aug 27, 2024 • edited by pshipton Loading

c-koell commented Aug 27, 2024

JasonFengJ9 commented Aug 27, 2024

hzongaro commented Aug 27, 2024

pshipton commented Aug 27, 2024 • edited Loading

c-koell commented Aug 28, 2024

pshipton commented Aug 28, 2024

JasonFengJ9 commented Sep 10, 2024

c-koell commented Sep 11, 2024

JasonFengJ9 commented Sep 11, 2024

JasonFengJ9 commented Sep 12, 2024

c-koell commented Oct 2, 2024

github-actions bot commented Oct 2, 2024

JasonFengJ9 commented Oct 2, 2024

c-koell commented Jun 27, 2024 •

edited

Loading

c-koell commented Jun 27, 2024 •

edited

Loading

pshipton commented Jun 27, 2024 •

edited

Loading

JasonFengJ9 commented Aug 27, 2024 •

edited by pshipton

Loading

pshipton commented Aug 27, 2024 •

edited

Loading