HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

JasonFengJ9 · 2024-05-06T21:48:12Z

Failure link

From an internal build(macaarch64rt8):

openjdk version "22.0.1-beta" 2024-04-16
IBM Semeru Runtime Open Edition 22.0.1+8-202405041602 (build 22.0.1-beta+8-202405041602)
Eclipse OpenJ9 VM 22.0.1+8-202405041602 (build master-2c34fcf0fb, JRE 22 Mac OS X aarch64-64-Bit 20240504_59 (JIT enabled, AOT enabled)
OpenJ9   - 2c34fcf0fb
OMR      - acd6f7e0b
JCL      - 7b1ea2cb9 based on jdk-22.0.1+8)

Rerun in Grinder - Change TARGET to run only the failed test targets.

Optional info

Failure output (captured from console output)

[2024-05-05T04:45:48.550Z] variation: Mode110
[2024-05-05T04:45:48.550Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

[2024-05-05T04:48:11.849Z] HHLT 00:48:10.589 - Completed 46.7%. Number of tests started=341621 (+47856)
[2024-05-05T04:48:23.425Z] HHLT stderr Unhandled exception
[2024-05-05T04:48:23.425Z] HHLT stderr Type=Segmentation error vmState=0x00000000
[2024-05-05T04:48:23.425Z] HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2024-05-05T04:48:23.425Z] HHLT stderr Handler1=0000000100A9DC98 Handler2=000000010094CFE0 InaccessibleAddress=0000000099669956
[2024-05-05T04:48:23.425Z] HHLT stderr x0=000000013C964500 x1=00000001705C27C8 x2=0000000150133C00 x3=0000000150133C00
[2024-05-05T04:48:23.425Z] HHLT stderr x4=00000001705C27D8 x5=00000001705C27C0 x6=00000001705C27B8 x7=000000036ACE7C60
[2024-05-05T04:48:23.425Z] HHLT stderr x8=000000014169B100 x9=000000013C964500 x10=0000000099669966 x11=00000001705C27C8
[2024-05-05T04:48:23.425Z] HHLT stderr x12=00000001705C27D8 x13=00000001705C27B0 x14=0000000100AE98FC x15=0000000000000010
[2024-05-05T04:48:23.425Z] HHLT stderr x16=00000001705C2820 x17=00000001EA494DB0 x18=00000001705C2301 x19=0000000040000000
[2024-05-05T04:48:23.425Z] HHLT stderr x20=0000000099669966 x21=00000001705C27B0 x22=0000000110D3ED1C x23=00000001705C27C0
[2024-05-05T04:48:23.425Z] HHLT stderr x24=00000001705C27B0 x25=00000001705C27C8 x26=00000001705C27C0 x27=00000001705C27B8
[2024-05-05T04:48:23.425Z] HHLT stderr x28=0000000100BC6D84 x29(FP)=00000001705C27A0 x30(LR)=0000000100AE937C x31(SP)=00000001705C22C0
[2024-05-05T04:48:23.425Z] HHLT stderr PC=0000000100AFE318 SP=00000001705C22C0
[2024-05-05T04:48:23.425Z] HHLT stderr v0 00000001340a0309 (f: 873071360.000000, d: 2.553350e-314)
[2024-05-05T04:48:23.425Z] HHLT stderr v1 000000013680bfb8 (f: 914407360.000000, d: 2.573773e-314)
[2024-05-05T04:48:23.425Z] HHLT stderr v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2024-05-05T04:48:23.425Z] HHLT stderr v3 3f51d8e2cbc49e05 (f: 3418660352.000000, d: 1.089307e-03)
[2024-05-05T04:48:23.425Z] HHLT stderr v4 bf9411c4195bf130 (f: 425455936.000000, d: -1.959902e-02)
[2024-05-05T04:48:23.425Z] HHLT stderr v5 3f51d8e2cbc49e1c (f: 3418660352.000000, d: 1.089307e-03)
[2024-05-05T04:48:23.425Z] HHLT stderr v6 4018000000000000 (f: 0.000000, d: 6.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v7 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
[2024-05-05T04:48:23.425Z] HHLT stderr v8 000003c10000745f (f: 29791.000000, d: 2.039238e-311)
[2024-05-05T04:48:23.425Z] HHLT stderr v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2024-05-05T04:48:23.425Z] HHLT stderr v17 3fd54f257ba2fd55 (f: 2074279296.000000, d: 3.329557e-01)
[2024-05-05T04:48:23.425Z] HHLT stderr v18 3f58ba9f20072c0c (f: 537340928.000000, d: 1.509338e-03)
[2024-05-05T04:48:23.425Z] HHLT stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2024-05-05T04:48:23.425Z] HHLT stderr v20 c1150fb645a1cac1 (f: 1168231168.000000, d: -3.450696e+05)
[2024-05-05T04:48:23.425Z] HHLT stderr v21 c1150fb600000000 (f: 0.000000, d: -3.450695e+05)
[2024-05-05T04:48:23.425Z] HHLT stderr v22 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-05T04:48:23.425Z] HHLT stderr v23 bca8000000000000 (f: 0.000000, d: -1.665335e-16)
[2024-05-05T04:48:23.425Z] HHLT stderr v24 b948000000000000 (f: 0.000000, d: -9.244464e-33)
[2024-05-05T04:48:23.425Z] HHLT stderr v25 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-05T04:48:23.425Z] HHLT stderr v26 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-05T04:48:23.425Z] HHLT stderr v27 401c125cd987c2bb (f: 3649553152.000000, d: 7.017932e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v28 bff0000000000000 (f: 0.000000, d: -1.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v29 bff0000000000000 (f: 0.000000, d: -1.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v30 3ff5555555555555 (f: 1431655808.000000, d: 1.333333e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr v31 4000000000000000 (f: 0.000000, d: 2.000000e+00)
[2024-05-05T04:48:23.425Z] HHLT stderr Module=/Users/jenkins/workspace/Test_openjdk22_j9_extended.system_aarch64_mac_testList_0/jdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2024-05-05T04:48:23.425Z] HHLT stderr Module_base_address=0000000100A78000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
[2024-05-05T04:48:23.425Z] HHLT stderr Symbol_address=0000000100AE9388
[2024-05-05T04:48:23.425Z] HHLT stderr Target=2_90_20240504_59 (Mac OS X 13.0)
[2024-05-05T04:48:23.425Z] HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
[2024-05-05T04:48:23.425Z] HHLT stderr ----------- Stack Backtrace -----------
[2024-05-05T04:48:23.425Z] HHLT stderr ---------------------------------------
[2024-05-05T04:48:23.425Z] HHLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/05/05 00:48:21 - please wait.

[2024-05-05T04:48:47.158Z] HeapHogLoadTest_5m_0_FAILED

50x internal grinder - 2/50 failed at macaarch64rt5

There was a similar failure:

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #14713

The text was updated successfully, but these errors were encountered:

pshipton · 2024-05-07T19:33:14Z

@knn-k can you pls take the first look at the core.

pshipton · 2024-05-07T19:35:42Z

@babsingh fyi in case it's related to #14713

knn-k · 2024-05-08T08:02:40Z

It is a SEGV caused by an instruction trying to load from an address pointed by x10, which holds 0x99669966 eyecatcher value.

   8630c: 08 17 40 f9  	ldr	x8, [x24, #40]
   86310: 14 01 40 f9  	ldr	x20, [x8]
   86314: ea 03 14 aa  	mov	x10, x20
   86318: 4b 0d 9f b8  	ldrsw	x11, [x10, #-16]! <- SEGV here, x10=x20=0x99669966
   8631c: 4a 01 0b 8b  	add	x10, x10, x11
   86320: 0b 03 40 f9  	ldr	x11, [x24]
   86324: 6b 15 4c f9  	ldr	x11, [x11, #6184]
   86328: 4c 01 40 79  	ldrh	w12, [x10]
   8632c: 8d 69 6a 38  	ldrb	w13, [x12, x10]
   86330: bf 6d 01 71  	cmp	w13, #91
   86334: a0 01 00 54  	b.eq	0x86368 <__ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread+0x14fe0>

knn-k · 2024-05-08T10:29:51Z

x20 is loaded from [x8], and jdmpview shows x8 is a pointer to a J9Class MethodTypeForm as shown below.

> !j9class 0x14169b100
J9Class at 0x14169b100 {
  Fields for J9Class:
	0x0: UDATA eyecatcher = 0x0000000099669966 (2573637990)
	0x8: struct J9ROMClass* romClass = !j9romclass 0x0000000121429CA8
	0x10: void** superclasses = !j9x 0x000000014169AFE0
	0x18: UDATA classDepthAndFlags = 0x00000000020E0001 (34471937)
	0x20: U32 classDepthWithFlags = 0x00000000 (0)
	0x24: U32 classFlags = 0x00000000 (0)
	0x28: struct J9ClassLoader* classLoader = !j9classloader 0x000000013C088668
	0x30: struct J9Object* classObject = !j9object 0x00000002800087A0 // java/lang/Class
	0x38: volatile UDATA initializeStatus = 0x0000000000000001 (1)
	0x40: struct J9Method* ramMethods = !j9method 0x000000014169B318 // java/lang/invoke/MethodTypeForm.erasedType()Ljava/lang/invoke/MethodType;
	0x48: UDATA* ramStatics = !j9x 0x000000014169B518
	0x50: struct J9Class* arrayClass = !j9class 0x0000000000000000
	0x58: UDATA totalInstanceSize = 0x0000000000000030 (48)
	0x60: struct J9ITable* lastITable = !j9itable 0x0000000100BC7878
	0x68: UDATA* instanceDescription = !j9x 0x000000000000003D
	0x70: UDATA* instanceLeafDescription = !j9x 0x0000000000000001
	0x78: UDATA instanceHotFieldDescription = 0x0000000000000004 (4)
	0x80: UDATA selfReferencingField1 = 0x0000000000000000 (0)
	0x88: UDATA selfReferencingField2 = 0x0000000000000000 (0)
	0x90: struct J9Method* initializerCache = !j9method 0x0000000000000000
	0x98: UDATA romableAotITable = 0x000000010209C000 (4329160704)
	0xa0: UDATA packageID = 0x0000000121391F01 (4852358913)
	0xa8: struct J9Module* module = !j9module 0x000000013C089868
	0xb0: struct J9Class* subclassTraversalLink = !j9class 0x000000014169AA00 // java/lang/invoke/LambdaForm
	0xb8: struct J9Class* subclassTraversalReverseLink = !j9class 0x000000014169C100 // jdk/internal/vm/Continuation
	0xc0: void** iTable = !j9x 0x0000000000000000
	0xc8: UDATA castClassCache = 0x00000001416BC701 (5392549633)
	0xd0: void** jniIDs = !j9x 0x0000000000000000
	0xd8: UDATA lockOffset = 0x0000000000000008 (8)
	0xe0: U32 paddingForGLRCounters = 0x00000000 (0)
	0xe4: U16 reservedCounter = 0x0000 (0)
	0xe6: U16 cancelCounter = 0x0000 (0)
	0xe8: UDATA newInstanceCount = 0x00000000000003E8 (1000)
	0xf0: IDATA backfillOffset = 0x0000000000000038 (56)
	0xf8: struct J9Class* replacedClass = !j9class 0x0000000000000000
	0x100: UDATA finalizeLinkOffset = 0x0000000000000000 (0)
	0x108: struct J9Class* nextClassInSegment = !j9class 0x000000014169AA00 // java/lang/invoke/LambdaForm
	0x110: UDATA* ramConstantPool = !j9x 0x000000014169ACA0
	0x118: struct J9Object** callSites = !j9x 0x0000000000000000
	0x120: struct J9Object** invokeCache = !j9x 0x0000000000000000
	0x128: struct J9Object** varHandleMethodTypes = !j9x 0x0000000000000000
	0x130: struct J9VMCustomSpinOptions* customSpinOption = !j9vmcustomspinoptions 0x0000000000000000
	0x138: void** staticSplitMethodTable = !j9x 0x0000000000000000
	0x140: void** specialSplitMethodTable = !j9x 0x0000000000000000
	0x148: struct J9JITExceptionTable* jitMetaDataList = !j9jitexceptiontable 0x0000000000000000
	0x150: struct J9Class* gcLink = !j9class 0x0000000000000000
	0x158: struct J9Class* hostClass = !j9class 0x000000014169B100 // java/lang/invoke/MethodTypeForm
	0x160: struct J9Class* nestHost = !j9class 0x0000000000000000
	0x168: struct J9FlattenedClassCache* flattenedClassCache = !j9flattenedclasscache 0x0000000000000000
	0x170: struct J9ClassHotFieldsInfo* hotFieldsInfo = !j9classhotfieldsinfo 0x0000000000000000
	0x178: struct J9MemberNameListNode* memberNames = !j9membernamelistnode 0x0000000000000000
}
Class name: java/lang/invoke/MethodTypeForm
To view static fields, use !j9statics 0x000000014169B100
To view instance shape, use !j9classshape 0x000000014169B100

knn-k · 2024-05-09T05:27:43Z

Many registers in the register dump have the addresses of a specific region.
0x1705C27A0-0x1705C27D8 in x1, x4, x5, x6, x11, x12, x13, x21, x23, x24, x25, x26, x27, x29

The memory dump looks like a table of addresses.

The x8 value for MethodTypeForm above was loaded from 0x1705c27d8.
0x13c019e20 at 0x1705c27b0 is j9javavm.
0x110d3ed1c at 0x1705c27c8 is a jitReturnAddress in crashInfo. It is in the address range of JITed invokeVirtual().

> hexdump 0x1705C27A0 80

1705c27a0: 30285c70 01000000 7c93ae00 01000000  |0(\p....|.......|
1705c27b0: 209e013c 01000000 0045963c 01000000  | ..<.....E.<....|
1705c27c0: a0000a34 01000000 1cedd310 01000000  |...4............|
1705c27d0: 00000000 00000000 00b16941 01000000  |..........iA....|
1705c27e0: 01000000 00000000 10000000 00000000  |................|

> !whatis 0x013c019e20
Found 0x000000013C019E20 as !j9javavm 0x13c019e20
Match found
> !whatis 0x0110d3ed1c
Found 0x0000000110D3ED1C as !void 0x110d3ed1c: !j9javavm 0x13c019e20->j9ras->crashInfo->failingThread->linkNext->jitReturnAddress
Match found

> info jitm
...
        start=0x110d3ec68  end=0x110d3edd0   java/lang/invoke/LambdaForm$DMH/0x000000003c99c820::invokeVirtual(Ljava/lang/Object;Ljava/lang/Object;I)F
...

babsingh · 2024-05-09T15:00:33Z

@knn-k DDR cmds to find the location of the crash in the interpreter:

> !gpinfo
Failing Thread: !j9vmthread 0x13c95ef00
Failing Thread ID: 0x14797b8a (343505802)
....

> !stackslots 0x13c95ef00
<13c95ef00> *** BEGIN STACK WALK, flags = 00400001 walkThread = 5,311,426,304 ***
<13c95ef00> 	ITERATE_O_SLOTS
<13c95ef00> 	RECORD_BYTECODE_PC_OFFSET
<13c95ef00> Initial values: walkSP = 0x0000000134810CB0, PC = 0x000000013622C451, literals = 0x0000000136819588, A0 = 0x0000000134810D88, j2iFrame = 0x0000000134810D78, decomp = 0x0000000000000000
<13c95ef00> J2I frame: bp = 0x0000000134810D78, sp = 0x0000000134810CB0, pc = 0x000000013622C451, cp = 0x00000001368194C0, arg0EA = 0x0000000134810D88, flags = 0x0000000010000000
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000136819588
<13c95ef00> 	Bytecode index = 13 <--- Bytecode being executed
<13c95ef00> 	Using local mapper
<13c95ef00> 	Locals starting at 0x0000000134810D88 for 0x0000000000000002 slots
<13c95ef00> 		I-Slot: a0[0x0000000134810D88] = 0x00000002805D0080
<13c95ef00> 		I-Slot: a1[0x0000000134810D80] = 0x000000036C3169F8
May 08, 2024 6:23:59 PM com.ibm.j9ddr.vm29.events.DefaultEventListener corruptData
WARNING: CorruptData encountered iterating o-slots. walkThread = 0x000000013C95EF00
com.ibm.j9ddr.CorruptDataException: Operand stack underflow in StackMap
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$J9MappingStack.POP(StackMap.java:184)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$StackMap_V1.outputStackMap(StackMap.java:728)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap$StackMap_V1.j9stackmap_StackBitsForPC(StackMap.java:271)
	at com.ibm.j9ddr.vm29.j9.stackmap.StackMap.j9stackmap_StackBitsForPC(StackMap.java:106)
	....

<13c95ef00> JIT frame: bp = 0x0000000134810DC8, pc = 0x0000000110521538, unwindSP = 0x0000000134810D80, cp = 0x000000013C138AC0, arg0EA = 0x0000000134810DE0, jitInfo = 0x0000000141A9A9E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$DMH/0x000000003c09ac20.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x000000013C138BA8
<13c95ef00> 	Bytecode index = 11, inlineDepth = 0, PC offset = 0x0000000000000110
<13c95ef00> 	stackMap=0x0000000141A9AB04, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFD8)
<13c95ef00> 	Described JIT args starting at 0x0000000134810DD0 for U16(0x0003) slots
<13c95ef00> 		O-Slot: : a2[0x0000000134810DD0] = 0x000000036C3169F8
<13c95ef00> 		O-Slot: : a1[0x0000000134810DD8] = 0x00000002805D0080
<13c95ef00> 		O-Slot: : a0[0x0000000134810DE0] = 0x0000000280BEAD58
<13c95ef00> 	Described JIT temps starting at 0x0000000134810DA0 for IDATA(0x0000000000000005) slots
<13c95ef00> 		O-Slot: : t4[0x0000000134810DA0] = 0x0000000000000000
...

<13c95ef00> JIT inline frame: bp = 0x0000000134810E68, pc = 0x00000001109CA128, unwindSP = 0x0000000134810DD0, cp = 0x00000001501302C0, arg0EA = 0x0000000000000000, jitInfo = 0x000000016007D7E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NamedFunction.invokeWithArguments([Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000150130A10
<13c95ef00> 	Bytecode index = 21, inlineDepth = 1, PC offset = 0x00000001109C9DA7
<13c95ef00> JIT frame: bp = 0x0000000134810E68, pc = 0x00000001109CA128, unwindSP = 0x0000000134810DD0, cp = 0x0000000141698B00, arg0EA = 0x0000000134810E80, jitInfo = 0x000000016007D7E8
<13c95ef00> 	Method: java/lang/invoke/LambdaForm.interpretName(Ljava/lang/invoke/LambdaForm$Name;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x000000014169A3F8
<13c95ef00> 	Bytecode index = 123, inlineDepth = 0, PC offset = 0x0000000000000338
<13c95ef00> 	stackMap=0x000000016007DAE4, slots=I16(0x0003) parmBaseOffset=I16(0x0008), parmSlots=U16(0x0003), localBaseOffset=I16(0xFFC8)
<13c95ef00> 	Described JIT args starting at 0x0000000134810E70 for U16(0x0003) slots
<13c95ef00> 		O-Slot: : a2[0x0000000134810E70] = 0x000000036C316998
<13c95ef00> 		O-Slot: : a1[0x0000000134810E78] = 0x0000000280BECFD0
<13c95ef00> 		O-Slot: : a0[0x0000000134810E80] = 0x0000000280BECEC8
<13c95ef00> 	Described JIT temps starting at 0x0000000134810E30 for IDATA(0x0000000000000007) slots
<13c95ef00> 		I-Slot: : t6[0x0000000134810E30] = 0x0000000000000001
...

> !j9method 0x0000000136819588
J9Method at 0x136819588 {
  Fields for J9Method:
	0x0: U8* bytecodes = !j9x 0x000000013622C444 // "*+2+2�"
	0x8: struct J9ConstantPool* constantPool = !j9constantpool 0x00000001368194C0 (flags = 0x0)
	0x10: void* methodRunAddress = !j9x 0x0000000000000006
	0x18: volatile void* extra = !j9x 0x0000000000000F93
}
Signature: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !bytecodes 0x0000000136819588
ROM Method: !j9rommethod 0x000000013622C430
Next Method: !j9method 0x00000001368195A8

> !bytecodes 0x0000000136819588
  Name: invoke_LI_F
  Signature: (Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
  Access Flags (20080008): default static
  Internal Attribute Flags: @FrameIteratorSkip
  ...

    0 aload0
    ...
    7 checkcast 3 java/lang/Integer
   10 invokevirtual 4 java/lang/Integer.intValue()I
   13 invokevirtual 6 java/lang/invoke/MethodHandle.invokeBasic(Ljava/lang/Object;I)F <--- Bytecode being run
   16 invokestatic 8 java/lang/Float.valueOf(F)Ljava/lang/Float;
   19 return1

> !j9vmthread 0x13c95ef00
J9VMThread at 0x13c95ef00 {
  Fields for J9VMThread:
        ...
	0x20: UDATA* sp = !j9x 0x0000000134810CB0
	0x28: U8* pc = !j9x 0x000000013622C451 // "�"
	0x30: struct J9Method* literals = !j9method 0x0000000136819588 // java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
	0x38: UDATA jitStackFrameFlags = 0x0000000000000000 (0)
	0x40: struct J9Object* jitException = !j9object 0x000000036BFDA050 // java/lang/ArrayIndexOutOfBoundsException
	0x48: struct J9Object* currentException = !j9object 0x0000000000000000
	0x50: UDATA* stackOverflowMark = !j9x 0xFFFFFFFFFFFFFFFF
	0x58: UDATA* stackOverflowMark2 = !j9x 0x000000013480CC28
        ...
	0xe8: UDATA tempSlot = 0xFFFFFFFFFFFFFFA8 (-88)
	0xf0: void* jitReturnAddress = !j9x 0x0000000110988410
	0xf8: void* floatTemp1 = !j9x 0x0000000000000000
	0x100: void* floatTemp2 = !j9x 0x00000000000CFB1C
	0x108: void* floatTemp3 = !j9x 0x0000000110C01C68
	0x110: void* floatTemp4 = !j9x 0x0000000000000000
	0x118: UDATA returnValue = 0x000000000000000D (13)
	0x120: UDATA returnValue2 = 0x0000000280618598 (10743809432)
        ...

babsingh · 2024-05-09T15:19:54Z

in case it's related to #14713

I don't see the symptoms from #14713 (comment). J9VMThread->literals is good in this crash. There are similarities in terms of tempSlot = -88 and invokebasic being run.

But, this crash looks like the reverse of #14713 (comment).

In this case, there is a J2I frame at the top, and we are running the invokebasic INL.
While running the invokebasic INL, J9VMThread->jitStackFrameFlags is 0. The fromJIT code-path won't be taken in invokebasic.
Due to the J2I frame, JIT has setup the arguments for invokebasic and the fromJIT code-path needs to be taken in invokebasic. Taking the interpreter path has probably led to the crash.
Also, DDR indicates corrupt data while processing the J2I frame. So, there is probably a bad O-slot in the J2I frame.

fyi @nbhuiyan @jdmpapin @0xdaryl @gacholio

gacholio · 2024-05-10T13:46:45Z

If there is a J2I frame, then the jitStackFrameFlags have already been consumed. Is the invokebasic coming from the interpreted method that's running in the J2I frame?

babsingh · 2024-05-10T14:05:28Z

Is the invokebasic coming from the interpreted method that's running in the J2I frame?

Yes, bytecode 13 in the J2I frame is an invokebasic.

gacholio · 2024-05-10T15:10:07Z

That contradicts your statement:

Due to the J2I frame, JIT has setup the arguments for [invokebasic](https://github.com/eclipse-openj9/openj9/blob/master/runtime/vm/BytecodeInterpreter.hpp#L9246) and the fromJIT code-path needs to be taken in [invokebasic](https://github.com/eclipse-openj9/openj9/blob/master/runtime/vm/BytecodeInterpreter.hpp#L9246). Taking the interpreter path has probably led to the crash.

as we are running in an interpreted method that was invoked by the JIT.

nbhuiyan · 2024-05-10T17:30:33Z

@babsingh

Due to the J2I frame, JIT has setup the arguments for invokebasic

The JIT would not have set up the arguments for invokebasic. The tempSlot value that would have been set by the compiled code for invokebasic calls would be indicating the number of stack slots occupied by the args, and therefore would be a positive number.

babsingh · 2024-05-10T17:47:01Z

I see. So, we are taking the !fromJIT code-path in invokeBasic, which is correct since an interpreted method is being run.

For the interpreted method (invoke_LI_F) in the J2I frame, are the arguments on the stack being populated from the JIT?

<13c95ef00> J2I frame: bp = 0x0000000134810D78, sp = 0x0000000134810CB0, pc = 0x000000013622C451, cp = 0x00000001368194C0, arg0EA = 0x0000000134810D88, flags = 0x0000000010000000
<13c95ef00> 	Method: java/lang/invoke/LambdaForm$NFI/0x000000003622c320.invoke_LI_F(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object; !j9method 0x0000000136819588
<13c95ef00> 	Bytecode index = 13
<13c95ef00> 	Using local mapper
<13c95ef00> 	Locals starting at 0x0000000134810D88 for 0x0000000000000002 slots
<13c95ef00> 		I-Slot: a0[0x0000000134810D88] = 0x00000002805D0080
<13c95ef00> 		I-Slot: a1[0x0000000134810D80] = 0x000000036C3169F8

nbhuiyan · 2024-05-10T17:56:18Z

are the arguments on the stack being populated from the JIT?

Given that the previous frame is a jitted method, yes.

babsingh · 2024-05-10T17:58:21Z

The crash happened on an amac: https://na.artifactory.swg-devops.com/artifactory/sys-rt-generic-local/hyc-runtimes-jenkins.swg-devops.com/Test_openjdk22_j9_extended.system_aarch64_mac_testList_0/10/system_test_output.tar.gz. I am unable to view the native stack in lldb. Can anyone see the line of code in lldb where the crash happened?

knn-k · 2024-05-15T05:06:54Z

OpenJ9 for AArch64 macOS does not support native stack backtrace: Issue #15925.
It is not supported on x86 macOS, either.

JasonFengJ9 · 2024-05-21T18:59:27Z

JDK17 aarch64_mac(macaarch64rt1)

[2024-05-18T13:26:15.887Z] variation: Mode110
[2024-05-18T13:26:15.887Z] JVM_OPTIONS:  -Xjit -Xgcpolicy:gencon -Xnocompressedrefs 

[2024-05-18T13:26:18.068Z] java version "17.0.11" 2024-04-16
[2024-05-18T13:26:18.068Z] IBM Semeru Runtime Certified Edition 17.0.11.0-rc3 (build 17.0.11+9)
[2024-05-18T13:26:18.068Z] Eclipse OpenJ9 VM 17.0.11.0-rc3 (build v0.44.0-release-b0699311c7, JRE 17 Mac OS X aarch64-64-Bit 20240416_560 (JIT enabled, AOT enabled)
[2024-05-18T13:26:18.068Z] OpenJ9   - b0699311c7
[2024-05-18T13:26:18.068Z] OMR      - 254af5a04
[2024-05-18T13:26:18.068Z] JCL      - b522ac89460 based on jdk-17.0.11+9)

[2024-05-18T13:27:22.988Z] HHLT 09:27:18.694 - Completed 20.0%. Number of tests started=61161 (+13542)
[2024-05-18T13:27:37.922Z] HHLT stderr Unhandled exception
[2024-05-18T13:27:37.922Z] HHLT stderr Type=Segmentation error vmState=0x00000000
[2024-05-18T13:27:37.922Z] HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2024-05-18T13:27:37.922Z] HHLT stderr Handler1=000000010113A82C Handler2=0000000100FF4DA4 InaccessibleAddress=0000000000000018
[2024-05-18T13:27:37.922Z] HHLT stderr x0=00000001501BEF00 x1=00000001780467C8 x2=0000000379028408 x3=00000001412B9200
[2024-05-18T13:27:37.922Z] HHLT stderr x4=00000001780467D8 x5=00000001780467C0 x6=00000001780467B8 x7=0000000178046820
[2024-05-18T13:27:37.922Z] HHLT stderr x8=0000000000000000 x9=0000000000000018 x10=00000001780467C0 x11=00000001780467C8
[2024-05-18T13:27:37.922Z] HHLT stderr x12=00000001780467D8 x13=00000001780467B0 x14=0000000101184030 x15=0000000000000010
[2024-05-18T13:27:37.922Z] HHLT stderr x16=0000000101183A9C x17=0000000000000000 x18=0000000178046D00 x19=0000000040000000
[2024-05-18T13:27:37.922Z] HHLT stderr x20=0000000140C68620 x21=00000001780467B0 x22=00000001103B07CC x23=00000001780467C0
[2024-05-18T13:27:37.922Z] HHLT stderr x24=00000001780467D8 x25=00000001780467C8 x26=00000001780467C0 x27=00000001780467B8
[2024-05-18T13:27:37.922Z] HHLT stderr x28=000000010125ED84 x29(FP)=00000001780467A0 x30(LR)=0000000101183A90 x31(SP)=0000000178046500
[2024-05-18T13:27:37.922Z] HHLT stderr PC=00000001011985A0 SP=0000000178046500
[2024-05-18T13:27:37.922Z] HHLT stderr v0 0000000135010ef9 (f: 889261824.000000, d: 2.561350e-314)
[2024-05-18T13:27:37.922Z] HHLT stderr v1 000000014803bb98 (f: 1208204160.000000, d: 2.718928e-314)
[2024-05-18T13:27:37.922Z] HHLT stderr v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v3 c0e0000000000000 (f: 0.000000, d: -3.276800e+04)
[2024-05-18T13:27:37.922Z] HHLT stderr v4 bc6ce0f170000000 (f: 1879048192.000000, d: -1.252413e-17)
[2024-05-18T13:27:37.922Z] HHLT stderr v5 3df73cd68165e929 (f: 2170939648.000000, d: 3.381523e-10)
[2024-05-18T13:27:37.922Z] HHLT stderr v6 bd803323f5b53b23 (f: 4122295040.000000, d: -1.841700e-12)
[2024-05-18T13:27:37.922Z] HHLT stderr v7 000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
[2024-05-18T13:27:37.922Z] HHLT stderr v8 000000003f6b86d4 (f: 1064011456.000000, d: 5.256915e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v9 0000000041000000 (f: 1090519040.000000, d: 5.387880e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2024-05-18T13:27:37.922Z] HHLT stderr v17 3fd56847b4415555 (f: 3024180480.000000, d: 3.344898e-01)
[2024-05-18T13:27:37.922Z] HHLT stderr v18 bf72fd9fa58ecb18 (f: 2777598720.000000, d: -4.636406e-03)
[2024-05-18T13:27:37.922Z] HHLT stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2024-05-18T13:27:37.922Z] HHLT stderr v20 7fefffffffffffff (f: 4294967296.000000, d: 1.797693e+308)
[2024-05-18T13:27:37.922Z] HHLT stderr v21 c1150fb645a1cac1 (f: 1168231168.000000, d: -3.450696e+05)
[2024-05-18T13:27:37.922Z] HHLT stderr v22 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-18T13:27:37.922Z] HHLT stderr v23 c1150fb600000000 (f: 0.000000, d: -3.450695e+05)
[2024-05-18T13:27:37.922Z] HHLT stderr v24 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-18T13:27:37.922Z] HHLT stderr v25 bca8000000000000 (f: 0.000000, d: -1.665335e-16)
[2024-05-18T13:27:37.922Z] HHLT stderr v26 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2024-05-18T13:27:37.922Z] HHLT stderr v27 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-18T13:27:37.922Z] HHLT stderr v28 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v29 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.922Z] HHLT stderr v30 c0e0000000000000 (f: 0.000000, d: -3.276800e+04)
[2024-05-18T13:27:37.922Z] HHLT stderr v31 c0e0000000000000 (f: 0.000000, d: -3.276800e+04)

......

[2024-05-18T13:27:37.923Z] HHLT stderr Unhandled exception
[2024-05-18T13:27:37.923Z] HHLT stderr Type=Segmentation error vmState=0x00000000
[2024-05-18T13:27:37.923Z] HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
[2024-05-18T13:27:37.923Z] HHLT stderr Handler1=000000010113A82C Handler2=0000000100FF4DA4 InaccessibleAddress=0000000000000016
[2024-05-18T13:27:37.923Z] HHLT stderr x0=00000001501CB700 x1=00000001780DE7C8 x2=000000037A040E50 x3=00000001412B9200
[2024-05-18T13:27:37.923Z] HHLT stderr x4=00000001780DE7D8 x5=00000001780DE7C0 x6=00000001780DE7B8 x7=00000001780DE820
[2024-05-18T13:27:37.923Z] HHLT stderr x8=000000015001A620 x9=000000000000000E x10=00000001780DE7C0 x11=00000001780DE7C8
[2024-05-18T13:27:37.923Z] HHLT stderr x12=00000001780DE7D8 x13=00000001780DE7B0 x14=0000000101184030 x15=0000000000000010
[2024-05-18T13:27:37.923Z] HHLT stderr x16=0000000101183A9C x17=0000000000000000 x18=00000001780DED00 x19=0000000040000000
[2024-05-18T13:27:37.923Z] HHLT stderr x20=0000000000000006 x21=00000001780DE7B0 x22=00000001103B07CC x23=00000001780DE7C0
[2024-05-18T13:27:37.923Z] HHLT stderr x24=00000001780DE7D8 x25=00000001780DE7C8 x26=00000001780DE7C0 x27=00000001780DE7B8
[2024-05-18T13:27:37.923Z] HHLT stderr x28=000000010125ED84 x29(FP)=00000001780DE7A0 x30(LR)=0000000101183A90 x31(SP)=00000001780DE500
[2024-05-18T13:27:37.923Z] HHLT stderr PC=0000000101198510 SP=00000001780DE500
[2024-05-18T13:27:37.923Z] HHLT stderr v0 000000014804b759 (f: 1208268672.000000, d: 2.718960e-314)
[2024-05-18T13:27:37.923Z] HHLT stderr v1 000000014803bb98 (f: 1208204160.000000, d: 2.718928e-314)
[2024-05-18T13:27:37.923Z] HHLT stderr v2 0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
[2024-05-18T13:27:37.923Z] HHLT stderr v3 7ff0000000000000 (f: 0.000000, d: inf)
[2024-05-18T13:27:37.923Z] HHLT stderr v4 00000000c0000000 (f: 3221225472.000000, d: 1.591497e-314)
[2024-05-18T13:27:37.923Z] HHLT stderr v5 be9e70430f68aab5 (f: 258517680.000000, d: -4.535693e-07)
[2024-05-18T13:27:37.923Z] HHLT stderr v6 40c0000000000000 (f: 0.000000, d: 8.192000e+03)
[2024-05-18T13:27:37.923Z] HHLT stderr v7 c0c0000000000000 (f: 0.000000, d: -8.192000e+03)
[2024-05-18T13:27:37.923Z] HHLT stderr v8 000000003f6b86d4 (f: 1064011456.000000, d: 5.256915e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v9 0000000041000000 (f: 1090519040.000000, d: 5.387880e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
[2024-05-18T13:27:37.924Z] HHLT stderr v17 3fd54a2ca0742f55 (f: 2691968768.000000, d: 3.326522e-01)
[2024-05-18T13:27:37.924Z] HHLT stderr v18 3f6649a4fe80e0b3 (f: 4269859072.000000, d: 2.720663e-03)
[2024-05-18T13:27:37.924Z] HHLT stderr v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
[2024-05-18T13:27:37.924Z] HHLT stderr v20 7fefffffffffffff (f: 4294967296.000000, d: 1.797693e+308)
[2024-05-18T13:27:37.924Z] HHLT stderr v21 c1150fb645a1cac1 (f: 1168231168.000000, d: -3.450696e+05)
[2024-05-18T13:27:37.924Z] HHLT stderr v22 bfb16872b03ff0cf (f: 2956980480.000000, d: -6.800000e-02)
[2024-05-18T13:27:37.924Z] HHLT stderr v23 c1150fb600000000 (f: 0.000000, d: -3.450695e+05)
[2024-05-18T13:27:37.924Z] HHLT stderr v24 bc90000000000000 (f: 0.000000, d: -5.551115e-17)
[2024-05-18T13:27:37.924Z] HHLT stderr v25 bca8000000000000 (f: 0.000000, d: -1.665335e-16)
[2024-05-18T13:27:37.924Z] HHLT stderr v26 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
[2024-05-18T13:27:37.924Z] HHLT stderr v27 00000000bf800000 (f: 3212836864.000000, d: 1.587352e-314)
[2024-05-18T13:27:37.924Z] HHLT stderr v28 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v29 000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
[2024-05-18T13:27:37.924Z] HHLT stderr v30 7ff0000000000000 (f: 0.000000, d: inf)
[2024-05-18T13:27:37.924Z] HHLT stderr v31 7ff0000000000000 (f: 0.000000, d: inf)
[2024-05-18T13:27:37.924Z] HHLT stderr Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_testList_2/jdkbinary/j2sdk-image/Contents/Home/lib/default/libj9vm29.dylib
[2024-05-18T13:27:37.924Z] HHLT stderr Module_base_address=0000000101118000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
[2024-05-18T13:27:37.924Z] HHLT stderr Symbol_address=0000000101183A9C
[2024-05-18T13:27:37.924Z] HHLT stderr Target=2_90_20240416_560 (Mac OS X 12.6.9)
[2024-05-18T13:27:37.924Z] HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
[2024-05-18T13:27:37.924Z] HHLT stderr ----------- Stack Backtrace -----------
[2024-05-18T13:27:37.924Z] HHLT stderr ---------------------------------------
[2024-05-18T13:27:37.924Z] HHLT stderr JVMDUMP039I Processing dump event "gpf", detail "" at 2024/05/18 09:27:36 - please wait.

[2024-05-18T13:29:23.366Z] HeapHogLoadTest_5m_0_FAILED

50x grinder - passed

babsingh · 2024-05-27T15:40:27Z

are the arguments on the stack being populated from the JIT?

Given that the previous frame is a jitted method, yes.

DDR is reporting corrupt data while iterating the O-slots of the J2I frame. Since the arguments on the stack are being populated by the JIT, @nbhuiyan can you please check if the JIT is supplying the correct arguments for the J2I frame?

pshipton · 2024-08-02T13:03:34Z

https://openj9-jenkins.osuosl.org/job/Test_openjdk22_j9_extended.system_aarch64_mac_Nightly_testList_2/2
HeapHogLoadTest_5m_0

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk22_j9_extended.system_aarch64_mac_Nightly_testList_2/2/system_test_output.tar.gz

22:54:06  HHLT 12:54:04.599 - Completed 73.4%. Number of tests started=585590 (+53010)
22:54:25  HHLT stderr Unhandled exception
22:54:25  HHLT stderr Type=Segmentation error vmState=0x00000000
22:54:25  HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
22:54:25  HHLT stderr Handler1=0000000101309004 Handler2=00000001011B9E74 InaccessibleAddress=0000000000000018
22:54:25  HHLT stderr x0=0000000138094D00 x1=000000016FF9E7C8 x2=000000037C3862B8 x3=0000000130E24000
22:54:25  HHLT stderr x4=000000016FF9E7D8 x5=000000016FF9E7C0 x6=000000016FF9E7B8 x7=0000000000000003
22:54:25  HHLT stderr x8=0000000000000000 x9=0000000000000018 x10=000000016FF9E7C0 x11=000000016FF9E7C8
22:54:25  HHLT stderr x12=000000016FF9E7D8 x13=000000016FF9E7B0 x14=0000000101354D68 x15=0000000000000010
22:54:25  HHLT stderr x16=000000016FF9E820 x17=00000001FEAA5DE0 x18=000000016FF9E101 x19=0000000040000000
22:54:25  HHLT stderr x20=0000000140C15590 x21=000000016FF9E7B0 x22=000000011143435C x23=000000016FF9E7C0
22:54:25  HHLT stderr x24=000000016FF9E7D8 x25=000000016FF9E7C8 x26=000000016FF9E7C0 x27=000000016FF9E7B8
22:54:25  HHLT stderr x28=000000010142ED94 x29(FP)=000000016FF9E7A0 x30(LR)=00000001013547E8 x31(SP)=000000016FF9E2C0
22:54:25  HHLT stderr PC=0000000101369834 SP=000000016FF9E2C0
22:54:25  HHLT stderr v0=000000015098e9e9 (f: 1352198656.000000, d: 2.790071e-314)
22:54:25  HHLT stderr v1=0000000150239198 (f: 1344508288.000000, d: 2.786271e-314)
22:54:25  HHLT stderr v2=3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
22:54:25  HHLT stderr v3=47efffffe0000000 (f: 3758096384.000000, d: 3.402823e+38)
22:54:25  HHLT stderr v4=3f51d8e2cbc49e05 (f: 3418660352.000000, d: 1.089307e-03)
22:54:25  HHLT stderr v5=bf9411c4195bf130 (f: 425455936.000000, d: -1.959902e-02)
22:54:25  HHLT stderr v6=3f51d8e2cbc49e1c (f: 3418660352.000000, d: 1.089307e-03)
22:54:25  HHLT stderr v7=bca71547652b82fe (f: 1697350400.000000, d: -1.601713e-16)
22:54:25  HHLT stderr v8=35e0000000000000 (f: 0.000000, d: 3.421139e-49)
22:54:25  HHLT stderr v9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:25  HHLT stderr v10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:25  HHLT stderr v11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:54:25  HHLT stderr v12=00000000ff7fffff (f: 4286578688.000000, d: 2.117851e-314)
22:54:25  HHLT stderr v13=00000000ff7fffff (f: 4286578688.000000, d: 2.117851e-314)
22:54:25  HHLT stderr v14=000000007ba3d70a (f: 2074334976.000000, d: 1.024858e-314)
22:54:25  HHLT stderr v15=000000007ba3d70a (f: 2074334976.000000, d: 1.024858e-314)
22:54:25  HHLT stderr v16=00000000fba3d70a (f: 4221818624.000000, d: 2.085856e-314)
22:54:25  HHLT stderr v17=00000000fba3d70a (f: 4221818624.000000, d: 2.085856e-314)
22:54:25  HHLT stderr v18=00000000fba3d70a (f: 4221818624.000000, d: 2.085856e-314)
22:54:25  HHLT stderr v19=000000007b03126e (f: 2063798912.000000, d: 1.019652e-314)
22:54:25  HHLT stderr v20=000000007b03126e (f: 2063798912.000000, d: 1.019652e-314)
22:54:25  HHLT stderr v21=00000000fb03126e (f: 4211282432.000000, d: 2.080650e-314)
22:54:25  HHLT stderr v22=00000000fb03126e (f: 4211282432.000000, d: 2.080650e-314)
22:54:25  HHLT stderr v23=00000000fb03126e (f: 4211282432.000000, d: 2.080650e-314)
22:54:25  HHLT stderr v24=00000000000007d0 (f: 2000.000000, d: 9.881313e-321)
22:54:25  HHLT stderr v25=000000007f7fffff (f: 2139095040.000000, d: 1.056853e-314)
22:54:25  HHLT stderr v26=000000007f7fffff (f: 2139095040.000000, d: 1.056853e-314)
22:54:25  HHLT stderr v27=000000007f7fffff (f: 2139095040.000000, d: 1.056853e-314)
22:54:25  HHLT stderr v28=000000007f7fffff (f: 2139095040.000000, d: 1.056853e-314)
22:54:25  HHLT stderr v29=000000007f7fffff (f: 2139095040.000000, d: 1.056853e-314)
22:54:25  HHLT stderr v30=00000002806fba90 (f: 2154806016.000000, d: 5.308607e-314)
22:54:25  HHLT stderr v31=4000000000000000 (f: 0.000000, d: 2.000000e+00)
22:54:25  HHLT stderr Module=/Users/jenkins/workspace/Test_openjdk22_j9_extended.system_aarch64_mac_Nightly_testList_2/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
22:54:25  HHLT stderr Module_base_address=00000001012E4000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
22:54:25  HHLT stderr Symbol_address=00000001013547F4
22:54:25  HHLT stderr Target=2_90_20240802_71 (Mac OS X 12.6.7)
22:54:25  HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
22:54:25  HHLT stderr ----------- Stack Backtrace -----------
22:54:25  HHLT stderr ---------------------------------------

knn-k · 2024-08-05T04:42:50Z

I ran two 30x Grinder jobs, and got one failure:
https://openj9-jenkins.osuosl.org/job/Grinder/3758/
https://openj9-jenkins.osuosl.org/job/Grinder/3759/

HHLT stderr Unhandled exception
HHLT stderr Type=Segmentation error vmState=0x00000000
HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
HHLT stderr Handler1=0000000105025004 Handler2=0000000100E89E74 InaccessibleAddress=0000000000000028
HHLT stderr x0=000000012D9BE100 x1=00000001781767C8 x2=0000000130184000 x3=0000000130184000
HHLT stderr x4=00000001781767D8 x5=00000001781767C0 x6=00000001781767B8 x7=0000000000000002
HHLT stderr x8=0000000000000020 x9=000000037E76A988 x10=00000001781767C0 x11=00000001781767C8
HHLT stderr x12=00000001781767D8 x13=00000001781767B0 x14=0000000105070D68 x15=0000000000000010
HHLT stderr x16=0000000178176820 x17=0000000203B25AB0 x18=0000000178176101 x19=0000000040000000
HHLT stderr x20=0000000000000000 x21=00000001781767B0 x22=000000014841AC54 x23=00000001781767C0
HHLT stderr x24=00000001781767D8 x25=00000001781767C8 x26=00000001781767C0 x27=00000001781767B8
HHLT stderr x28=000000010514AD94 x29(FP)=00000001781767A0 x30(LR)=00000001050707E8 x31(SP)=00000001781762C0
HHLT stderr PC=0000000105085828 SP=00000001781762C0
HHLT stderr v0=000000013e107609 (f: 1041266176.000000, d: 2.636450e-314)
HHLT stderr v1=000000013d9eafb8 (f: 1033809856.000000, d: 2.632766e-314)
HHLT stderr v2=0000000040000000 (f: 1073741824.000000, d: 5.304989e-315)
HHLT stderr v3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v4=000000003f99999a (f: 1067030912.000000, d: 5.271833e-315)
HHLT stderr v5=bf9411c4195bf130 (f: 425455936.000000, d: -1.959902e-02)
HHLT stderr v6=3f51d8e2cbc49e1c (f: 3418660352.000000, d: 1.089307e-03)
HHLT stderr v7=bca71547652b82fe (f: 1697350400.000000, d: -1.601713e-16)
HHLT stderr v8=35e0000000000000 (f: 0.000000, d: 3.421139e-49)
HHLT stderr v9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v12=00000000ff7fffff (f: 4286578688.000000, d: 2.117851e-314)
HHLT stderr v13=00000000ff7fffff (f: 4286578688.000000, d: 2.117851e-314)
HHLT stderr v14=000000007ba3d70a (f: 2074334976.000000, d: 1.024858e-314)
HHLT stderr v15=000000007ba3d70a (f: 2074334976.000000, d: 1.024858e-314)
HHLT stderr v16=bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
HHLT stderr v17=3fd544fdba269155 (f: 3123089664.000000, d: 3.323359e-01)
HHLT stderr v18=3f704f484e37a536 (f: 1312269568.000000, d: 3.981860e-03)
HHLT stderr v19=3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
HHLT stderr v20=000000007b03126e (f: 2063798912.000000, d: 1.019652e-314)
HHLT stderr v21=00000000fb03126e (f: 4211282432.000000, d: 2.080650e-314)
HHLT stderr v22=00000000fb03126e (f: 4211282432.000000, d: 2.080650e-314)
HHLT stderr v23=00000000fb03126e (f: 4211282432.000000, d: 2.080650e-314)
HHLT stderr v24=00000000000007d0 (f: 2000.000000, d: 9.881313e-321)
HHLT stderr v25=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v26=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v27=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v28=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v29=0000000000000000 (f: 0.000000, d: 0.000000e+00)
HHLT stderr v30=0000000280fd6740 (f: 2164090624.000000, d: 5.313194e-314)
HHLT stderr v31=4000000000000000 (f: 0.000000, d: 2.000000e+00)
HHLT stderr Module=/Users/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
HHLT stderr Module_base_address=0000000105000000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
HHLT stderr Symbol_address=00000001050707F4
HHLT stderr Target=2_90_20240802_71 (Mac OS X 11.7.1)
HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
HHLT stderr ----------- Stack Backtrace -----------
HHLT stderr ---------------------------------------

gacholio · 2024-08-05T13:37:44Z

Why is there never a native stack trace?

knn-k · 2024-08-05T23:02:53Z

@gacholio Because no one has implemented it for macOS (neither x86 nor AArch64) yet.
Issue for AArch64 macOS: #15925

pshipton · 2024-10-03T21:25:13Z

jdk24 testing

https://openj9-jenkins.osuosl.org/job/Test_openjdknext_j9_extended.system_aarch64_mac_Nightly_testList_2/66 - mac11-aarch64-4a
HeapHogLoadTest_5m_0 -Xjit -Xgcpolicy:gencon -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdknext_j9_extended.system_aarch64_mac_Nightly_testList_2/66/system_test_output.tar.gz

15:57:10  HHLT 05:57:06.945 - Completed 46.7%. Number of tests started=301016 (+42729)
15:57:10  HHLT stderr Unhandled exception
15:57:10  HHLT stderr Type=Segmentation error vmState=0x00000000
15:57:10  HHLT stderr J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
15:57:10  HHLT stderr Handler1=00000001004D4F98 Handler2=000000010481DD8C InaccessibleAddress=3F36539F00000028
15:57:10  HHLT stderr x0=000000013E1A4D00 x1=000000017088E7C8 x2=000000035A6B96D8 x3=000000014039B400
15:57:10  HHLT stderr x4=000000017088E7D8 x5=000000017088E7C0 x6=000000017088E7B8 x7=000000017088E228
15:57:10  HHLT stderr x8=3F36539F00000020 x9=00000002803FE950 x10=000000017088E7C0 x11=000000017088E7C8
15:57:10  HHLT stderr x12=000000017088E7D8 x13=000000017088E7B0 x14=0000000100520E9C x15=0000000000000010
15:57:10  HHLT stderr x16=000000017088E820 x17=0000000000000001 x18=000000013D810031 x19=0000000040000000
15:57:10  HHLT stderr x20=3F36539F00000000 x21=000000017088E7B0 x22=00000001143B7F3C x23=000000017088E7C0
15:57:10  HHLT stderr x24=000000017088E7D8 x25=000000017088E7C8 x26=000000017088E7C0 x27=000000017088E7B8
15:57:10  HHLT stderr x28=00000001005FAE34 x29(FP)=000000017088E7A0 x30(LR)=00000001005208FC x31(SP)=000000017088E2C0
15:57:10  HHLT stderr PC=0000000100535740 SP=000000017088E2C0
15:57:10  HHLT stderr v0=00000000000000ff (f: 255.000000, d: 1.259867e-321)
15:57:10  HHLT stderr v1=ffffffffffffffff (f: 4294967296.000000, d: nan)
15:57:10  HHLT stderr v2=000000004b955e0c (f: 1268080128.000000, d: 6.265148e-315)
15:57:10  HHLT stderr v3=3fd5d1bdff5809ea (f: 4283959808.000000, d: 3.409266e-01)
15:57:10  HHLT stderr v4=55a868dcfd94108c (f: 4254339328.000000, d: 4.373706e+104)
15:57:10  HHLT stderr v5=000000009a7e2f63 (f: 2591960832.000000, d: 1.280599e-314)
15:57:10  HHLT stderr v6=94446f0194446f01 (f: 2487512832.000000, d: -4.855772e-211)
15:57:10  HHLT stderr v7=000e1781000e1781 (f: 923521.000000, d: 1.959708e-308)
15:57:10  HHLT stderr v8=000003c10000745f (f: 29791.000000, d: 2.039238e-311)
15:57:10  HHLT stderr v9=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v10=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v11=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v12=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v13=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v14=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v15=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v16=bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
15:57:10  HHLT stderr v17=3fd56c16d1555555 (f: 3512030464.000000, d: 3.347222e-01)
15:57:10  HHLT stderr v18=bf76d1ba04992a43 (f: 77146688.000000, d: -5.571105e-03)
15:57:10  HHLT stderr v19=3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
15:57:10  HHLT stderr v20=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v21=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v22=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v23=0000000000000000 (f: 0.000000, d: 0.000000e+00)
15:57:10  HHLT stderr v24=000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
15:57:10  HHLT stderr v25=000000003f800000 (f: 1065353216.000000, d: 5.263544e-315)
15:57:10  HHLT stderr v26=3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
15:57:10  HHLT stderr v27=bf1455a662423200 (f: 1648505344.000000, d: -7.757023e-05)
15:57:10  HHLT stderr v28=000000003e000000 (f: 1040187392.000000, d: 5.139209e-315)
15:57:10  HHLT stderr v29=000000003f400000 (f: 1061158912.000000, d: 5.242822e-315)
15:57:10  HHLT stderr v30=6e616c2e6176616a (f: 1635148160.000000, d: 5.038246e+223)
15:57:10  HHLT stderr v31=b00d00b632052b32 (f: 839199552.000000, d: -3.130911e-77)
15:57:10  HHLT stderr Module=/Users/jenkins/workspace/Test_openjdknext_j9_extended.system_aarch64_mac_Nightly_testList_2/jdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
15:57:10  HHLT stderr Module_base_address=00000001004B0000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
15:57:10  HHLT stderr Symbol_address=0000000100520908
15:57:10  HHLT stderr Target=2_90_20241004_132 (Mac OS X 11.7.1)
15:57:10  HHLT stderr CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
15:57:10  HHLT stderr ----------- Stack Backtrace -----------
15:57:10  HHLT stderr ---------------------------------------

1XMCURTHDINFO  Current thread
3XMTHREADINFO      "load-1" J9VMThread:0x000000013E1A4D00, omrthread_t:0x000000013D906758, java/lang/Thread:0x000000028045B0C8, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800632F8)
3XMTHREADINFO1            (native thread ID:0x26FE4AC4, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000021)
3XMTHREADINFO2            (native stack address range from:0x000000017084C000, to:0x000000017088F000, size:0x43000)
3XMCPUTIME               CPU usage total: 75.836507000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=1175736 (0x11F0B8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/invoke/LambdaForm$NFI/0x000000003d6d90d0.invoke_L3_L(LambdaForm$NFI)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$DMH/0x000000003d8ed220.invokeStatic(LambdaForm$DMH(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/LambdaForm$NamedFunction.invokeWithArguments(LambdaForm.java:1176)
4XESTACKTRACE                at java/lang/invoke/LambdaForm.interpretName(LambdaForm.java:955)
4XESTACKTRACE                at java/lang/invoke/LambdaForm.interpretWithArguments(LambdaForm.java:932)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$LFI/0x00000000251277e0.interpret_L(LambdaForm$LFI)
4XESTACKTRACE                at java/lang/invoke/LambdaForm$MH/0x000000003e16f220.invokeExact_MT(LambdaForm$MH(Compiled Code))
4XESTACKTRACE                at net/adoptopenjdk/test/lang/invoke/BasicTest2.testCollectorWithStaticObjects(BasicTest2.java:376(Compiled Code))

pshipton · 2024-10-03T21:26:51Z

This seems a dup of #19081

JasonFengJ9 · 2024-10-22T21:15:24Z

Closing as a dup of #19081

github-actions · 2024-10-22T21:15:37Z

Issue Number: 19456
Status: Closed
Actual Components: test failure, segfault
Actual Assignees: No one :(
PR Assignees: knn-k

JasonFengJ9 added test failure segfault Issues that describe segfaults / JVM crashes labels May 6, 2024

knn-k mentioned this issue May 8, 2024

Fix the output of floating point values in writeGPInfo() #19463

Merged

knn-k mentioned this issue Aug 6, 2024

HeapHogLoadTest_5m crash vmState=0x00000000 #19081

Open

JasonFengJ9 closed this as completed Oct 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

JasonFengJ9 commented May 6, 2024 •

edited

Loading

pshipton commented May 7, 2024

pshipton commented May 7, 2024

knn-k commented May 8, 2024

knn-k commented May 8, 2024

knn-k commented May 9, 2024

babsingh commented May 9, 2024 •

edited

Loading

babsingh commented May 9, 2024 •

edited

Loading

gacholio commented May 10, 2024

babsingh commented May 10, 2024 •

edited

Loading

gacholio commented May 10, 2024

nbhuiyan commented May 10, 2024

babsingh commented May 10, 2024 •

edited

Loading

nbhuiyan commented May 10, 2024

babsingh commented May 10, 2024 •

edited

Loading

knn-k commented May 15, 2024

JasonFengJ9 commented May 21, 2024 •

edited

Loading

babsingh commented May 27, 2024 •

edited

Loading

pshipton commented Aug 2, 2024

knn-k commented Aug 5, 2024 •

edited

Loading

gacholio commented Aug 5, 2024

knn-k commented Aug 5, 2024

pshipton commented Oct 3, 2024

pshipton commented Oct 3, 2024

JasonFengJ9 commented Oct 22, 2024

github-actions bot commented Oct 22, 2024

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

HeapHogLoadTest_5m_0_FAILED Segmentation error vmState=0x00000000 #19456

Comments

JasonFengJ9 commented May 6, 2024 • edited Loading

Failure link

Optional info

Failure output (captured from console output)

pshipton commented May 7, 2024

pshipton commented May 7, 2024

knn-k commented May 8, 2024

knn-k commented May 8, 2024

knn-k commented May 9, 2024

babsingh commented May 9, 2024 • edited Loading

babsingh commented May 9, 2024 • edited Loading

gacholio commented May 10, 2024

babsingh commented May 10, 2024 • edited Loading

gacholio commented May 10, 2024

nbhuiyan commented May 10, 2024

babsingh commented May 10, 2024 • edited Loading

nbhuiyan commented May 10, 2024

babsingh commented May 10, 2024 • edited Loading

knn-k commented May 15, 2024

JasonFengJ9 commented May 21, 2024 • edited Loading

babsingh commented May 27, 2024 • edited Loading

pshipton commented Aug 2, 2024

knn-k commented Aug 5, 2024 • edited Loading

gacholio commented Aug 5, 2024

knn-k commented Aug 5, 2024

pshipton commented Oct 3, 2024

pshipton commented Oct 3, 2024

JasonFengJ9 commented Oct 22, 2024

github-actions bot commented Oct 22, 2024

JasonFengJ9 commented May 6, 2024 •

edited

Loading

babsingh commented May 9, 2024 •

edited

Loading

babsingh commented May 9, 2024 •

edited

Loading

babsingh commented May 10, 2024 •

edited

Loading

babsingh commented May 10, 2024 •

edited

Loading

babsingh commented May 10, 2024 •

edited

Loading

JasonFengJ9 commented May 21, 2024 •

edited

Loading

babsingh commented May 27, 2024 •

edited

Loading

knn-k commented Aug 5, 2024 •

edited

Loading