Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ConcurrentLoadTest_0 hang #9999

Closed
pshipton opened this issue Jun 24, 2020 · 3 comments
Closed

ConcurrentLoadTest_0 hang #9999

pshipton opened this issue Jun 24, 2020 · 3 comments
Labels
test failure

Comments

@pshipton
Copy link
Member

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.system_x86-64_mac_Nightly/406
ConcurrentLoadTest_0

LT  22:41:20.271 - Completed 100.0%. Number of tests started=540 (+22)
LT  22:41:22.239 - Thread completed. Suite=0 thread=5
LT  22:41:38.774 - Thread completed. Suite=0 thread=2
LT  22:41:40.233 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:41:43.160 - Thread completed. Suite=0 thread=11
LT  22:42:00.274 - Thread completed. Suite=0 thread=0
LT  22:42:00.277 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:42:00.672 - Thread completed. Suite=0 thread=17
LT  22:42:20.271 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:42:40.246 - Completed 100.0%. Number of tests started=540 (+0)
STF 22:42:59.471 - Heartbeat: Process LT  is still running
LT  22:43:00.282 - Completed 100.0%. Number of tests started=540 (+0)
...
STF 22:52:59.023 - Heartbeat: Process LT  is still running
LT  22:53:00.293 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:53:20.223 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:53:40.199 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:54:00.224 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:54:20.204 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:54:40.246 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:55:00.266 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:55:20.203 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:55:40.249 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:56:00.202 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:56:20.257 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:56:40.286 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:57:00.185 - Completed 100.0%. Number of tests started=540 (+0)
LT  22:57:00.722 - **POSSIBLE HANG DETECTED**
STF 22:57:01.127 - **FAILED** Process LT  has hung
STF 22:57:01.127 - Collecting dumps for: LT
@pshipton
Copy link
Member Author

pshipton commented Jun 24, 2020 •
edited
Loading

It's stuck here

3XMTHREADINFO      "load-10" J9VMThread:0x0000000004535600, omrthread_t:0x00007FBFF501F1D8, java/lang/Thread:0x0000000080151F68, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1C, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0xA347C71, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000100)
3XMTHREADINFO2            (native stack address range from:0x0000700000B88000, to:0x0000700000BC8000, size:0x40000)
3XMCPUTIME               CPU usage total: 0.029679000 secs, current category="Application"
3XMTHREADBLOCK     Waiting on: net/adoptopenjdk/test/concurrent/support/jsr166y/ArrayDoubler@0x00000000804DA398 Owned by: <unowned>
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Object.wait(Native Method)
4XESTACKTRACE                at java/lang/Object.wait(Object.java:218)
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.externalAwaitDone(ForkJoinTask.java:334)
5XESTACKTRACE                   (entered lock: net/adoptopenjdk/test/concurrent/support/jsr166y/ArrayDoubler@0x00000000804DA398, entry count: 1)
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.doJoin(ForkJoinTask.java:391(Compiled Code))
4XESTACKTRACE                at java/util/concurrent/ForkJoinTask.join(ForkJoinTask.java:719)
4XESTACKTRACE                at java/util/concurrent/ForkJoinPool.invoke(ForkJoinPool.java:2616)
4XESTACKTRACE                at net/adoptopenjdk/test/concurrent/jsr166y/ForkJoinTaskTest.testAPI(ForkJoinTaskTest.java:111(Compiled Code))

1LKMONPOOLDUMP Monitor Pool Dump (flat & inflated object-monitors):
2LKMONINUSE      sys_mon_t:0x00007FBFF41C8D80 infl_mon_t: 0x00007FBFF41C8E00:
3LKMONOBJECT       net/adoptopenjdk/test/concurrent/support/jsr166y/ArrayDoubler@0x00000000804DA398: <unowned>
3LKNOTIFYQ            Waiting to be notified:
3LKWAITNOTIFY            "load-10" (J9VMThread:0x0000000004535600)

@pshipton
Copy link
Member Author

pshipton commented Jan 20, 2021 •
edited
Loading

https://ci.eclipse.org/openj9/job/Test_openjdk8_j9_extended.system_x86-32_windows_Release_testList_0/8
ConcurrentLoadTest_2

LT  19:11:04.883 - Completed 100.0%. Number of tests started=540 (+0)
LT  19:11:24.898 - Completed 100.0%. Number of tests started=540 (+0)
LT  19:11:44.914 - Completed 100.0%. Number of tests started=540 (+0)
LT  19:12:04.930 - Completed 100.0%. Number of tests started=540 (+0)
LT  19:12:14.227 - **POSSIBLE HANG DETECTED**
STF 19:12:14.556 - **FAILED** Process LT  has hung

There are 3 dump files in the diagnostics, but no javacore files.

Cores are in the same state. I see all the load threads parked on load-14

  thread id: 2712
   registers:
    gs     = 0x0000002b   fs     = 0x00000053   es     = 0x0000002b   ds     = 0x0000002b
    edi    = 0x000006e0   esi    = 0x00000000   ebx    = 0x26fc5014   edx    = 0x00000000
    ecx    = 0x00000000   eax    = 0x00000000   ebp    = 0x27eff8c8   eip    = 0x76f3c0bc
    cs     = 0x00000023   flags  = 0x00000202   esp    = 0x27eff858   ss     = 0x0000002b
   native stack sections:
    0x27eff854 to 0x27f00000 (length 0x7ac)
   native stack frames:
    bp: 0x27eff8c8 pc: 0x76f3c0bc C:\Windows\System32\ntdll.dll::NtWaitForSingleObject+0xc
    bp: 0x27eff8dc pc: 0x765b2c02 C:\Windows\System32\KERNELBASE.dll::WaitForSingleObject+0x12
    bp: 0x76f40c10 pc: 0x71262a70 F:\Users\jenkins\workspace\Test_openjdk8_j9_extended.system_x86-32_windows_Release_testList_0\openjdkbinary\j2sdk-image\jre\bin\default\j9thr29.dll::omrthread_sleep_interruptable+0x100
   properties:
    Affinity=0x00000000000000FF                CreateTime=132555776048983344
    CreateTime_Formatted=2021/01/19 19:53:24   DumpError=0x00000000
    DumpFlags=0x00000000                       ExitStatus=0x00000103
    ExitTime=0                                 KernelTime=4375000
    StartAddress=0x714DBFB4                    UserTime=110625000
    priority=0                                 priorityClass=32
   associated Java thread: 
    name:          load-14
    Thread object: java/lang/Thread @ 0x161c428
    Native info:   !j9vmthread 0x26fb3700  !stack 0x26fb3700
    Daemon:        false
    ID:            40 (0x28)
    Priority:      5
    Thread.State:  <no matching state> 
    JVMTI state:   ALIVE WAITING SLEEPING 
    Java stack frames: 
     bp: 0x26f93e18  method: void java/lang/Thread.sleep(long, int)  (Native Method)
      objects: <no objects in this frame>
     bp: 0x00000000  method: void java/lang/Thread.sleep(long)  source: Thread.java:953
      objects: <no objects in this frame>
     bp: 0x26f93e18  method: void net/adoptopenjdk/test/concurrent/ConcurrentTest.sleep(long)  source: ConcurrentTest.java:118
      objects: <no objects in this frame>
     bp: 0x26f93e50  method: void net/adoptopenjdk/test/concurrent/jsr166y/ForkJoinPoolTestAPI.testAPI()  source: ForkJoinPoolTestAPI.java:144
      objects: 0x1a2e8a0 0x1a2e8b0 0x1a2e8d0 0x1a2e8f0 0x18249c8
     bp: 0x26f93eec  method: Object sun/reflect/NativeMethodAccessorImpl.invoke0(reflect.Method, Object, Object[])  (Native Method)
      objects: <no objects in this frame>
     bp: 0x00000000  method: Object sun/reflect/NativeMethodAccessorImpl.invoke(Object, Object[])  source: NativeMethodAccessorImpl.java:62
      objects: <no objects in this frame>
     bp: 0x00000000  method: Object sun/reflect/DelegatingMethodAccessorImpl.invoke(Object, Object[])  source: DelegatingMethodAccessorImpl.java:43
      objects: <no objects in this frame>
     bp: 0x26f93ed0  method: Object java/lang/reflect/Method.invoke(Object, Object[])  source: Method.java:488
      objects: 0x1a2e910 0x1a2e8a0 0x1a2e920 0x1600808
     bp: 0x26f93eec  method: void junit/framework/TestCase.runTest()  source: TestCase.java:176
      objects: <no objects in this frame>
     bp: 0x26f93f0c  method: void junit/framework/TestCase.runBare()  source: TestCase.java:141
      objects: 0x1a2e8a0
     bp: 0x26f93f1c  method: void junit/framework/TestResult$1.protect()  source: TestResult.java:122
      objects: <no objects in this frame>
     bp: 0x26f93f38  method: void junit/framework/TestResult.runProtected(junit.framework.Test, junit.framework.Protectable)  source: TestResult.java:142
      objects: 0x1a2e978 0x1a2e8a0
     bp: 0x26f93f50  method: void junit/framework/TestResult.run(junit.framework.TestCase)  source: TestResult.java:125
      objects: 0x1a2e978 0x1a2e8a0
     bp: 0x26f93f64  method: void junit/framework/TestCase.run(junit.framework.TestResult)  source: TestCase.java:129
      objects: <no objects in this frame>
     bp: 0x26f93f7c  method: void junit/framework/TestSuite.runTest(junit.framework.Test, junit.framework.TestResult)  source: TestSuite.java:252
      objects: <no objects in this frame>
     bp: 0x26f93f98  method: void junit/framework/TestSuite.run(junit.framework.TestResult)  source: TestSuite.java:247
      objects: 0x1a2e998 0x1a2e978 0x1a2e9a8
     bp: 0x26f93fb0  method: void org/junit/internal/runners/JUnit38ClassRunner.run(org.junit.runner.notification.RunNotifier)  source: JUnit38ClassRunner.java:86
      objects: <no objects in this frame>
     bp: 0x26f93fc8  method: void org/junit/runners/Suite.runChild(org.junit.runner.Runner, org.junit.runner.notification.RunNotifier)  source: Suite.java:128
      objects: <no objects in this frame>
     bp: 0x26f93fe0  method: void org/junit/runners/Suite.runChild(Object, org.junit.runner.notification.RunNotifier)  source: Suite.java:27
      objects: <no objects in this frame>
     bp: 0x26f93ff0  method: void org/junit/runners/ParentRunner$3.run()  source: ParentRunner.java:290
      objects: <no objects in this frame>
     bp: 0x26f94004  method: void org/junit/runners/ParentRunner$1.schedule(Runnable)  source: ParentRunner.java:71
      objects: <no objects in this frame>
     bp: 0x26f94028  method: void org/junit/runners/ParentRunner.runChildren(org.junit.runner.notification.RunNotifier)  source: ParentRunner.java:288
      objects: 0x1a2e9c0 0x1a2e9e0 0x1a2e9f0 0x1a2ea00
     bp: 0x26f9403c  method: void org/junit/runners/ParentRunner.access$000(org.junit.runners.ParentRunner, org.junit.runner.notification.RunNotifier)  source: ParentRunner.java:58
      objects: <no objects in this frame>
     bp: 0x26f9404c  method: void org/junit/runners/ParentRunner$2.evaluate()  source: ParentRunner.java:268
      objects: <no objects in this frame>
     bp: 0x26f94068  method: void org/junit/runners/ParentRunner.run(org.junit.runner.notification.RunNotifier)  source: ParentRunner.java:363
      objects: 0x1a2ea10
     bp: 0x26f940ac  method: org.junit.runner.Result org/junit/runner/JUnitCore.run(org.junit.runner.Runner)  source: JUnitCore.java:137
      objects: 0x1a2ea20 0x1a2ea30 0x1a2ea50
     bp: 0x00000000  method: org.junit.runner.Result org/junit/runner/JUnitCore.run(org.junit.runner.Request)  source: JUnitCore.java:115
      objects: <no objects in this frame>
     bp: 0x26f94140  method: net.adoptopenjdk.loadTest.adaptors.AdaptorInterface$ResultStatus net/adoptopenjdk/loadTest/adaptors/JUnitAdaptor.executeTest()  source: JUnitAdaptor.java:130
      objects: 0x1589678 0x1a2e9c0 0x1a2ea60
     bp: 0x26f94180  method: void net/adoptopenjdk/loadTest/LoadTestRunner$2.run()  source: LoadTestRunner.java:182
      objects: 0x161c4e0 0x161c508 0x161c4c0 0x1589678
     bp: 0x26f941b4  method: void java/util/concurrent/ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)  source: ThreadPoolExecutor.java:1149
      objects: 0x14dd448 0x161c520 0x161c428 0x161c4e0
     bp: 0x26f941c4  method: void java/util/concurrent/ThreadPoolExecutor$Worker.run()  source: ThreadPoolExecutor.java:624
      objects: <no objects in this frame>
     bp: 0x26f941d4  method: void java/lang/Thread.run()  source: Thread.java:823
      objects: <no objects in this frame>

@pshipton pshipton mentioned this issue Feb 3, 2021
Open
@pshipton pshipton mentioned this issue Apr 19, 2021
Open
@pshipton
Copy link
Member Author

Close this as a dup of #11868

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
test failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pshipton