[Bug] start skywalking deadlock #12708

yuchao86 · 2024-10-17T13:43:18Z

yuchao86
Oct 17, 2024

Search before asking

I had searched in the issues and found no similar issues.

Apache SkyWalking Component

OAP server (apache/skywalking)

What happened

2024-10-17 21:09:09
Full thread dump Java HotSpot(TM) 64-Bit Server VM (21.0.2+13-LTS-58 mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f73c4000c40, length=23, elements={
0x00007f786801e760, 0x00007f7868146330, 0x00007f7868147df0, 0x00007f7868149d60,
0x00007f786814b810, 0x00007f786814d220, 0x00007f786814f1d0, 0x00007f7868150d00,
0x00007f786817da10, 0x00007f7869a98720, 0x00007f7869d2c170, 0x00007f7869d2eb80,
0x00007f7869d30820, 0x00007f7869d31e90, 0x00007f7869d33ba0, 0x00007f7869d35710,
0x00007f7869d377e0, 0x00007f7869d39520, 0x00007f7869d3aff0, 0x00007f7869d3cc00,
0x00007f7869d3e8f0, 0x00007f7869d4a5f0, 0x00007f73c4000ff0
}

"main" #1 [61967] prio=5 os_prio=0 cpu=1242.09ms elapsed=1650.87s tid=0x00007f786801e760 nid=61967 waiting for monitor entry  [0x00007f786d44b000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:651)
	- waiting to lock <0x000004000acc40f0> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$TypeExtractor$RecordComponentExtractor.visitEnd(TypePool.java:8946)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.readRecordComponent(ClassReader.java:1053)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:732)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.parse(TypePool.java:880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:864)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:944)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1042)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1111)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getModifiers(TypeDescription.java:8535)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.SWDescriptionStrategy$SWTypeDescriptionWrapper.getModifiers(SWDescriptionStrategy.java:332)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:60)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:27)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1971)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12385)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12094)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12872)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12804)
	at java.security.AccessController.executePrivileged([email protected]/AccessController.java:778)
	at java.security.AccessController.doPrivileged([email protected]/AccessController.java:400)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12328)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at sun.instrument.TransformerManager.transform([email protected]/TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform([email protected]/InstrumentationImpl.java:610)
	at jdk.internal.misc.ThreadTracker.begin([email protected]/ThreadTracker.java:70)
	at java.util.jar.JarFile.beginInit([email protected]/JarFile.java:1054)
	at java.util.jar.JarFile.ensureInitialization([email protected]/JarFile.java:1068)
	- locked <0x000004000018d5c8> (a java.util.jar.JarFile)
	at java.util.jar.JavaUtilJarAccessImpl.ensureInitialization([email protected]/JavaUtilJarAccessImpl.java:42)
	at jdk.internal.loader.URLClassPath$JarLoader$2.getManifest([email protected]/URLClassPath.java:852)
	at jdk.internal.loader.BuiltinClassLoader.defineClass([email protected]/BuiltinClassLoader.java:848)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull([email protected]/BuiltinClassLoader.java:760)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:681)
	- locked <0x0000040009d7a358> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at java.lang.Class.forName0([email protected]/Native Method)
	at java.lang.Class.forName([email protected]/Class.java:534)
	at java.lang.Class.forName([email protected]/Class.java:513)
	at sun.launcher.LauncherHelper.loadMainClass([email protected]/LauncherHelper.java:797)
	at sun.launcher.LauncherHelper.checkAndLoadMain([email protected]/LauncherHelper.java:692)

"Reference Handler" #14 [62003] daemon prio=10 os_prio=0 cpu=0.11ms elapsed=1650.64s tid=0x00007f7868146330 nid=62003 waiting on condition  [0x00007f786c100000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method)
	at java.lang.ref.Reference.processPendingReferences([email protected]/Reference.java:246)
	at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Reference.java:208)

"Finalizer" #15 [62004] daemon prio=8 os_prio=0 cpu=0.10ms elapsed=1650.64s tid=0x00007f7868147df0 nid=62004 in Object.wait()  [0x00007f786c0bf000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait0([email protected]/Native Method)
	- waiting on <0x00000400000117d0> (a java.lang.ref.NativeReferenceQueue$Lock)
	at java.lang.Object.wait([email protected]/Object.java:366)
	at java.lang.Object.wait([email protected]/Object.java:339)
	at java.lang.ref.NativeReferenceQueue.await([email protected]/NativeReferenceQueue.java:48)
	at java.lang.ref.ReferenceQueue.remove0([email protected]/ReferenceQueue.java:158)
	at java.lang.ref.NativeReferenceQueue.remove([email protected]/NativeReferenceQueue.java:89)
	- locked <0x00000400000117d0> (a java.lang.ref.NativeReferenceQueue$Lock)
	at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Finalizer.java:173)

"Signal Dispatcher" #16 [62005] daemon prio=9 os_prio=0 cpu=0.20ms elapsed=1650.64s tid=0x00007f7868149d60 nid=62005 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #17 [62006] daemon prio=9 os_prio=0 cpu=0.18ms elapsed=1650.64s tid=0x00007f786814b810 nid=62006 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #18 [62007] daemon prio=9 os_prio=0 cpu=57.66ms elapsed=1650.64s tid=0x00007f786814d220 nid=62007 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #19 [62008] daemon prio=9 os_prio=0 cpu=967.21ms elapsed=1650.64s tid=0x00007f786814f1d0 nid=62008 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #20 [62009] daemon prio=9 os_prio=0 cpu=660.49ms elapsed=1650.64s tid=0x00007f7868150d00 nid=62009 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Common-Cleaner" #21 [62010] daemon prio=8 os_prio=0 cpu=1.26ms elapsed=1650.63s tid=0x00007f786817da10 nid=62010 waiting on condition  [0x00007f78507d5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x00000400000ab040> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:1847)
	at java.lang.ref.ReferenceQueue.await([email protected]/ReferenceQueue.java:71)
	at java.lang.ref.ReferenceQueue.remove0([email protected]/ReferenceQueue.java:143)
	at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:218)
	at jdk.internal.ref.CleanerImpl.run([email protected]/CleanerImpl.java:140)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)
	at jdk.internal.misc.InnocuousThread.run([email protected]/InnocuousThread.java:186)

"SkywalkingAgent-LogFileWriter" #22 [62013] prio=5 os_prio=0 cpu=37.73ms elapsed=1649.42s tid=0x00007f7869a98720 nid=62013 waiting on condition  [0x00007f7850756000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep0([email protected]/Native Method)
	at java.lang.Thread.sleep([email protected]/Thread.java:558)
	at java.util.concurrent.TimeUnit.sleep([email protected]/TimeUnit.java:446)
	at org.apache.skywalking.apm.agent.core.logging.core.FileWriter$1.run(FileWriter.java:89)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"DataCarrier.DEFAULT.Consumer.0.Thread" #24 [62014] daemon prio=5 os_prio=0 cpu=613.79ms elapsed=1648.95s tid=0x00007f7869d2c170 nid=62014 waiting on condition  [0x00007f7850707000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep0([email protected]/Native Method)
	at java.lang.Thread.sleep([email protected]/Thread.java:509)
	at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:55)

"SkywalkingAgent-5-JVMService-produce-0" #25 [62015] daemon prio=5 os_prio=0 cpu=31.59ms elapsed=1648.95s tid=0x00007f7869d2eb80 nid=62015 waiting for monitor entry  [0x00007f78506c1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.util.zip.ZipFile.getEntry([email protected]/ZipFile.java:337)
	- waiting to lock <0x000004000018d5c8> (a java.util.jar.JarFile)
	at java.util.jar.JarFile.getEntry([email protected]/JarFile.java:516)
	at java.util.jar.JarFile.getJarEntry([email protected]/JarFile.java:471)
	at jdk.internal.loader.URLClassPath$JarLoader.getResource([email protected]/URLClassPath.java:923)
	at jdk.internal.loader.URLClassPath.getResource([email protected]/URLClassPath.java:316)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull([email protected]/BuiltinClassLoader.java:757)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:681)
	- locked <0x000004000acc40f0> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$TypeExtractor$RecordComponentExtractor.visitEnd(TypePool.java:8946)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.readRecordComponent(ClassReader.java:1053)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:732)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.parse(TypePool.java:880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:864)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:944)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1042)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1111)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getModifiers(TypeDescription.java:8535)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.SWDescriptionStrategy$SWTypeDescriptionWrapper.getModifiers(SWDescriptionStrategy.java:332)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:60)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:27)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1971)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12385)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12094)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12872)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12804)
	at java.security.AccessController.executePrivileged([email protected]/AccessController.java:778)
	at java.security.AccessController.doPrivileged([email protected]/AccessController.java:400)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12328)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at sun.instrument.TransformerManager.transform([email protected]/TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform([email protected]/InstrumentationImpl.java:610)
	at java.nio.DirectByteBuffer.<init>([email protected]/DirectByteBuffer.java:128)
	at java.nio.ByteBuffer.allocateDirect([email protected]/ByteBuffer.java:360)
	at sun.nio.ch.Util.getTemporaryDirectBuffer([email protected]/Util.java:242)
	at sun.nio.ch.IOUtil.read([email protected]/IOUtil.java:303)
	at sun.nio.ch.IOUtil.read([email protected]/IOUtil.java:283)
	at sun.nio.ch.FileChannelImpl.read([email protected]/FileChannelImpl.java:234)
	- locked <0x0000040004b322e8> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read([email protected]/ChannelInputStream.java:74)
	at sun.nio.ch.ChannelInputStream.read([email protected]/ChannelInputStream.java:103)
	- locked <0x000004000ac18f68> (a sun.nio.ch.ChannelInputStream)
	at sun.nio.cs.StreamDecoder.readBytes([email protected]/StreamDecoder.java:329)
	at sun.nio.cs.StreamDecoder.implRead([email protected]/StreamDecoder.java:372)
	at sun.nio.cs.StreamDecoder.lockedRead([email protected]/StreamDecoder.java:215)
	at sun.nio.cs.StreamDecoder.read([email protected]/StreamDecoder.java:169)
	at java.io.InputStreamReader.read([email protected]/InputStreamReader.java:188)
	at java.io.BufferedReader.fill([email protected]/BufferedReader.java:160)
	at java.io.BufferedReader.implReadLine([email protected]/BufferedReader.java:370)
	at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:347)
	at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:436)
	at java.nio.file.Files.readAllLines([email protected]/Files.java:3395)
	at java.nio.file.Files.readAllLines([email protected]/Files.java:3433)
	at jdk.internal.platform.CgroupUtil.lambda$readAllLinesPrivileged$2([email protected]/CgroupUtil.java:83)
	at jdk.internal.platform.CgroupUtil$$Lambda/0x00007f73e01fd2c8.run([email protected]/Unknown Source)
	at java.security.AccessController.executePrivileged([email protected]/AccessController.java:809)
	at java.security.AccessController.doPrivileged([email protected]/AccessController.java:571)
	at jdk.internal.platform.CgroupUtil.readAllLinesPrivileged([email protected]/CgroupUtil.java:84)
	at jdk.internal.platform.CgroupSubsystemFactory.determineType([email protected]/CgroupSubsystemFactory.java:143)
	at jdk.internal.platform.CgroupSubsystemFactory.create([email protected]/CgroupSubsystemFactory.java:85)
	at jdk.internal.platform.CgroupMetrics.getInstance([email protected]/CgroupMetrics.java:193)
	at jdk.internal.platform.SystemMetrics.instance([email protected]/SystemMetrics.java:29)
	at jdk.internal.platform.Metrics.systemMetrics([email protected]/Metrics.java:58)
	at jdk.internal.platform.Container.metrics([email protected]/Container.java:43)
	at com.sun.management.internal.OperatingSystemImpl.<init>([email protected]/OperatingSystemImpl.java:175)
	at com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean([email protected]/PlatformMBeanProviderImpl.java:280)
	- locked <0x000004000ae97408> (a java.lang.Class for com.sun.management.internal.PlatformMBeanProviderImpl)
	at com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap([email protected]/PlatformMBeanProviderImpl.java:199)
	at sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans([email protected]/PlatformMBeanProvider.java:195)
	at java.lang.management.ManagementFactory.getPlatformMXBean([email protected]/ManagementFactory.java:691)
	at java.lang.management.ManagementFactory.getOperatingSystemMXBean([email protected]/ManagementFactory.java:391)
	at org.apache.skywalking.apm.agent.core.os.ProcessorUtil.getNumberOfProcessors(ProcessorUtil.java:25)
	at org.apache.skywalking.apm.agent.core.jvm.cpu.CPUProvider.<init>(CPUProvider.java:31)
	at org.apache.skywalking.apm.agent.core.jvm.cpu.CPUProvider.<clinit>(CPUProvider.java:27)
	at org.apache.skywalking.apm.agent.core.jvm.JVMService.run(JVMService.java:101)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:572)
	at java.util.concurrent.FutureTask.runAndReset([email protected]/FutureTask.java:358)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:305)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-6-JVMService-consume-0" #26 [62016] daemon prio=5 os_prio=0 cpu=59.88ms elapsed=1648.95s tid=0x00007f7869d30820 nid=62016 waiting on condition  [0x00007f7850685000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040009cc2e38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-7-ServiceManagementClient-0" #27 [62017] daemon prio=5 os_prio=0 cpu=1.47ms elapsed=1648.95s tid=0x00007f7869d31e90 nid=62017 waiting on condition  [0x00007f7850644000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040009cc5f18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-1-CommandService-0" #28 [62018] daemon prio=5 os_prio=0 cpu=1.10ms elapsed=1648.94s tid=0x00007f7869d33ba0 nid=62018 waiting on condition  [0x00007f7850603000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040004db37f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:371)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block([email protected]/AbstractQueuedSynchronizer.java:519)
	at java.util.concurrent.ForkJoinPool.unmanagedBlock([email protected]/ForkJoinPool.java:3780)
	at java.util.concurrent.ForkJoinPool.managedBlock([email protected]/ForkJoinPool.java:3725)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:1707)
	at java.util.concurrent.LinkedBlockingQueue.take([email protected]/LinkedBlockingQueue.java:435)
	at org.apache.skywalking.apm.agent.core.commands.CommandService.run(CommandService.java:66)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:572)
	at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-8-ProfileGetTaskService-0" #29 [62019] daemon prio=5 os_prio=0 cpu=1.58ms elapsed=1648.94s tid=0x00007f7869d35710 nid=62019 waiting on condition  [0x00007f78505c2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040009cce340> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-9-ProfileSendSnapshotService-0" #30 [62020] daemon prio=5 os_prio=0 cpu=135.37ms elapsed=1648.94s tid=0x00007f7869d377e0 nid=62020 waiting on condition  [0x00007f7850581000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040009cd16f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-10-MeterReportService-0" #31 [62021] daemon prio=5 os_prio=0 cpu=1.40ms elapsed=1648.94s tid=0x00007f7869d39520 nid=62021 waiting on condition  [0x00007f7850540000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040009cd7b20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"DataCarrier.gRPC-log.Consumer.0.Thread" #32 [62022] daemon prio=5 os_prio=0 cpu=606.55ms elapsed=1648.94s tid=0x00007f7869d3aff0 nid=62022 waiting on condition  [0x00007f78504ff000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep0([email protected]/Native Method)
	at java.lang.Thread.sleep([email protected]/Thread.java:509)
	at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:55)

"SkywalkingAgent-11-ConfigurationDiscoveryService-0" #33 [62023] daemon prio=5 os_prio=0 cpu=1.92ms elapsed=1648.94s tid=0x00007f7869d3cc00 nid=62023 waiting on condition  [0x00007f78504be000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000040009cdea50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos([email protected]/AbstractQueuedSynchronizer.java:1758)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take([email protected]/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1070)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"SkywalkingAgent-12-GRPCChannelManager-0" #34 [62024] daemon prio=5 os_prio=0 cpu=11.58ms elapsed=1648.94s tid=0x00007f7869d3e8f0 nid=62024 waiting for monitor entry  [0x00007f785047c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.util.zip.ZipFile.getEntry([email protected]/ZipFile.java:337)
	- waiting to lock <0x000004000018d5c8> (a java.util.jar.JarFile)
	at java.util.jar.JarFile.getEntry([email protected]/JarFile.java:516)
	at java.util.jar.JarFile.getJarEntry([email protected]/JarFile.java:471)
	at jdk.internal.loader.URLClassPath$JarLoader.getResource([email protected]/URLClassPath.java:923)
	at jdk.internal.loader.URLClassPath.getResource([email protected]/URLClassPath.java:316)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull([email protected]/BuiltinClassLoader.java:757)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:681)
	- locked <0x0000040004971250> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at org.apache.skywalking.apm.dependencies.io.grpc.netty.NettyChannelBuilder.<clinit>(NettyChannelBuilder.java:84)
	at org.apache.skywalking.apm.agent.core.remote.GRPCChannel.<init>(GRPCChannel.java:40)
	at org.apache.skywalking.apm.agent.core.remote.GRPCChannel.<init>(GRPCChannel.java:31)
	at org.apache.skywalking.apm.agent.core.remote.GRPCChannel$Builder.build(GRPCChannel.java:105)
	at org.apache.skywalking.apm.agent.core.remote.GRPCChannelManager.run(GRPCChannelManager.java:148)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:572)
	at java.util.concurrent.FutureTask.runAndReset([email protected]/FutureTask.java:358)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:305)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

"Notification Thread" #36 [62025] daemon prio=9 os_prio=0 cpu=0.05ms elapsed=1648.94s tid=0x00007f7869d4a5f0 nid=62025 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #37 [63756] daemon prio=9 os_prio=0 cpu=0.21ms elapsed=0.20s tid=0x00007f73c4000ff0 nid=63756 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"VM Thread" os_prio=0 cpu=19.94ms elapsed=1650.64s tid=0x00007f78681362a0 nid=62002 runnable  

"ZDirector" os_prio=0 cpu=986.37ms elapsed=1650.65s tid=0x00007f78680efca0 nid=61997 runnable  

"ZDriverMajor" os_prio=0 cpu=4.88ms elapsed=1650.65s tid=0x00007f78680ee6f0 nid=61996 runnable  

"ZDriverMinor" os_prio=0 cpu=0.06ms elapsed=1650.65s tid=0x00007f78680ed140 nid=61995 runnable  

"ZStat" os_prio=0 cpu=33.44ms elapsed=1650.65s tid=0x00007f78680f0f20 nid=61998 runnable  

"ZUnmapper" os_prio=0 cpu=0.04ms elapsed=1650.87s tid=0x00007f78680e2710 nid=61970 runnable  

"ZUncommitter" os_prio=0 cpu=0.03ms elapsed=1650.86s tid=0x00007f78680e44a0 nid=61971 runnable  

"ZWorkerYoung#0" os_prio=0 cpu=48.76ms elapsed=1650.86s tid=0x00007f78680ec050 nid=61973 runnable  

"ZWorkerOld#0" os_prio=0 cpu=14.63ms elapsed=1650.86s tid=0x00007f78680e5c10 nid=61972 runnable  

"RuntimeWorker#0" os_prio=0 cpu=0.08ms elapsed=1650.65s tid=0x00007f78680f21c0 nid=61999 runnable  

"RuntimeWorker#1" os_prio=0 cpu=0.02ms elapsed=1650.65s tid=0x00007f78680f33e0 nid=62000 runnable  

"VM Periodic Task Thread" os_prio=0 cpu=250.46ms elapsed=1650.64s tid=0x00007f78680ff0f0 nid=62001 waiting on condition  

JNI global refs: 19, weak refs: 0


Found one Java-level deadlock:
=============================
"main":
  waiting to lock monitor 0x00007f7869cc9800 (object 0x000004000acc40f0, a java.lang.Object),
  which is held by "SkywalkingAgent-5-JVMService-produce-0"

"SkywalkingAgent-5-JVMService-produce-0":
  waiting to lock monitor 0x00007f73a40254d0 (object 0x000004000018d5c8, a java.util.jar.JarFile),
  which is held by "main"

Java stack information for the threads listed above:
===================================================
"main":
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:651)
	- waiting to lock <0x000004000acc40f0> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$TypeExtractor$RecordComponentExtractor.visitEnd(TypePool.java:8946)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.readRecordComponent(ClassReader.java:1053)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:732)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.parse(TypePool.java:880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:864)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:944)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1042)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1111)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getModifiers(TypeDescription.java:8535)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.SWDescriptionStrategy$SWTypeDescriptionWrapper.getModifiers(SWDescriptionStrategy.java:332)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:60)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:27)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1971)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12385)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12094)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12872)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12804)
	at java.security.AccessController.executePrivileged([email protected]/AccessController.java:778)
	at java.security.AccessController.doPrivileged([email protected]/AccessController.java:400)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12328)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at sun.instrument.TransformerManager.transform([email protected]/TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform([email protected]/InstrumentationImpl.java:610)
	at jdk.internal.misc.ThreadTracker.begin([email protected]/ThreadTracker.java:70)
	at java.util.jar.JarFile.beginInit([email protected]/JarFile.java:1054)
	at java.util.jar.JarFile.ensureInitialization([email protected]/JarFile.java:1068)
	- locked <0x000004000018d5c8> (a java.util.jar.JarFile)
	at java.util.jar.JavaUtilJarAccessImpl.ensureInitialization([email protected]/JavaUtilJarAccessImpl.java:42)
	at jdk.internal.loader.URLClassPath$JarLoader$2.getManifest([email protected]/URLClassPath.java:852)
	at jdk.internal.loader.BuiltinClassLoader.defineClass([email protected]/BuiltinClassLoader.java:848)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull([email protected]/BuiltinClassLoader.java:760)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:681)
	- locked <0x0000040009d7a358> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at java.lang.Class.forName0([email protected]/Native Method)
	at java.lang.Class.forName([email protected]/Class.java:534)
	at java.lang.Class.forName([email protected]/Class.java:513)
	at sun.launcher.LauncherHelper.loadMainClass([email protected]/LauncherHelper.java:797)
	at sun.launcher.LauncherHelper.checkAndLoadMain([email protected]/LauncherHelper.java:692)
"SkywalkingAgent-5-JVMService-produce-0":
	at java.util.zip.ZipFile.getEntry([email protected]/ZipFile.java:337)
	- waiting to lock <0x000004000018d5c8> (a java.util.jar.JarFile)
	at java.util.jar.JarFile.getEntry([email protected]/JarFile.java:516)
	at java.util.jar.JarFile.getJarEntry([email protected]/JarFile.java:471)
	at jdk.internal.loader.URLClassPath$JarLoader.getResource([email protected]/URLClassPath.java:923)
	at jdk.internal.loader.URLClassPath.getResource([email protected]/URLClassPath.java:316)
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull([email protected]/BuiltinClassLoader.java:757)
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:681)
	- locked <0x000004000acc40f0> (a java.lang.Object)
	at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass([email protected]/ClassLoaders.java:188)
	at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$TypeExtractor$RecordComponentExtractor.visitEnd(TypePool.java:8946)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.readRecordComponent(ClassReader.java:1053)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:732)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.jar.asm.ClassReader.accept(ClassReader.java:425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.parse(TypePool.java:880)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default.doDescribe(TypePool.java:864)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.access$001(TypePool.java:944)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution.doResolve(TypePool.java:1042)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$WithLazyResolution$LazyTypeDescription.delegate(TypePool.java:1111)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$AbstractBase$OfSimpleType$WithDelegation.getModifiers(TypeDescription.java:8535)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.SWDescriptionStrategy$SWTypeDescriptionWrapper.getModifiers(SWDescriptionStrategy.java:332)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:60)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ModifierMatcher.doMatch(ModifierMatcher.java:27)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1971)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:12425)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12385)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1800(AgentBuilder.java:12094)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12872)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$Java9CapableVmDispatcher.run(AgentBuilder.java:12804)
	at java.security.AccessController.executePrivileged([email protected]/AccessController.java:778)
	at java.security.AccessController.doPrivileged([email protected]/AccessController.java:400)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:12328)
	at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$ByteBuddy$ModuleSupport.transform(Unknown Source)
	at sun.instrument.TransformerManager.transform([email protected]/TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform([email protected]/InstrumentationImpl.java:610)
	at java.nio.DirectByteBuffer.<init>([email protected]/DirectByteBuffer.java:128)
	at java.nio.ByteBuffer.allocateDirect([email protected]/ByteBuffer.java:360)
	at sun.nio.ch.Util.getTemporaryDirectBuffer([email protected]/Util.java:242)
	at sun.nio.ch.IOUtil.read([email protected]/IOUtil.java:303)
	at sun.nio.ch.IOUtil.read([email protected]/IOUtil.java:283)
	at sun.nio.ch.FileChannelImpl.read([email protected]/FileChannelImpl.java:234)
	- locked <0x0000040004b322e8> (a java.lang.Object)
	at sun.nio.ch.ChannelInputStream.read([email protected]/ChannelInputStream.java:74)
	at sun.nio.ch.ChannelInputStream.read([email protected]/ChannelInputStream.java:103)
	- locked <0x000004000ac18f68> (a sun.nio.ch.ChannelInputStream)
	at sun.nio.cs.StreamDecoder.readBytes([email protected]/StreamDecoder.java:329)
	at sun.nio.cs.StreamDecoder.implRead([email protected]/StreamDecoder.java:372)
	at sun.nio.cs.StreamDecoder.lockedRead([email protected]/StreamDecoder.java:215)
	at sun.nio.cs.StreamDecoder.read([email protected]/StreamDecoder.java:169)
	at java.io.InputStreamReader.read([email protected]/InputStreamReader.java:188)
	at java.io.BufferedReader.fill([email protected]/BufferedReader.java:160)
	at java.io.BufferedReader.implReadLine([email protected]/BufferedReader.java:370)
	at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:347)
	at java.io.BufferedReader.readLine([email protected]/BufferedReader.java:436)
	at java.nio.file.Files.readAllLines([email protected]/Files.java:3395)
	at java.nio.file.Files.readAllLines([email protected]/Files.java:3433)
	at jdk.internal.platform.CgroupUtil.lambda$readAllLinesPrivileged$2([email protected]/CgroupUtil.java:83)
	at jdk.internal.platform.CgroupUtil$$Lambda/0x00007f73e01fd2c8.run([email protected]/Unknown Source)
	at java.security.AccessController.executePrivileged([email protected]/AccessController.java:809)
	at java.security.AccessController.doPrivileged([email protected]/AccessController.java:571)
	at jdk.internal.platform.CgroupUtil.readAllLinesPrivileged([email protected]/CgroupUtil.java:84)
	at jdk.internal.platform.CgroupSubsystemFactory.determineType([email protected]/CgroupSubsystemFactory.java:143)
	at jdk.internal.platform.CgroupSubsystemFactory.create([email protected]/CgroupSubsystemFactory.java:85)
	at jdk.internal.platform.CgroupMetrics.getInstance([email protected]/CgroupMetrics.java:193)
	at jdk.internal.platform.SystemMetrics.instance([email protected]/SystemMetrics.java:29)
	at jdk.internal.platform.Metrics.systemMetrics([email protected]/Metrics.java:58)
	at jdk.internal.platform.Container.metrics([email protected]/Container.java:43)
	at com.sun.management.internal.OperatingSystemImpl.<init>([email protected]/OperatingSystemImpl.java:175)
	at com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean([email protected]/PlatformMBeanProviderImpl.java:280)
	- locked <0x000004000ae97408> (a java.lang.Class for com.sun.management.internal.PlatformMBeanProviderImpl)
	at com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap([email protected]/PlatformMBeanProviderImpl.java:199)
	at sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans([email protected]/PlatformMBeanProvider.java:195)
	at java.lang.management.ManagementFactory.getPlatformMXBean([email protected]/ManagementFactory.java:691)
	at java.lang.management.ManagementFactory.getOperatingSystemMXBean([email protected]/ManagementFactory.java:391)
	at org.apache.skywalking.apm.agent.core.os.ProcessorUtil.getNumberOfProcessors(ProcessorUtil.java:25)
	at org.apache.skywalking.apm.agent.core.jvm.cpu.CPUProvider.<init>(CPUProvider.java:31)
	at org.apache.skywalking.apm.agent.core.jvm.cpu.CPUProvider.<clinit>(CPUProvider.java:27)
	at org.apache.skywalking.apm.agent.core.jvm.JVMService.run(JVMService.java:101)
	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:572)
	at java.util.concurrent.FutureTask.runAndReset([email protected]/FutureTask.java:358)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run([email protected]/ScheduledThreadPoolExecutor.java:305)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith([email protected]/Thread.java:1596)
	at java.lang.Thread.run([email protected]/Thread.java:1583)

Found 1 deadlock.

What you expected to happen

start the skywalking

Found 0 deadlock.

How to reproduce

start the skywalking

Anything else

none

Are you willing to submit a pull request to fix on your own?

Yes I am willing to submit a pull request on my own!

Code of Conduct

I agree to follow this project's Code of Conduct

wu-sheng · 2024-10-17T14:09:02Z

wu-sheng
Oct 17, 2024
Collaborator

Which version of Java agent are you using?

I am not sure why a jar file - locked <0x000004000018d5c8> (a java.util.jar.JarFile) could be locked.

7 replies

ddavidzhang Oct 18, 2024

Did this happen every time? We tested JDK21, there is no such error.

not every time. about 1/10

wu-sheng Oct 18, 2024
Collaborator

This looks like a concurrency class loading conflict. But I am not sure how this happens.

ddavidzhang Oct 21, 2024

I am not sure if latest version has fixed this bug. We met this bug very often these pasted days.

ddavidzhang Oct 21, 2024

happens very often when using jdk21. dont happen ever on jdk8

wu-sheng Oct 21, 2024
Collaborator

I am not sure if latest version has fixed this bug. We met this bug very often these pasted days.

We don't fix this as it doesn't happen in the CI process. Personally, I don't use agent, my daily job is about service mesh observability.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] start skywalking deadlock #12708

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Bug] start skywalking deadlock #12708

yuchao86 Oct 17, 2024

Search before asking

Apache SkyWalking Component

What happened

What you expected to happen

How to reproduce

Anything else

Are you willing to submit a pull request to fix on your own?

Code of Conduct

Replies: 1 comment · 7 replies

wu-sheng Oct 17, 2024 Collaborator

ddavidzhang Oct 18, 2024

wu-sheng Oct 18, 2024 Collaborator

ddavidzhang Oct 21, 2024

ddavidzhang Oct 21, 2024

wu-sheng Oct 21, 2024 Collaborator

yuchao86
Oct 17, 2024

Replies: 1 comment 7 replies

wu-sheng
Oct 17, 2024
Collaborator

wu-sheng Oct 18, 2024
Collaborator

wu-sheng Oct 21, 2024
Collaborator