Doris fe master node http port request cannot be made, the 8030 web ui cannot be opened, and Prometheus metrics cannot be collected

[codejiwei]

Doris2.1.1 version
3FE, 5BE Doris cluster, 3 FEs are followers, all configured with Prometheus to collect Doris Metrics for cluster monitoring.

Problem phenomenon:
After the Master node in FE runs for a period of time, the HTTP 8030 interface of the Master node cannot be accessed; at the same time, the Metrics of the Master node cannot be collected. However, the FE process is still running and the FE status is Alive, and JDBC read and write are normal.

Current information:

1. There is no obvious error information in the FE log
2. The three nodes are load balanced, and the number of requests is not much different
3. The number of threads of the FE Master node has been increasing until it reaches 1w+ and the Metrics cannot be collected
4. The following are some screenshots on Grafana's Dashaboard

线程状态:
`"Thread-52171" #115023 daemon prio=5 os_prio=0 tid=0x00007f541499e000 nid=0x102485 waiting on condition [0x00007f5058fd0000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f66c4007878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:148)
at java.lang.Thread.run(Thread.java:750)

"Thread-52170" #115022 daemon prio=5 os_prio=0 tid=0x00007f541499c000 nid=0x102484 waiting on condition [0x00007f50597d1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f66c4007878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:148)
at java.lang.Thread.run(Thread.java:750)

"Thread-52169" #115021 daemon prio=5 os_prio=0 tid=0x00007f541480b800 nid=0x102483 waiting on condition [0x00007f5059fd2000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00007f66c4007878> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:47)
at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:148)
at java.lang.Thread.run(Thread.java:750)`

May I ask those with experience, what may be the reason?

[ixzc]

中文问题可以去ask.selectdb.com论坛提问