Service Data Loss

[ArcherDestroy]

SkyWalking was deployed on October 30. After the deployment was completed, the service data could be queried normally. However, at around 2:50 pm on November 7, it was suddenly found that all service data in SW disappeared.
After I restarted SW, it returned to normal.
The configuration of SW uses the default configuration and there is no special customized adjustment. MySQL is used for data storage.
After consulting the relevant logs, there is the following error. The log only mentions "Connection refused", but there is no specific reason.
The deployment situation of SW is as follows:
SkyWalking 10.1.0
SkyWalking-go 0.5.0
MySQL 8.0

Here is the error log I queried：

2024-11-07 14:51:54,589 - com.linecorp.armeria.server.DefaultUnloggedExceptionsReporter - 104 [armeria-common-worker-epoll-2-5] WARN  [] - Observed 2 exception(s) t
hat didn't reach a LoggingService in the last 10000ms(10000000000ns). Please consider adding a LoggingService as the outermost decorator to get detailed error logs.
 One of the thrown exceptions:
com.linecorp.armeria.client.UnprocessedRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: l
ocalhost/127.0.0.1:12800
        at com.linecorp.armeria.client.UnprocessedRequestException.of(UnprocessedRequestException.java:45) ~[skywalking-webapp.jar:?]
        at com.linecorp.armeria.client.HttpChannelPool.notifyConnect(HttpChannelPool.java:541) ~[skywalking-webapp.jar:?]
        at com.linecorp.armeria.client.HttpChannelPool.lambda$connect$1(HttpChannelPool.java:363) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[skywalking-webapp.jar:?]
        at com.linecorp.armeria.client.HttpChannelPool.lambda$connect$2(HttpChannelPool.java:414) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:679) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:698) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[skywalking-webapp.jar:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[skywalking-webapp.jar:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[skywalking-webapp.jar:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:12800
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:166) ~[skywalking-webapp.jar:?]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[skywalking-webapp.jar:?]
        at io.netty.channel.unix.Socket.finishConnect(Socket.java:359) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:715) ~[skywalking-webapp.jar:?]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:692) ~[skywalking-webapp.jar:?]
        ... 7 more

[wu-sheng]

The connection refused usually means, your OAP servers are down.

[ArcherDestroy]

How can I know the specific reason for the OAP servers down? Or is there a way to avoid this problem? Because judging from the time when this problem occurred, the OAP servers has not been running for a long time.

[wu-sheng]

You should watch all server status, set up self observability, or your own way.

After all, my point is, this log is not root cause at all.

[ArcherDestroy]

OK, I will try. If I have any new discoveries, I will ask again.