Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API Services Fail to Start Silently #5820

Closed
halibobo1205 opened this issue May 13, 2024 · 14 comments · May be fixed by #5857
Closed

API Services Fail to Start Silently #5820

halibobo1205 opened this issue May 13, 2024 · 14 comments · May be fixed by #5857
Labels

Comments

@halibobo1205
Copy link
Contributor

Software Versions

OS: Linux
JVM: Oracle Corporation 1.8.0_161 AMD64
Version: 4.7.4
Code: 18260

Expected behavior

An exception should be thrown and the node service should exit when the API services fail to start.

Actual behavior

Node starts with no API services and no exception about it thrown.

Steps to reproduce the behavior

  1. Make gRPC service port 50051 occupied in advance
  2. Start the node
  3. Try to access the API endpoint

Backtrace

14:54:41.976 ERROR [rpc] RpcApiService
java.io.IOException: Failed to bind to address 0.0.0.0/0.0.0.0:50051
	at io.grpc.netty.NettyServer.start(NettyServer.java:328)
Caused by: java.net.BindException: Address already in use
14:54:41.983 INFO [app] All api services started.
UNIMPLEMENTED: Method not found: protocol.Wallet/GetAccount
io.grpc.StatusRuntimeException: UNIMPLEMENTED: Method not found: protocol.Wallet/GetAccount
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
@zeusoo001
Copy link
Contributor

@halibobo1205 In addition to api services, do other services also have such demands?

@halibobo1205
Copy link
Contributor Author

@zeusoo001, IMHO: external services should not silently fail to start, such as event-plugins.

@halibobo1205
Copy link
Contributor Author

halibobo1205 commented May 15, 2024

external services :

TYPE SERVICE default available note
API RpcApiService ⚠️ force-enabled FullNode\SolidityNode  GRPC
API RpcApiServiceOnSolidity ⚠️ force-enabled FullNode  GRPC
API RpcApiServiceOnPBFT ⚠️ force-enabled   FullNode  GRPC
API FullNodeHttpApiService ✅ enabled FullNode  HTTP
API HttpApiOnSolidityService ✅ enabled   FullNode  HTTP
API HttpApiOnPBFTService ⚠️ force-enabled   FullNode  HTTP
API SolidityNodeHttpApiService ❌ disabled SolidityNode  HTTP
API FullNodeJsonRpcHttpService ❌ disabled FullNode  JSON-RPC
API JsonRpcServiceOnSolidity ❌ disabled FullNode  JSON-RPC
API JsonRpcServiceOnPBFT ❌ disabled FullNode  JSON-RPC
metric MetricsUtil ❌ disabled FullNode\SolidityNode  influxdb
metric Metrics ❌ disabled FullNode\SolidityNode  Prometheus
EVENT EventPluginLoader ❌ disabled FullNode\SolidityNode event subscribe
NET TronNetService ✅ enabled FullNode\SolidityNode  
ZK-SNARK ZksnarkInitService ✅ enabled FullNode\SolidityNode  shield transaction

@halibobo1205
Copy link
Contributor Author

Force-enabled services: RpcApiService, RpcApiServiceOnSolidity, RpcApiServiceOnPBFT, and HttpApiOnPBFTService need to be adjusted to optional before the fix.

@lxcmyf
Copy link
Contributor

lxcmyf commented May 16, 2024

Do you mean that if API services fail to start, they need to throw real-time exception errors instead of only reporting errors when accessing the interface? This may require adjusting the priority of each service startup. It would be great if the startup mechanism could be optimized and error feedback could be provided.

@halibobo1205
Copy link
Contributor Author

This may require adjusting the priority of each service startup.
@lxcmyf why it needs to be adjusted?

@lxcmyf
Copy link
Contributor

lxcmyf commented May 17, 2024

@halibobo1205
I would like to say that various services may be divided into many types according to their functions, but there are also similar services that can be classified under the same category. Real time startup errors can be classified by type, making the sorting of errors clearer.

@317787106
Copy link
Contributor

@halibobo1205 Should you check whether some configured ports are occupied before the main thread starts ?

@halibobo1205
Copy link
Contributor Author

halibobo1205 commented May 17, 2024

@halibobo1205 Should you check whether some configured ports are occupied before the main thread starts ?

@317787106 There is no need to do this, just throw the associated error. See besu how to deal with this:

  private void waitForServiceToStart(
      final String serviceName, final CompletableFuture<?> startFuture) {
    do {
      try {
        startFuture.get(60, TimeUnit.SECONDS);
      } catch (final InterruptedException e) {
        Thread.currentThread().interrupt();
        throw new IllegalStateException("Interrupted while waiting for service to start", e);
      } catch (final ExecutionException e) {
        throw new IllegalStateException("Service " + serviceName + " failed to start", e);
      } catch (final TimeoutException e) {
        LOG.warn("Service {} is taking an unusually long time to start", serviceName);
      }
    } while (!startFuture.isDone());
  }

@halibobo1205
Copy link
Contributor Author

Real time startup errors can be classified by type, making the sorting of errors clearer.

@lxcmyf I don't think it's necessary, the error is even less clear, and the original error is still needed, you can see the sample above.

@halibobo1205 halibobo1205 changed the title API Services Fail to Start Sliently API Services Fail to Start Silently May 17, 2024
@halibobo1205
Copy link
Contributor Author

An exception should be thrown and the node service should exit when all external services fail to start.

Working on this.

@halibobo1205
Copy link
Contributor Author

An exception should be thrown and the node service should exit when the API services fail to start.

Is it necessary to exit the node service? PTAL @317787106 @lxcmyf @tomatoishealthy

@tomatoishealthy
Copy link
Contributor

At present, some products I have used will not interrupt the main process when encountering some non-core service exceptions. Maybe TRON does not need to be forced to stop in this scenario?

But this is a very open topic, and it is also reasonable to stop the service when an exception is thrown.

I'd probably prefer to keep the service running, but I'd be fine with either.

@halibobo1205
Copy link
Contributor Author

Final decision: exit the node service when API services fail to start. cc @tomatoishealthy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants