Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The firebase-admin-java 9.4.0 does not work with NIO (Non-blocking I/O). #1029

Open
stager0909 opened this issue Oct 28, 2024 · 1 comment
Open

Comments

@stager0909
Copy link

[READ] Step 1: Are you in the right place?

  • For issues or feature requests related to the code in this repository
    file a Github issue.
    • If this is a feature request make sure the issue title starts with "FR:".
  • For general technical questions, post a question on StackOverflow
    with the firebase tag.
  • For general Firebase discussion, use the firebase-talk
    google group.
  • For help troubleshooting your application that does not fall under one
    of the above categories, reach out to the personalized
    Firebase support channel.

[REQUIRED] Step 2: Describe your environment

  • Operating System version: �ubuntu
  • Firebase SDK version: firebase-admin-java 9.4.0
  • Library version: java 17
  • Firebase Product: fcm

[REQUIRED] Step 3: Describe the problem

Hello,
I am testing the firebase-admin-java SDK 9.4.0 with the following configuration:

FirebaseOptions options = FirebaseOptions.builder()
    .setCredentials(googleCredentials(firebaseCredential, httpTransport))
    .setDatabaseUrl(firebaseDatabaseUrl)
    .setHttpTransport(httpTransport)
    .setConnectTimeout(FIREBASE_CONNECT_TIMEOUT)
    .setReadTimeout(FIREBASE_READ_TIMEOUT)
    .setThreadManager(new CustomFirebaseThreadManager(
        firebaseTaskExecutor.getThreadPoolExecutor()))
    .build();

The reason I explicitly added ThreadManager is to monitor the ThreadPoolExecutor. Without this configuration, the ThreadPoolExecutor generated by FirebaseThreadManagers.DefaultThreadManager.doInit() will be used, which has a queue size of Integer.MAX_VALUE. If FCM responses are delayed, the queue can pile up, potentially causing an OOM (OutOfMemoryError).

Below is the configuration for H2AsyncClientBuilder:

H2AsyncClientBuilder h2AsyncClientBuilder = H2AsyncClientBuilder.create()
    .setDefaultConnectionConfig(connectionConfig)
    .evictIdleConnections(TimeValue.of(Duration.ofMinutes(10)))
    .setDefaultRequestConfig(requestConfig())
    .setIOReactorConfig(IOReactorConfig.custom()
        .setSoTimeout(FIREBASE_READ_TIMEOUT, TimeUnit.MILLISECONDS)
        .build())
    .addRequestInterceptorFirst(
        (HttpRequest request, EntityDetails entity, HttpContext context) -> {
            activeStream.incrementAndGet();
            if (context != null) {
                context.setAttribute("startTime", System.nanoTime());
            }
        })
    .addResponseInterceptorFirst(
        (HttpResponse response, EntityDetails entity, HttpContext context) -> {
            if (context == null || context.getAttribute("startTime") == null) {
                return;
            }
            long startTime = Long.parseLong(context.getAttribute("startTime").toString());
            long endTime = System.nanoTime();
            if (startTime < endTime) {
                Timer.builder(FCM_RESPONSE_TIMER_NAME)
                    .publishPercentiles(0.50, 0.95, 0.99)
                    .tags(
                        Lists.newArrayList(
                            Tag.of("host", ((HttpRoute) context.getAttribute("http.route")).getTargetHost().getHostName()),
                            Tag.of("http_version", context.getProtocolVersion().toString())
                        )
                    )
                    .register(meterRegistry)
                    .record(endTime - startTime, TimeUnit.NANOSECONDS);
            }
            activeStream.decrementAndGet();
        })
    .disableCookieManagement()
    .setH2Config(H2Config.custom().setMaxConcurrentStreams(500 * 10).build())
    .disableRedirectHandling()
    .disableAutomaticRetries();

Here is the code for sending FCM messages. The postProcessExecutor is simply responsible for logging the send results:

ApiFuture<String> responseApiFuture = firebaseMessaging.sendAsync(fcmMessage);
responseApiFuture.addListener(() -> {
    try {
        responseApiFuture.get();
        message.setResponseCode(FcmResult.SUCCESS.getCode());
        postProcessor.postProcess(MessageStatus.SENT, message);
    } catch (Exception e) {
        log ~~~
    }
}, postProcessorExecutor);

When testing this, if I set a breakpoint inside addRequestInterceptorFirst (or addResponseInterceptorFirst) in the second code block, the thread executing addRequestInterceptorFirst is the httpclient-dispatch thread.
At this point, firebaseThreadPoolTask.getActiveCount() (as set in FirebaseOptions.builder().setThreadManager()) is 1.
In other words, it seems like the firebaseThread is staying active while the httpclient-dispatch thread is running.

I would like the behavior to work as follows:

The firebaseThread performs tasks (such as SDK validation) before the HTTP call.
After handing over to httpclient-dispatch, the firebaseThread is released.
The httpclient-dispatch makes the request to the FCM server, and the IO selector detects when a response is received.
Once the IO selector detects the response, it hands over to httpclient-dispatch, which processes operations like interceptors and then passes it to postProcessorExecutor.
This behavior should prevent the firebaseThread from getting blocked when the FCM server is down or latency is high.
If the firebaseThread gets blocked, it means that the queue in the thread pool fills up, which can lead to OOM or RejectedExecutionException.

If I have misunderstood anything, please let me know.
If my understanding is correct, how can I resolve this issue?

@google-oss-bot
Copy link

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants