Subscription is not automatically listening for new messages

[wagnerww]

Thanks for stopping by to ask us a question! Please make sure to include:

• What you're trying to do:
  I'm trying to read a subscription. However, when starting the process it receives some messages that are already published and after it is "idle", stopped, doing nothing. Wouldn't that be similar to Go or Java, where it runs and listens for messages being posted?

• What code you've already tried:

   const pubSubClient = new PubSub();
   const subscription = pubSubClient.subscription("my-sub");
   
   subscription.on('message', (message) => {
      /// process
      message.ack();
  });
  
  subscription.on('error', logger.log);
  

• Any error messages you're getting:
  I don't get an error, but I had to write this code below to let my CLI/Batch application listen for messages....but it doesn't seem like a good solution :(

     const pubSubClient = new PubSub();
  
     const listenSub =  () => {
         const subscription = pubSubClient.subscription("my-sub");
         
         subscription.on('message', (message) => {
            /// process
            message.ack();
        });
        
        subscription.on('error', logger.log);
          
          setTimeout(() => {
            subscription.removeListener('message', messageHandler);
            subscription.removeListener('error', logger.log);
           
            logger.log(`Messagens recebidas: ${messageCount}`);
            
           return listenSub();
          }, timeout);
      }
  
      listenSub()
  

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

[brianofrokk3r]

I believe I am having similar/same issue.

Getting about 50-75 messages on the subscription, then the connection goes idle. Tried both ack, and ackWithResponse.

Meanwhile, the topic still contains > 1000 unacked messages.

When I restart the subscription, it gets another 50-75 messages, and does the same thing. I am using the default topic and subscription configuration with an extended acknowledgement deadline.

[brianofrokk3r]

@wagnerww

Using flowControl within subscriber options (see example), I was able to process any number of messages over a longer period of time. This seemed to resolve my specific issue. Using this in combination with your connection restart as per the example increases overall stability.

// configure options
    const subscriptionName = "SomeSubscription";
    const maxInProgress = 5; // 1-3 worked well for me, 5 started hanging occasionally
    const subscriberOptions = {
        flowControl: {
          maxMessages: maxInProgress,
        },
      };

//create subscription
const subscription = pubsub.subscription(subscriptionName, subscriberOptions);

// on message do something
subscription.on('message', (message) => {
      // messagehandler acks message
      messageHandler(message)
});

[feywind]

Hmm, I wonder if (like #1665) the acks don't seem to be going through. The server side itself does some flow control, and it would probably stop delivering messages if it's not receiving acks, which does seem to line up with the delivery behaviour.

If either of you have a support contract, it might be useful to have someone look at what's happening on the back end.

Out of curiosity, do either of you see the same issue if you go back to 3.0.x? I'm trying to figure out if there was a library regression during the exactly once delivery updates.

[himkwtn]

We are also facing the same issue where the subscriber client randomly stop pulling in new messages.

Environment

• docker image: node:16.18.1-bullseye-slim
• pubsub client version 3.1.0
• default flow control options
• messageOrdering enabled

What is weird is that not all subscriptions are facing this problem and some subscriptions stop more often than other.
Sometimes the subscription can recover itself for a few minutes and then goes idle.
This also rarely happened in the past but these past week it has been happening very often.

Right now we have to manually restart our backend application which is not ideal at all.

---

Examples

Subscription 1

Subscription 2

[regevbr]

same issue here

[panoc1]

Any updates on this? We are hitting the same issue here

[iamcrisb]

updates?

[feywind]

In regards to subscriptions randomly stopping, that's a long term issue we've been looking for solutions for, and we might've found some recently in the latest 4.x branch updates, with grpc keepalives. If possible, try the 4.x branch (requires Node 14, but otherwise basically the same as 3.x), and if problems are still happening, please file a support ticket, as we will likely need to take a look at your specific config and logs.