-
Notifications
You must be signed in to change notification settings - Fork 294
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Endless pipeline loop when trying to import corrupted document #408
Comments
Does this happen only with SimpleQueues? By design SimpleQueues doesn't support poison queues, so that at dev time one can debug without having to worry about the number of retries. With AzureQueue and RabbitMQ the number of retries should be capped, so that eventually the service stops retrying. |
I have tried with Azure Queues, I have verified that the message is moved to a poison queue after a certain number of retries. However, the number of retries is hard-coded:
What do you think about adding this parameter in AzureQueuesConfig? For what concerns RabbitMQ, it seems that the message is always put in the queue again: kernel-memory/extensions/RabbitMQ/RabbitMQPipeline.cs Lines 105 to 143 in f266b25
|
no problem about making it configurable if it helps. If I remember correctly Azure Queues uses a count to decide when to discard a message, while RabbitMQ uses an expiration date, time to live. I would check the logs for this message |
If I correctly understand the meaning of the
It enqueue the a new message again in the queue, with a new expiration time, endlessly. |
By the way, there are some issues in
It seems that all the properties of |
@dluc I have made a PR to set the missing properties in RabbitMQ: #454. After that, I think that also in |
## Motivation and Context (Why the change? What's the scenario?) See #408 (comment). This PR sets the `MessageId` and `Expiration` properties for RabbitMQ messages. --------- Co-authored-by: Devis Lucato <[email protected]>
There's a couple of approaches, IIRC all of them require multiple queues per queue. Worth reading: |
So, do you prefer using such solutions instead of the quorum queue? |
quorum queues don't seem to offer a delayed delivery, do they? When a message delivery fails we need to allow a configurable retry strategy, e.g.
AFAIK with RabbitMQ it will require multiple queues. |
I have understood, so at least we need something like this, right? |
Hi @dluc! Finally, I can return to this issue. What do you think if we start implementing also in RabbitMQ the same behavior we have in Azure Queues? kernel-memory/extensions/AzureQueues/AzureQueuesPipeline.cs Lines 192 to 231 in 8c0ad8c
|
hey @marcominerva that would be nice, if possible. It might be a lot of work considering the extra queues to introduce - so, before starting, do you think we could break it down in multiple steps, to have separate PRs? |
Yes. I can start, for example, creating the extra queues and then making a draft PR, so you can see the work in progress. Do you agree? |
yes please go ahead - what would be the name of the queues? I'm assuming when missing, if deleted, these extra queues will be automatically recreated, without causing errors. |
Actually, adding poison queues support like in Azure Queues has required a smaller amount of changes than I thought, so I have make a single PR with all the changes: #648. Let me know if it is OK, or do you prefer to split it in any case. |
## Motivation and Context (Why the change? What's the scenario?) Avoid infinite retries when using RabbitMQ, and move poison messages after N retries. See #408. --------- Co-authored-by: Devis Lucato <[email protected]>
Context / Scenario
In service mode (using a queue), when trying to import a corrupted document (i.e., invalid PDF file), decoder will throw an exception, but then the message will be put again in the queue, generating an endless loop:
What happened?
If decoding fails with an exception, I expect that the document will be marked as
failed to process
and not process again. I see also that in the DataPipelineStatus class, there is aFailed
property, but it is alwaysfalse
:kernel-memory/service/Abstractions/Pipeline/DataPipeline.cs
Lines 432 to 448 in babedc0
Importance
I cannot use Kernel Memory
Platform, Language, Versions
Kernel Memory v0.36.240415.2
Relevant log output
The text was updated successfully, but these errors were encountered: