Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Instantiate SR only once per sensor - Kafka Trigger Sensor #3380

Open
Thomasr-02 opened this issue Dec 3, 2024 · 1 comment
Open

Instantiate SR only once per sensor - Kafka Trigger Sensor #3380

Thomasr-02 opened this issue Dec 3, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@Thomasr-02
Copy link
Contributor

Thomasr-02 commented Dec 3, 2024

Describe the bug

When a sensor is frequently triggered, it results in over X requests per second (rq/s) to the Schema Registry. This level of traffic is unnecessary and causes performance issues in a shared Schema Registry, leading to degraded functionality. While the library used for schema management is capable of caching, the cache is not utilized effectively due to individual triggers for each event.

To Reproduce
Steps to reproduce the behavior:

  1. Create a kafka sensor
  2. Trigger the kafka sensor
  3. Monitoring schema registry requests

Expected behavior
The sensor should instantiate and retrieve the schema from the registry only once per sensor activation. This approach reduces the total number of requests made to the Schema Registry, improving overall performance and efficiency.

Details
The instante SR in kafka.go already doing 1 call and 1 call for get schema, after them seeing it on the riferrei/srclient lib used, cache already implemented by default. The problem here is about many insantiate of trigger called by sensorCtx.triggerActions

Screenshots
image
Requests per second to SR

Environment (please complete the following information):

  • Kubernetes: 1.27.13
  • Argo Events: 1.7.5

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

@Thomasr-02 Thomasr-02 added the bug Something isn't working label Dec 3, 2024
@Thomasr-02 Thomasr-02 changed the title Cache Schema Registry Schema - Kafka Trigger Sensor Cache Schema Registry Client - Kafka Trigger Sensor Dec 7, 2024
@Thomasr-02 Thomasr-02 changed the title Cache Schema Registry Client - Kafka Trigger Sensor Instantiate SR only once per sensor - Kafka Trigger Sensor Dec 10, 2024
@Thomasr-02
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant