Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize StrawberryFlavor (OCR, etc) Indexing/Updates and aggregation #256

Open
DiegoPino opened this issue Feb 13, 2023 · 0 comments
Open
Assignees
Labels
enhancement New feature or request Events and Subscriber JMESPath a.*.wonderful.life[0] JSON Postprocessors Drupal Plugins that do stuff with JSON data Property Keys Providers Drupal plugins that harvest the possible keys that will be exposed to the rest of Drupal Search API F around and find out Strawberry Flavor Post Processing data extracted that goes into Solr Typed Data and Search
Milestone

Comments

@DiegoPino
Copy link
Member

DiegoPino commented Feb 13, 2023

What?

For the sake of consistency we do a lot of update/tracking/parent and child re-tracking on any changes on ADOs and Flavor Data Sources. But on a large production server a few ADO updates that really don't affect at all Flavors end reindexing 10K++ (add an extra +) Flavor Documents and the number of Queries/Back and forth that are happening are/might end being too much with danger of timeouts and server slowdown.

We can Optimize and make better decisions on when these heavy Server tasks are needed using more logic/conditionals and maybe even aggregation tasks into single backend queue items to avoid so much processing.

Pieces we need to touch are:

https://github.com/esmero/strawberryfield/blob/1.1.0/src/Plugin/search_api/processor/StrawberryFlavorAggregate.php

https://github.com/esmero/strawberryfield/blob/1.1.0/src/EventSubscriber/StrawberryEventSaveFlavorSubscriber.php

https://github.com/esmero/strawberryfield/blob/1.1.0/src/Plugin/search_api/datasource/StrawberryfieldFlavorDatasource.php

And their interactions. There is a lot of Dinosaur-egg-Dinosaur situations in these parent-child interactions and sometimes the lack of context (is the operation an Update or a First Time Index?) makes all more complex

@alliomeria @aksm @karomabiles This is what was discussed today

Thanks

@DiegoPino DiegoPino added enhancement New feature or request JSON Postprocessors Drupal Plugins that do stuff with JSON data Property Keys Providers Drupal plugins that harvest the possible keys that will be exposed to the rest of Drupal Events and Subscriber Typed Data and Search JMESPath a.*.wonderful.life[0] Strawberry Flavor Post Processing data extracted that goes into Solr Search API F around and find out labels Feb 13, 2023
@DiegoPino DiegoPino modified the milestones: 1.0.0-beta3, 1.1.0 Feb 13, 2023
@aksm aksm removed their assignment Sep 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Events and Subscriber JMESPath a.*.wonderful.life[0] JSON Postprocessors Drupal Plugins that do stuff with JSON data Property Keys Providers Drupal plugins that harvest the possible keys that will be exposed to the rest of Drupal Search API F around and find out Strawberry Flavor Post Processing data extracted that goes into Solr Typed Data and Search
Projects
None yet
Development

No branches or pull requests

2 participants