Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Late Filtering Operator for Queries #121

Open
ppanopticon opened this issue Nov 22, 2024 · 1 comment
Open

[FEATURE] Late Filtering Operator for Queries #121

ppanopticon opened this issue Nov 22, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@ppanopticon
Copy link
Member

Description

In the XReco context (and potentially for some other applications) we require late-stage filtering for queries. This type of filter removes Retrievables from a result stream generated by some upstream Retriever.

## Solution
I'd propose to have a LateFilter that can do late-stage filtering of Retrievables based on some boolean predicate. The way this filter works is roughly as follows:

  • It fetches the necessary field from the schema.
  • It performs a comparison of the field's value with a predicate.
  • It removes all Retrievables that don't match the predicate.
  • As a side-effect, the LateFilterOperator can append the fetched field to the Retrievable (similar to the FieldLookup).

For the sake of simplicity, a first implementation compares a single field to a single predicate. Logical combinations (AND, OR) are not supported. There are some implementation details to consider and optimise here:

  • Batch processing vs. stream processing for large result-sets
  • Attribute matching for structs

## Dependencies
No know dependencies.

Context

It is currently not in scope to push-down these filters and combine them with the retrievable operation (which could be done, e.g., in PostgreSQL). This would constitute a next step.

@net-cscience-raphael
Copy link
Contributor

See #123

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants