Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

K2 FIND comamnd #535

Merged
merged 26 commits into from
Jul 11, 2024
Merged

K2 FIND comamnd #535

merged 26 commits into from
Jul 11, 2024

Conversation

subbyte
Copy link
Member

@subbyte subbyte commented Jul 4, 2024 •
edited
Loading

Done

  1. Add relation resolution tables
  2. Add multi-column support to RefComparison (matching both pid and endpoint.id for identifying process)
    • Update instruction and comparison data structure
    • Upgrade codegen for default SQL engine
    • Upgrade codegen for default Dataframe engine
    • Upgrade sql cache module to support multi-column in RefComparison
    • Upgrade inmemory cache module to support multi-column in RefComparison
  3. Add CTE support to Sqlalchemy Interface
  4. Add multi-column support of RefComparison to Sqlalchemy Interface
  5. Add ReferenceDataSource instruction (will be resolved in parser directly)
  6. Add kestrel.irgraph.get_datasource_of_node() method
  7. Upgrade CTE support to Sqlalchemy Interface to fix a bug on retrieving dialect
  8. Upgrade CTE support in sql cache module (applying the same code as above)
  9. Add preliminary writable data lake in Sqlalchemy Interface support (resolved How to use the data from another interface/cache #532)
  10. FIND parsing support (entity identifier config)
  11. Add irgraph into parse_kestrel()
  12. Fixed Better way to handle variable type for APPLY #534
  13. Upgrade all methods in parser transformer to update graph in place
  14. Add transactional roll back in session if parsing failed
  15. Add empty Filter support
  16. IRGraph generation for FIND
  17. Add support for ProjectEntity in SQL cache
  18. Add tests
    • parser
    • cache inmemory (dataframe)
    • cache sql

Todo (Next PR)

  1. Add FIND tests for Sqlalchemy Interface

@subbyte subbyte changed the title K2 Relation Resolution K2 FIND comamnd Jul 6, 2024
@subbyte subbyte marked this pull request as ready for review July 8, 2024 15:38
@subbyte subbyte requested a review from pcoccoli July 8, 2024 15:38
@subbyte subbyte requested a review from pcoccoli July 9, 2024 20:42
@codecov Codecov
Copy link

codecov bot commented Jul 9, 2024

Codecov Report

Attention: Patch coverage is 82.34086% with 86 lines in your changes missing coverage. Please review.

Project coverage is 81.63%. Comparing base (f0b8e6a) to head (d35bf14).
Report is 8 commits behind head on develop.

Files Patch % Lines
...hemy/src/kestrel_interface_sqlalchemy/interface.py 18.36% 40 Missing ⚠️
...kages/kestrel_core/src/kestrel/frontend/compile.py 90.90% 16 Missing ⚠️
...emy/src/kestrel_interface_sqlalchemy/translator.py 85.45% 8 Missing ⚠️
...lalchemy/src/kestrel_interface_sqlalchemy/utils.py 22.22% 7 Missing ⚠️
...el_core/src/kestrel/interface/codegen/dataframe.py 84.61% 4 Missing ⚠️
packages/kestrel_core/src/kestrel/config/utils.py 76.92% 3 Missing ⚠️
packages/kestrel_core/src/kestrel/session.py 72.72% 3 Missing ⚠️
.../kestrel_core/src/kestrel/interface/codegen/sql.py 95.23% 2 Missing ⚠️
packages/kestrel_core/src/kestrel/cache/sql.py 95.65% 1 Missing ⚠️
packages/kestrel_core/src/kestrel/ir/filter.py 90.90% 1 Missing ⚠️
... and 1 more
Additional details and impacted files 
@@             Coverage Diff             @@
##           develop     #535      +/-   ##
===========================================
- Coverage    82.19%   81.63%   -0.57%     
===========================================
  Files           43       46       +3     
  Lines         2752     2978     +226     
===========================================
+ Hits          2262     2431     +169     
- Misses         490      547      +57

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@subbyte subbyte merged commit d95c854 into develop Jul 11, 2024
37 checks passed
@subbyte subbyte deleted the k2-relation-resolution branch July 11, 2024 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pcoccoli pcoccoli pcoccoli approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Better way to handle variable type for APPLY How to use the data from another interface/cache
2 participants
@subbyte @pcoccoli