Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rib Robustness: Make sure the type building using AllOf and OneOf isn't giving memory pressure. #894

Open
afsalthaj opened this issue Sep 12, 2024 · 1 comment
Assignees
Labels

Comments

@afsalthaj
Copy link
Contributor

afsalthaj commented Sep 12, 2024

After multiple scanning for type inference going on, we keep building the types using AllOf/OneOf. For example: it looks AllOf(all_possibilities_of_record) is looking pretty huge!

Recently (first week of Sept), we made sure we are not storing duplicate types, and there are no logical errors in this side.
Yet I believe we still have a considerable memory footprint that we need to be cautious about. I will be doing a benchmark before we plan the solution for this ticket.
May be this is not an immediate issue, yet placing a ticket to track it down later.

There are various ways to handle this building of large data structures representing types.

  1. Maintaining a table instead of attaching the inferred type to every identifier as an optimisation. Expressions may not repeat, but identifiers do!
  2. Pre-unification strategies instead of accumulating types that could have gone through an obvious unification. Example: No need to keep this towards the end AllOf( Result { ok: Some(U64), error: Some(Unknown) }, Result { ok: Some(U64), error: Some(Str) }). We could come up with a progressive type elimination to keep the memory footprint low
  3. And a lot more...

Or may be there is a known theoretical way of doing this, and need discussions with @jdegoes .

@afsalthaj afsalthaj self-assigned this Sep 12, 2024
@vigoo vigoo added the rib label Sep 13, 2024
@afsalthaj afsalthaj changed the title Make sure the type building using AllOf and OneOf isn't giving memory pressure. Rib Robustness: Make sure the type building using AllOf and OneOf isn't giving memory pressure. Sep 18, 2024
@jdegoes
Copy link
Contributor

jdegoes commented Sep 18, 2024

It's easier in GC world because the values can be shared more easily. I would suggest two obvious things:

  1. Use a canonical representation: either "and of ors" or "or of ands".
  2. Use a set for the "ands" and a set for the "ords" so you don't store duplicates.

These two changes should have a very good effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants