Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support multi-tenant RAM buffers for IndexWriter #13913

Open
mdmarshmallow opened this issue Oct 14, 2024 · 3 comments
Open

Support multi-tenant RAM buffers for IndexWriter #13913

mdmarshmallow opened this issue Oct 14, 2024 · 3 comments
Labels
type:task

Comments

@mdmarshmallow
Copy link
Contributor

Description

This is related to #13883. The idea is to allow users to specify the RAM usage once and it will be automatically spread across N IndexWriter's so they will each have 1/N RAM allocated to them.
@mikemccand
Copy link
Member

There was also some quick discussion about multi-tenant (multiple active IndexWriters on a single JVM) RAM buffers easier at #13387.

@mdmarshmallow
Copy link
Contributor Author

After some discussion with @mikemccand, I'm currently planning on making a separate RAM manager that will force that largest IndexWriters to flush their buffers if the total RAM used by all the writers at the configured limit. Polling the writers might be tricky since I don't really want to make a separate thread to just poll. Anyways I will hopefully have something soon!

@mdmarshmallow
Copy link
Contributor Author

Here is a draft PR if anyone is interested in sanity checking my approach: #13951

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type:task
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikemccand @mdmarshmallow