-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Transactions for Indexer Code Execution #340
Comments
So after a lot of testing, I found the problem. It turns out that the lazy loading implementation I used for context.db was set up so that any calls to a db function would first check if the dmlHandler exists (Load on access). If not, then instantiate it. This is helpful for situations where the functions aren't used. However, this had a side effect: If a bunch of db function calls come in simultaneously, they will all poll dmlHandler and see it is uninitialized and all try and create a new one. Each initialization will try to create a pool with 10 available clients. And then all of them will create at least one client each. This will exhaust the clients very quick if there's enough of them. Changing it so that the dmlHandler is lazy loaded immediately instead. And on access, it will await its completion before calling any functions. This ensures there is no more than 1 client active for that invocation. I wrote a test as well which tries 100 simultaneous calls and verified 1 pool is created. |
We do have the intention of having transaction support as well. This requires using one connection and opening up the dmlHandler to be accessible by the Indexer class, so more work is needed there. And more tests too. |
Changes to fix connection issues: #413 |
Introducing the PR as it is might add more scaling barriers, which have become a greater concern lately after a prod crashing issue. It's probably more prudent to simply implement this with PgBouncer as there is no rush. |
pgBouncer was a blocker for this, we can start working on it now |
see https://github.com/near/queryapi-mvp/blob/0c9fa36142538e5d009b633209272fa412c47560/runner/src/pg-client.ts#L33
The text was updated successfully, but these errors were encountered: