Prod Incident 25/07/24 - Hasura 429/504 errors, Postgres Connection Limit, and Unexpected Deprovisioning

[darunrs]

On July 25, 2024, Rate Exceeded errors were observed from the production Hasura instance. Following this, an investigation was performed with the help of SRE. One of the actions taken was raising the concurrent request limit of each Hasura instance from 80 to 200 while increasing the max instances from 5 to 10. This increase was sufficient to stop the Rate Exceeded error. The following morning, it was discovered that the number of DB connections had spiked to 600 and was floating at 600, with the number of active connections locked at roughly 400. As a result, Hasura did not have enough connections to maintain its metadata, causing it to fall out of sync. This led to QueryApi once again experiencing issues. After QueryApi was shut down in prod, and the database restarted, the connection count fell. However, when QueryApi restarted, it immediately began to deprovision many indexers without cause. QueryApi was shut down again, and the deprovisioning was investigated. After the impacted indexers were documented, QueryApi was restarted with a custom commit which increased the timeout between stalled stream/executor restart attempts, and disabled deprovisioning. After this, the deprovisioned indexers were all brought back and backfilled on Jul 26, 2024.

TLDR:

• Hasura rejects all requests due to accumulated timeout queries from either a block stream which was being repeatedly restarted or from KitWallet which tried accessing Hasura after Postgres connections reached the limit
• Postgres connections rapidly rise to maximum due to above timing out queries from QueryApi creating permanently active connections
• Indexers suddenly deprovisioned when not deleted from contract

More details on Incident Document.

I've separated the task list into two as the two incidents are unrelated.

Hasura and Postgres Incident

Beta Give feedback

1. Add a timeout before restart of a stalled executor or block stream #931
   component: Coordinator +1
   
2. Temporarily disable deprovisioning #938
   component: Coordinator +1
   
3. Increase block stream stalled health check #948
   +0
4. Reduce Hasura timeout to 2 minutes #946
   +0
5. Add logs for transitions in health state #947
   +0
   
6. Introduce exponential retry for restarts of stalled stream/executor #949
   +0
   
7. Investigate perpetually active postgres connections from restarting block streams #950
   +0
8. Address relatedReceiptId INFO log message #951
   +0
9. Prevent crashes of worker due to errors originating from VM #952
   +0
10. Create Indexer Metrics Page which has all indexer metrics #978
    +0
11. Migrate Hasura to work through pgBouncer
12. Coordinator publishes metrics on indexer's resources #967
    +0

Deprovisioning Incident

Beta Give feedback

1. Update contract to return specific data when fetching deleted indexer #940
   component: Registry +1
   
2. Add Status Code and content check for indexer config fetch #941
   component: Coordinator +1
   
3. Restart all broken indexers #979
   component: Coordinator +1
   
4. Add logs for all state transitions #942
   +0
   
5. https://github.com/near/queryapi/issues/965
6. Refactor Provisioning into Multi-Step Process #915
   component: Runner +1
   
7. Update contract to use enums for config instead of struct #964
   +0
8. Convert deprovisioning to flagging #953
   +0
9. Add ability to trigger backfill manually #943
   +0
10. Add guard rail for deprovisioning to prevent deprovisioning core indexers #944
    +0
11. Prevent automatic deprovision of indexers with active hasura usage #945
    +0
12. Create scheduled job to deprovision indexers #954
    +0
13. Handle indexers deleted during Coordinator downtime #968
    Ungroomed component: Coordinator +2