Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move refreshCheckout out of db transaction #2379

Merged
merged 1 commit into from
May 9, 2023

Conversation

finestructure
Copy link
Member

@finestructure finestructure commented May 9, 2023

This should address our analysis hangs. There is perhaps an issue upstream how db transactions are being handled in FluentKit and console-kit should perhaps not hang (reported here) but with this in place I believe we'll get rid of the service hangs that require manual intervention.

This has become more frequent recently, and I believe the reason is simply that we're seeing more new packages being added, which leads to a higher chance of a checkout (+ update work) taking us across the 10s event loop timeout.

Fixes #2227

@cla-bot cla-bot bot added the cla-signed label May 9, 2023
@finestructure finestructure requested a review from daveverwer May 9, 2023 08:13
@finestructure finestructure enabled auto-merge May 9, 2023 08:13
@finestructure
Copy link
Member Author

Another reason this has likely become more frequently noticed is the following: A few weeks ago I added an alert that explicitly fires when analysis doesn't log anything within 5 minutes (== hang detection). So now we get a "hang alert" pretty much immediately.

Previously, we only received alerts when there were no build reports after 4 hours (!) - with the underlying reason being a stalled analysis.

This long delay between issue and alert meant that we probably "solved" a number of those hangs by redeploying while the system was in a pending alert state, thus reducing the perceived incident frequency. (In particular on dev, where we deploy quite frequently.)

@finestructure finestructure merged commit fbcae7a into main May 9, 2023
@finestructure finestructure deleted the issue-2227-connection-deadlock branch May 9, 2023 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"Received signal 11" in app_analyze
2 participants