Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting deployment status and logs can cause race conditions #73

Open
ankitkumarr opened this issue Jul 18, 2019 · 3 comments
Open

Getting deployment status and logs can cause race conditions #73

ankitkumarr opened this issue Jul 18, 2019 · 3 comments

Comments

@ankitkumarr
Copy link
Contributor

During kudulite function app build, if the client tries at any point of time to poll the deployment status and logs, we get locking issues which often cause the deployment to fail.

As a work-around, at the moment, the client is using /api/isdeploying endpoint to check if any deployments are ongoing. This can be unreliable if as it's not associated with a specific deployment id. Even with this setup, I still see a status.lock file in /home/site/locks with this error below --

"OperationName": "Getting deployment status",
  "AcquiredDateTime": "<my-timestamp>",
  "StackTrace": "   at System.Environment.get_StackTrace()\n   at Kudu.Core.Infrastructure.WindowsLockFile.WriteLockInfo(String operationName, Stream lockStream)
 in /tmp/KuduLite/Kudu.Core/Infrastructure/WindowsLockFile.cs:line 212
...
@Hazhzeng
Copy link
Contributor

Hazhzeng commented Aug 6, 2019

The root cause of this issue is due to a race condition when both DeploymentManager and the FileLogger try to access the same log file. Fixing the root issue should also mitigate this issue (also occurs in Kudu).

@ankitkumarr
Copy link
Contributor Author

@Hanzhzeng, do you know when does that happen (the code block), and if you have a fix recommendation?
I know Sanchit was working on some locking improvements for Kudulite.

@Hazhzeng
Copy link
Contributor

Hazhzeng commented Aug 6, 2019

@Hanzhzeng, do you know when does that happen (the code block), and if you have a fix recommendation?
I know Sanchit was working on some locking improvements for Kudulite.

I heard that will be deployed by ANT 84, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants