Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve consistency of sync_compatible when running flows in remote environments #14660

Merged
merged 3 commits into from
Jul 19, 2024

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented Jul 18, 2024

This PR ensures that functions wrapped in @sync_compatible execute as expected in remote environments by setting RUNNING_ASYNC_FLAG to False when running a sync function in a worker thread. Flow scripts are loaded in a worker thread without an event loop, so updating the RUNNING_ASYNC_FLAG context variable will ensure those wrapped async functions execute as sync.

I noticed that a couple of threads are created when loading a flow script. The number of created threads could be reduced by creating a sync implementation of load_flow_from_flow_run, but we'd likely still need to call storage block methods and pull steps in a separate thread since they are async by default.

Closes #14625

Example

Checklist

  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.
  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • If this pull request removes docs files, it includes redirect settings in mint.json.
  • If this pull request adds functions or classes, it includes helpful docstrings.

@desertaxle desertaxle added the fix A fix for a bug in an existing feature label Jul 18, 2024
Copy link

codspeed-hq bot commented Jul 18, 2024

CodSpeed Performance Report

Merging #14660 will not alter performance

Comparing sync-compatible-remote (cde51ec) with main (e091528)

Summary

✅ 5 untouched benchmarks

@desertaxle desertaxle marked this pull request as ready for review July 19, 2024 15:21
@desertaxle desertaxle requested a review from a team as a code owner July 19, 2024 15:21
Copy link
Collaborator

@zangell44 zangell44 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love the regression test!

@desertaxle desertaxle merged commit 96140de into main Jul 19, 2024
30 checks passed
@desertaxle desertaxle deleted the sync-compatible-remote branch July 19, 2024 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix A fix for a bug in an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sync_compatible decorator returning coroutines unexpectedly when running a flow
2 participants