Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EventsPipeline and testing utilities #14765

Merged
merged 6 commits into from
Jul 26, 2024
Merged

Add EventsPipeline and testing utilities #14765

merged 6 commits into from
Jul 26, 2024

Conversation

jakekaplan
Copy link
Contributor

@jakekaplan jakekaplan commented Jul 25, 2024

Implement the EventsPipeline and the events_pipeline fixture.

With client side task run orchestration, task-runs are now eventually consistent. There are a lot of places in our tests where we are reading task runs back from the server to assert something about that finished task run. We still want to support that pattern. To do this, this PR introduces the following:

async def test_run_task(prefect_client: PrefectClient, events_pipeline):
    @task
    async def foo():
       pass
       
    # run the task  
    await run_task_async(foo, task_run_id=task_run_id) 
    
    # run the events pipeline to ensure eventual consistency is actually achieved
    await events_pipeline.process_events()

    task_run = await prefect_client.read_task_run(task_run_id)
    assert task_run.id == task_run_id

Now you can run the task in your test. And then manually execute a version of the events processes that would normally happen on your sever. In the future EventsPipeline can also be used at teardown time of the ephemeral api, so that events and tasks can be recorded.

@jakekaplan jakekaplan marked this pull request as ready for review July 25, 2024 20:31
@jakekaplan jakekaplan requested a review from cicdw as a code owner July 25, 2024 20:31
messages.append(message)
return messages

async def process_messages(self, messages: List[MemoryMessage]):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future when shutting down the ephemeral version we can invoke this method on everything sitting inside of the in-memory events queue

Copy link

codspeed-hq bot commented Jul 25, 2024

CodSpeed Performance Report

Merging #14765 will not alter performance

Comparing run-events-pipeline (115f6ee) with main (b7cca3d)

Summary

✅ 5 untouched benchmarks

Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nits, makes sense to me

src/prefect/events/clients.py Show resolved Hide resolved
src/prefect/testing/fixtures.py Outdated Show resolved Hide resolved
@jakekaplan jakekaplan merged commit 334859f into main Jul 26, 2024
30 checks passed
@jakekaplan jakekaplan deleted the run-events-pipeline branch July 26, 2024 02:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants