Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flows which train a model and run inference from the trained model #4

Open
wants to merge 34 commits into
base: main
Choose a base branch
from

Conversation

daniel-prefect
Copy link

@daniel-prefect daniel-prefect commented Dec 17, 2024

Related to https://linear.app/prefect/issue/DOC-118/create-an-author-tutorial-for-model-training

Merge this before PrefectHQ/prefect#16632

Here's the full workflow that these flows enable:

  1. Upload training data to an S3 bucket, this triggers a webhook event that triggers a Prefect automation
  2. The Prefect automation runs a training flow (model_training.py), which uses Sagemaker to fit an XGBoost model to the training data, and then uploads the fitted model to a second S3 bucket.
  3. When a model is uploaded to the second S3 bucket, that bucket triggers a webhook event that triggers a second Prefect automation.
  4. The second Prefect automation runs an inference flow (model_inference.py), which loads the fitted model from S3 and generates predictions for a small set of test values.

NOTE: The code for using XGBoost with the Sagemaker Python SDK is based on these docs: https://sagemaker.readthedocs.io/en/stable/frameworks/xgboost/using_xgboost.html

@daniel-prefect daniel-prefect self-assigned this Dec 17, 2024
@daniel-prefect daniel-prefect changed the title Train model from s3 data Add flows which train a model and run inference from the trained model Dec 19, 2024
@daniel-prefect daniel-prefect marked this pull request as ready for review January 8, 2025 18:45
@daniel-prefect daniel-prefect force-pushed the train_model_from_s3_data branch from 488fd6f to 48a4c68 Compare January 8, 2025 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants