diff --git a/.github/workflows/build-and-push.yml b/.github/workflows/build-and-push.yml index f80386a..8ec54b7 100644 --- a/.github/workflows/build-and-push.yml +++ b/.github/workflows/build-and-push.yml @@ -3,7 +3,7 @@ name: Build and Push Docker Images on: push: tags: - - 'v*.*.*' + - "v*.*.*" release: types: [published] workflow_dispatch: @@ -44,4 +44,4 @@ jobs: tags: | ${{ secrets.DOCKERHUB_USERNAME }}/streamlit:latest ${{ secrets.DOCKERHUB_USERNAME }}/streamlit:${{ github.sha }} - ${{ secrets.DOCKERHUB_USERNAME }}/streamlit:${{ github.ref_name }} + ${{ secrets.DOCKERHUB_USERNAME }}/streamlit:${{ github.ref_name }} diff --git a/README.md b/README.md index efec9a6..3cced73 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,40 @@ + + +
+ + +Document My Project + + # Movie Recommendation MLOps Project +System of recommendation of movies based on the user's profile and preferences. + +
+

ℹ️ About the Project β€’ 🏞 Showcase β€’ ⭐️ Features β€’ πŸ›  Stack Tech β€’ βš™ ️Setup β€’ πŸŽ‰ Have fun! β€’ πŸ‘¨πŸ»β€ About the Authors β€’ πŸ“– License

+ +## ℹ️ About the Project + This project is a starting pack for MLOps projects focused on the subject of "movie recommendation". It provides a structured framework to develop, train, and deploy machine learning models for recommending movies to users. It uses Supabase for the backend, Airflow for the orchestration, MLflow for the tracking, Minio for the storage of the models, Prometheus and Grafana for the monitoring. + +## 🏞 Showcase + +

Main Image

+ +
+ + + + + + + + + +
+
+ ## Project Organization The project is organized as follows: @@ -124,7 +157,17 @@ The project is organized as follows: └── requirements-ref.txt ``` -## Tools Used +## ⭐️ Features + +- Very basic model to recommend movies. +- Scraping new movies from The Movie Database (TMDB) API. +- Training a machine learning model to recommend movies. +- Deploying the model as a REST API. +- Building an interactive web app to display recommendations. +- Orchestrating the workflow with Apache Airflow. +- Monitoring the system with Prometheus and Grafana. + +## πŸ›  Stack Tech - **Python**: The main programming language used for data processing, model training, and prediction. - **Docker & Docker Compose**: Used for containerizing the application and setting up a local development environment. @@ -136,36 +179,19 @@ The project is organized as follows: - **Prometheus**: For monitoring and alerting. - **Grafana**: For visualizing metrics. -## Before setting up the project +## βš™ ️Setup Make sure you have the following tools installed: -- Docker +Docker, Docker Compose, Python 3.10+, pip, git et make ```bash docker --version - ``` -- Docker Compose - ```bash docker compose version - ``` -- Python 3.10+ - ```bash python --version - ``` -- pip - ```bash pip --version - ``` -- git - ```bash git --version - ``` -- make - ```bash make --version ``` -## Setting Up the Project for Local Development - To set up the project for local development, from the root of the repository follow the steps below: 1. Run the `make setup1` command. @@ -177,19 +203,23 @@ To set up the project for local development, from the root of the repository fol 4. Run the `make start` command. 5. Setup access and the bucket for MLFlow: - - From your browser, go to `localhost:9001` to access the MinIO console. Sign in using the root user and password specified in the `.env` file. - - Once you sign in, navigate to the Access Keys tab and click the Create access key button. This will take you to the Create Access Key page. - - Your access key and secret key are not saved until you click the Create button. Do not navigate away from this page until this is done. Don’t worry about copying the keys from this screen. Once you click the Create button, you will be given the option to download the keys to your file system (in a JSON file). - - Next, create a bucket named `mlflow`. This is straightforward, go into the Buckets tab and click the `Create Bucket` button. - - Once you have your keys and you have created your bucket, you can finish setting up the services by stopping the containers, updating the `.env` file with your access and secret keys, and then restarting the containers. The command below will stop and remove your containers. + - Access MinIO console at `localhost:9001` and sign in with root credentials from `.env` + - Create access key and save the generated keys + - Create a bucket named `mlflow` + - Update `.env` with your access/secret keys and restart the containers: ```bash cd mlflow docker compose down docker compose --env-file ../.env up -d --build ``` - (may be useful to do it in a new terminal to avoid cache issues with the `.env` file) -**Access the Services**: +6. Secrets for the GitHub Actions workflow: + - In order to push the images to the Docker Hub registry, you need to create a personal access token with the necessary permissions. + - Add the token (read and write) for the Docker Hub registry as a secret with the name `DOCKERHUB_TOKEN`. + - Add the username for the Docker Hub registry as a secret with the name `DOCKERHUB_USERNAME`. + + +**Local access to the Services**: - Supabase: [http://localhost:8000](http://localhost:8000) - Airflow: [http://localhost:8080](http://localhost:8080) - Streamlit: [http://localhost:8501](http://localhost:8501) @@ -199,22 +229,39 @@ To set up the project for local development, from the root of the repository fol - Prometheus: [http://localhost:9090](http://localhost:9090) - Grafana: [http://localhost:3000](http://localhost:3000) +## πŸŽ‰ From here have fun! -## From here have fun! - -### First recommendations +#### First recommendations You can open the Streamlit app, create an account and have your first recommendations! In order for the app to work you need to use an email with a number just before the @ symbol (ex: `user1@example.com`, `user20@example.com`, etc.). It'll allow you to have an existing profil linked to your account. -### Run the DAGs +#### Run the DAGs You can run the DAGs in Airflow to scrape new movies and train the model. -### Analyse the model in MLFlow +#### Analyse the model in MLFlow You can explore the artefacts, metrics and logs of the model in MLFlow if you runned the training DAG. -### Explore Prometheus and Grafana +#### Explore Prometheus and Grafana You can explore the metrics in Grafana's dashboard. -### Explore Supabase +#### Explore Supabase You can explore the database in Supabase's dashboard. + +## πŸ‘¨πŸ»β€ About the Authors + +**Sarah Hemmel** + +**Mikhael Benilouz** + +**Antoine Pelamourgues** + +## πŸ“– License + +This project is licensed under the [MIT License](https://opensource.org/licenses/MIT). + + +

Top ⬆️

+ +--- +
Built with ❀️
diff --git a/assets/images/dmp_1.png b/assets/images/dmp_1.png new file mode 100644 index 0000000..8235bfd Binary files /dev/null and b/assets/images/dmp_1.png differ diff --git a/assets/images/dmp_s_1.png b/assets/images/dmp_s_1.png new file mode 100644 index 0000000..9237cd0 Binary files /dev/null and b/assets/images/dmp_s_1.png differ diff --git a/assets/images/dmp_s_2.png b/assets/images/dmp_s_2.png new file mode 100644 index 0000000..fe29b45 Binary files /dev/null and b/assets/images/dmp_s_2.png differ diff --git a/assets/images/dmp_s_3.jpeg b/assets/images/dmp_s_3.jpeg new file mode 100644 index 0000000..c841e0c Binary files /dev/null and b/assets/images/dmp_s_3.jpeg differ diff --git a/assets/images/dmp_s_4.png b/assets/images/dmp_s_4.png new file mode 100644 index 0000000..b6c4459 Binary files /dev/null and b/assets/images/dmp_s_4.png differ diff --git a/assets/images/icon.png b/assets/images/icon.png new file mode 100644 index 0000000..d909ca8 Binary files /dev/null and b/assets/images/icon.png differ