-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Expanded the README.md to include a detailed project overview, features, setup instructions, and author information. - Added new images to the assets directory for project showcase. - Updated the GitHub Actions workflow to ensure consistent tag formatting for Docker image builds. - Improved setup instructions for local development, including clearer steps for configuring MinIO and GitHub Actions secrets. This commit significantly improves documentation and workflow clarity for contributors and users.
- Loading branch information
1 parent
8b4995e
commit 795eb20
Showing
8 changed files
with
83 additions
and
36 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,40 @@ | ||
<a name="readme-top"></a> | ||
|
||
<div align="center"> | ||
|
||
<a href="" target="_blank" title="Go to website"> | ||
<img width="196px" alt="Document My Project" src="assets/images/icon.png"> | ||
</a> | ||
|
||
# Movie Recommendation MLOps Project | ||
|
||
System of recommendation of movies based on the user's profile and preferences. | ||
|
||
</div> | ||
<div align="center"><h4><a href="#-about-the-project">ℹ️ About the Project</a> • <a href="#-showcase">🏞 Showcase</a> • <a href="#-features">⭐️ Features</a> • <a href="#-stack-tech">🛠 Stack Tech</a> • <a href="#-setup">⚙ ️Setup</a> • <a href="#-from-here-have-fun">🎉 Have fun!</a> • <a href="#-about-the-authors">👨🏻 About the Authors</a> • <a href="#-license">📖 License</a></h4></div> | ||
|
||
## ℹ️ About the Project | ||
|
||
This project is a starting pack for MLOps projects focused on the subject of "movie recommendation". It provides a structured framework to develop, train, and deploy machine learning models for recommending movies to users. It uses Supabase for the backend, Airflow for the orchestration, MLflow for the tracking, Minio for the storage of the models, Prometheus and Grafana for the monitoring. | ||
|
||
|
||
## 🏞 Showcase | ||
|
||
<p align="center"><img src="assets/images/dmp_1.png" alt="Main Image"/></p> | ||
|
||
<center> | ||
<table> | ||
<tr> | ||
<td><a href="assets/images/dmp_s_1.png"><img width="320" height="200" src="assets/images/dmp_s_1.png"></a></td> | ||
<td><a href="assets/images/dmp_s_2.png"><img width="320" height="200" src="assets/images/dmp_s_2.png"></a></td> | ||
</tr> | ||
<tr> | ||
<td><a href="assets/images/dmp_s_3.jpeg"><img width="320" height="200" src="assets/images/dmp_s_3.jpeg"></a></td> | ||
<td><a href="assets/images/dmp_s_4.png"><img width="320" height="200" src="assets/images/dmp_s_4.png"></a></td> | ||
</tr> | ||
</table> | ||
</center> | ||
|
||
## Project Organization | ||
|
||
The project is organized as follows: | ||
|
@@ -124,7 +157,17 @@ The project is organized as follows: | |
└── requirements-ref.txt | ||
``` | ||
|
||
## Tools Used | ||
## ⭐️ Features | ||
|
||
- Very basic model to recommend movies. | ||
- Scraping new movies from The Movie Database (TMDB) API. | ||
- Training a machine learning model to recommend movies. | ||
- Deploying the model as a REST API. | ||
- Building an interactive web app to display recommendations. | ||
- Orchestrating the workflow with Apache Airflow. | ||
- Monitoring the system with Prometheus and Grafana. | ||
|
||
## 🛠 Stack Tech | ||
|
||
- **Python**: The main programming language used for data processing, model training, and prediction. | ||
- **Docker & Docker Compose**: Used for containerizing the application and setting up a local development environment. | ||
|
@@ -136,36 +179,19 @@ The project is organized as follows: | |
- **Prometheus**: For monitoring and alerting. | ||
- **Grafana**: For visualizing metrics. | ||
|
||
## Before setting up the project | ||
## ⚙ ️Setup | ||
|
||
Make sure you have the following tools installed: | ||
- Docker | ||
Docker, Docker Compose, Python 3.10+, pip, git et make | ||
```bash | ||
docker --version | ||
``` | ||
- Docker Compose | ||
```bash | ||
docker compose version | ||
``` | ||
- Python 3.10+ | ||
```bash | ||
python --version | ||
``` | ||
- pip | ||
```bash | ||
pip --version | ||
``` | ||
- git | ||
```bash | ||
git --version | ||
``` | ||
- make | ||
```bash | ||
make --version | ||
``` | ||
|
||
## Setting Up the Project for Local Development | ||
|
||
To set up the project for local development, from the root of the repository follow the steps below: | ||
|
||
1. Run the `make setup1` command. | ||
|
@@ -177,19 +203,23 @@ To set up the project for local development, from the root of the repository fol | |
4. Run the `make start` command. | ||
|
||
5. Setup access and the bucket for MLFlow: | ||
- From your browser, go to `localhost:9001` to access the MinIO console. Sign in using the root user and password specified in the `.env` file. | ||
- Once you sign in, navigate to the Access Keys tab and click the Create access key button. This will take you to the Create Access Key page. | ||
- Your access key and secret key are not saved until you click the Create button. Do not navigate away from this page until this is done. Don’t worry about copying the keys from this screen. Once you click the Create button, you will be given the option to download the keys to your file system (in a JSON file). | ||
- Next, create a bucket named `mlflow`. This is straightforward, go into the Buckets tab and click the `Create Bucket` button. | ||
- Once you have your keys and you have created your bucket, you can finish setting up the services by stopping the containers, updating the `.env` file with your access and secret keys, and then restarting the containers. The command below will stop and remove your containers. | ||
- Access MinIO console at `localhost:9001` and sign in with root credentials from `.env` | ||
- Create access key and save the generated keys | ||
- Create a bucket named `mlflow` | ||
- Update `.env` with your access/secret keys and restart the containers: | ||
```bash | ||
cd mlflow | ||
docker compose down | ||
docker compose --env-file ../.env up -d --build | ||
``` | ||
(may be useful to do it in a new terminal to avoid cache issues with the `.env` file) | ||
|
||
**Access the Services**: | ||
6. Secrets for the GitHub Actions workflow: | ||
- In order to push the images to the Docker Hub registry, you need to create a personal access token with the necessary permissions. | ||
- Add the token (read and write) for the Docker Hub registry as a secret with the name `DOCKERHUB_TOKEN`. | ||
- Add the username for the Docker Hub registry as a secret with the name `DOCKERHUB_USERNAME`. | ||
|
||
|
||
**Local access to the Services**: | ||
- Supabase: [http://localhost:8000](http://localhost:8000) | ||
- Airflow: [http://localhost:8080](http://localhost:8080) | ||
- Streamlit: [http://localhost:8501](http://localhost:8501) | ||
|
@@ -199,22 +229,39 @@ To set up the project for local development, from the root of the repository fol | |
- Prometheus: [http://localhost:9090](http://localhost:9090) | ||
- Grafana: [http://localhost:3000](http://localhost:3000) | ||
|
||
## 🎉 From here have fun! | ||
|
||
## From here have fun! | ||
|
||
### First recommendations | ||
#### First recommendations | ||
You can open the Streamlit app, create an account and have your first recommendations! | ||
|
||
In order for the app to work you need to use an email with a number just before the @ symbol (ex: `[email protected]`, `[email protected]`, etc.). It'll allow you to have an existing profil linked to your account. | ||
### Run the DAGs | ||
#### Run the DAGs | ||
You can run the DAGs in Airflow to scrape new movies and train the model. | ||
### Analyse the model in MLFlow | ||
#### Analyse the model in MLFlow | ||
You can explore the artefacts, metrics and logs of the model in MLFlow if you runned the training DAG. | ||
### Explore Prometheus and Grafana | ||
#### Explore Prometheus and Grafana | ||
You can explore the metrics in Grafana's dashboard. | ||
|
||
### Explore Supabase | ||
#### Explore Supabase | ||
You can explore the database in Supabase's dashboard. | ||
## 👨🏻 About the Authors | ||
**Sarah Hemmel** | ||
**Mikhael Benilouz** | ||
**Antoine Pelamourgues** | ||
## 📖 License | ||
This project is licensed under the [MIT License](https://opensource.org/licenses/MIT). | ||
<p align="right"><a href="#readme-top">Top ⬆️</a></p> | ||
--- | ||
<div align="center">Built with ❤️</div> |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.