This repository contains Jupyter notebooks and more extensive projects that illustrate various methods for loading data into different destinations (e.g. Weaviate database) using the dlt library.
To run the notebooks, you will need credentials for the tools being used. They are added to the .dlt folder. For instance, if you're working on a Weaviate notebook, you will have to add Weaviate credentials. Refer to the notebooks to find out which credentials are needed.
- schema_evolution.ipynb: shows how you can alert schema changes to slack.
- spotlight_demo.ipynb: shows how to get data from APIs, files, Python objects and move it into a local or remote database. Demo was created for a Data Talks Club: Open-Source Spotlight project.
- Pyladies 12.11.2024: Similar to Spotlight demo, shows dlt basics, demonstrates how to get data from APIs (GitHub and PokeAPI), database, JSON, and move it into a Duckdb.
- pdf_to_weaviate.ipynb: shows how to load data from PDF files, specifically invoices, into Weaviate.
- sql_to_weaviate.ipynb: shows how to import data from a public MySQL database into Weaviate.
- zendesk_to_weaviate.ipynb: loads data from a Zendesk dlt source into Weaviate.
- personio_demo.ipynb: shows how to load data from Personio to duckDB.
Project demos are more extensive compared to the notebook ones and have their own README files. Refer to each project for more details.
This repository is licensed under the Apache License 2.0. Please refer to the LICENSE.txt file for more details.
Happy coding and data loading! 🚀📊
