Useful Python Packages

Useful Python Packages

This page summarizes a variety of useful python packages with a link to the package as well as a short description of what it is useful for.

---

1. Data Analysis

1. Pandas profiling: Automatically generates standard data exploration analysis for pandas dataframes (e.g. columns, missing values, histograms, ...) with just one line of code. You can interact with the results direclty in jupyter notebooks or export them as html report. Great for a first look at your data before diving into further analyses.

2. Experimental Logging

1. Weights and Biases: Provides a framework to track compare and visualize ML experiments. Integrates with all major ML frameworks.
2. Sacred: General framework to configure, organize, log and reproduce computational experiments (i.e. also physical simulations, not just ML models). Stores all configurations, codefiles and experimental results automatically in a Databse. Can be visualized by using Scared Board.
3. Neptune: TODO

3. Managing Configurations

1. Hydra: Allows you to dynamically create a hierarchical configuration. It also allows you to override defaults through config files and the command line. It also allows you to run mutliple runs with different configurations from a simple command line call. Under the hood it creates configuration objects which are OmegaConf DictConfigs (see below).
2. OmegaConf: YAML based hierarchical configuration system that allows to merge configurations from multiple sources (e.g. files, CLI arugments, environment variables, ...) and creates a consistent API regardless of how the config was generated.

4. Visualization

1. Folium: For creating interactive maps in jupyter notebooks and exporting them to html. Convenient if you have to quickly and interactively visualize spatial data.
2. matplotlib and seaborn
3. ipygany Third dimension
4. ipympl renders matplotlib interactive
5. ipyleaflet for interactive maps in Jupyter notebooks
6. geemap for interactive plotting based on ipyleaflet and Google Earth Engine
7. voila turns Jupyter notebooks into standalone web applications
8. tqdm endlessly customisable progress bars for loops that may be faster than print statements

5. Debugging

1. Interactive Python Debugger: Allows you to walk through your code interactively instead of having to "print-debug". Also works nicely in notebooks. For additional syntax highlighting look at ipdb.