Skip to content

Commit

Permalink
update README (wip)
Browse files Browse the repository at this point in the history
  • Loading branch information
@LarsKue
LarsKue committed Dec 29, 2023
1 parent 3fc107c commit 71ef9f2
Showing 1 changed file with 54 additions and 31 deletions.
85 changes: 54 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,24 +14,26 @@ This repository contains parts of the code used in my master's thesis, titled
Our approach to context-aware learning is defined in the thesis as
deep learning using an embedding from a set of context inputs:

<p align="center">
<img src="docs/context-aware-learning.png" alt="Context-Aware Learning"/>
</p>
<div class="row">
<div class="col">
<img src="docs/context-aware-learning.webp" alt="Context-Aware Learning">
</div>
</div>


In this repo, we use [Optimal Transport Flow Matching](https://arxiv.org/abs/2302.00482) to leverage this
embedding and learn a generative model that can be conditioned on sampled
context embeddings, thus enabling interpolation between contexts:

<p align="center">
<img src="docs/context-aware-flow-matching.png" alt="Flow Matching" width=500 />
</p>
<div class="row">
<div class="col">
<img src="docs/context-aware-flow-matching.webp" alt="Interpolation">
</div>
</div>

## 2. Install

### 2.1 Using Conda

Create a new environment with the required dependencies:
Create a new conda environment with the required dependencies:
```bash
conda env create -f env.yaml
```
Expand All @@ -52,6 +54,11 @@ If you want to plot samples using blender, install the blender env instead:
conda env create -f blender.yaml
```

Activate and verify as above.

Note that these environments are incompatible with each other,
because they each require different python versions.

## 3. Experiments

Experiment notebooks can be found in the `experiments` folder. We use
Expand All @@ -60,35 +67,45 @@ to train our models. Each notebook contains the hyperparameters used for trainin

### 3.1 ModelNet10

Dataset: [ModelNet10](https://3dvision.princeton.edu/projects/2014/3DShapeNets/)

Samples from the dataset:

<p align="center">
<img src="docs/modelnet10/data_samples.png" alt="ModelNet10" width=500 />
</p>
Dataset: [ModelNet10](https://3dvision.princeton.edu/projects/2014/3DShapeNets/)

Samples from the trained model:
Left: Samples from the dataset. Right: Samples from the trained model.

<p align="center">
<img src="docs/modelnet10/model_samples.png" alt="ModelNet10" width=500 />
</p>
<div class="row">
<div class="col">
<img src="docs/modelnet10/data_samples.webp" alt="ModelNet10 Dataset Samples">
</div>
<div class="col">
<img src="docs/modelnet10/model_samples.webp" alt="ModelNet10 Model Samples">
</div>
</div>

We can interpolate between contexts:

<p align="center">
<img src="docs/modelnet10/interpolation.png" alt="ModelNet10" width=500 />
</p>
<div class="row">
<div class="col">
<img src="docs/modelnet10/interpolation.webp" alt="ModelNet10 Context Interpolation">
</div>
</div>

The shape manifold is particularly visible in this video,
where we rotate points in the latent space of the flow:

<p align="center">
<video width="500" autoplay loop playsinline>
<source src="docs/modelnet10/rotation.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</p>
<div class="row">
<div class="col">
<img src="docs/modelnet10/rotation.webp" alt="ModelNet10 Rotation">
</div>
</div>

The model is competitive with other state-of-the-art models:

<div class="row">
<div class="col">
<img src="docs/modelnet10/metrics.webp" alt="ModelNet10 Accuracy">
</div>
</div>


### 3.2 LIDAR-CS
Expand All @@ -97,10 +114,16 @@ where we rotate points in the latent space of the flow:

Dataset: [LIDAR-CS](https://github.com/LiDAR-Perception/LiDAR-CS)

Samples from the trained model:
Left: Samples from the dataset. Right: Samples from the trained model.

<div class="row">
<div class="col">
<img src="docs/lidar-cs/data_samples.png" alt="LIDAR-CS Dataset Samples">
</div>
<div class="col">
<img src="docs/lidar-cs/model_samples.png" alt="LIDAR-CS Model Samples">
</div>
</div>

<p align="center">
<img src="docs/lidar-cs/samples.png" alt="LIDAR-CS" width=500 />
</p>


0 comments on commit 71ef9f2

Please sign in to comment.