Update code to ICCV version

valeoai · Sep 21, 2023 · 1790897 · 1790897
1 parent 4b83d98
commit 1790897
Show file tree

Hide file tree

Showing 17 changed files with 551 additions and 412 deletions.
diff --git a/README.md b/README.md
@@ -2,45 +2,47 @@
 
 ![](./illustration.png)
 
-[**Using a Waffle Iron for Automotive Point Cloud Semantic Segmentation**]()  
+[**Using a Waffle Iron for Automotive Point Cloud Semantic Segmentation**](http://arxiv.org/abs/2301.10100)  
 [*Gilles Puy*<sup>1</sup>](https://sites.google.com/site/puygilles/home),
 [*Alexandre Boulch*<sup>1</sup>](http://boulch.eu),
 [*Renaud Marlet*<sup>1,2</sup>](http://imagine.enpc.fr/~marletr/)  
 <sup>1</sup>*valeo.ai, France* and <sup>2</sup>*LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France*.
 
 If you find this code or work useful, please cite the following [paper](http://arxiv.org/abs/2301.10100):
 ```
-@article{puy23waffleiron,
+@inproceedings{puy23waffleiron,
   title={Using a Waffle Iron for Automotive Point Cloud Semantic Segmentation},
   author={Puy, Gilles and Boulch, Alexandre and Marlet, Renaud},
-  journal={arxiv:2301.10100}
+  booktitle={ICCV},
   year={2023}
 }
 ```
 
+## Updates
+
+This work was accepted at ICCV23. The code and trained models were updated on September 29, 2023 to reproduce the scores in the published version. 
+
+If you need to access the preliminary trained models you can refer to this [section](#Preliminary-version). Note that those preliminary models are not performing as well as those used in the published version.
+
+
 ## Installation
 
+Setup the environment and clone this repo:
 ```
 pip install pyaml==6.0 tqdm=4.63.0 scipy==1.8.0 torch==1.11.0 tensorboard=2.8.0
 git clone https://github.com/valeoai/WaffleIron
 cd WaffleIron
 pip install -e ./
 ```
 
-Download the pretrained models:
+Download the trained models:
 ```
 wget https://github.com/valeoai/WaffleIron/files/10294733/info_datasets.tar.gz
 tar -xvzf info_datasets.tar.gz
-wget https://github.com/valeoai/WaffleIron/files/10294734/pretrained_nuscenes.tar.gz
-tar -xvzf pretrained_nuscenes.tar.gz
-wget https://github.com/valeoai/WaffleIron/files/10294735/pretrained_kitti.tar.gz
-tar -xvzf pretrained_kitti.tar.gz
-```
-
-Finally, indicate where the nuScenes and SemanticKITTI datasets are located on your system:
-```
-export PATH_NUSCENES="/PATH/TO/NUSCENES"
-export PATH_KITTI="/PATH/TO/KITTI/"
+wget [UPDATE]
+tar -xvzf waffleiron_nuscenes.tar.gz
+wget [UPDATE]
+tar -xvzf waffleiron_kitti.tar.gz
 ```
 
 If you want to uninstall this package, type `pip uninstall waffleiron`.
@@ -50,67 +52,64 @@ If you want to uninstall this package, type `pip uninstall waffleiron`.
 
 ### Option 1: Using this code
 
-To evaluate the pre-trained model on the train set of nuScenes used in Table 1 of our paper, type
+To evaluate the nuScenes trained model, type
 ```
 python launch_train.py \
 --dataset nuscenes \
---path_dataset $PATH_NUSCENES \
---log_path ./pretrained_models/WaffleIron-48-256__60cm-baseline-nuscenes/ \
---config ./configs/WaffleIron-48-256__60cm-baseline-nuscenes.yaml \
+--path_dataset /path/to/nuscenes/ \
+--log_path ./pretrained_models/WaffleIron-48-384__nuscenes/ \
+--config ./configs/WaffleIron-48-384__nuscenes.yaml \
 --fp16 \
---gpu 0 \
 --restart \
 --eval
 ```
+This should give you a final mIoU of 77.6%.
 
-To evaluate the pre-trained model on the train set of SemanticKITTI, with instance cutmix augmentation, type
+To evaluate the SemanticKITTI trained model, type
 ```
 python launch_train.py \
 --dataset semantic_kitti \
---path_dataset $PATH_KITTI \
---log_path ./pretrained_models/WaffleIron-48-256__40cm-BEV-cutmix-kitti/ \
---config ./configs/WaffleIron-48-256__40cm-BEV-cutmix-kitti.yaml \
+--path_dataset /path/to/kitti/ \
+--log_path ./pretrained_models/WaffleIron-48-256__kitti/ \
+--config ./configs/WaffleIron-48-256__kitti.yaml \
 --fp16 \
 --restart \
 --eval
 ```
+This should give you a final mIoU of 68.0%.
+
+**Remark:** *On SemanticKITTI, the code above will extract object instances on the train set (despite this being not necessary for validation) because this augmentation is activated for training on this dataset (and this code re-use the training script). This can be bypassed by editing the `yaml` config file and changing the entry `instance_cutmix` to `False`. The instances are saved automatically in `/tmp/semantic_kitti_instances/`.*
 
-**Remark:** *On SemanticKITTI, the code above will extract object instances on the train set (despite this 
-being not necessary for validation) because this augmentation is activated for training on this dataset (and this code
-re-use the training script). This can be bypassed by editing the `yaml` config file and changing the entry 
-`instance_cutmix` to `False`. The instances are saved automatically in `/tmp/semantic_kitti_instances/`.*
 
 ### Option 2: Using the official APIs
 
-The second option writes the predictions on disk and the results can be computed using the official 
-nuScenes or SemanticKITTI APIs. This option also allows you to perform test time augmentations, which is not possible 
-with Option 1 above. These scripts should be useable for submission of the official benchmarks.
+The second option writes the predictions on disk and the results can be computed using the official nuScenes or SemanticKITTI APIs. This option also allows you to perform test time augmentations, which is not possible with Option 1 above. These scripts should be useable for submission of the official benchmarks.
 
 #### nuScenes
 
-To extract the prediction with the pre-trained model on nuScenes, type
+To extract the prediction with the model trained on nuScenes, type
 ```
 python eval_nuscenes.py \
---path_dataset $PATH_NUSCENES \
---config ./configs/WaffleIron-48-256__60cm-baseline-nuscenes.yaml \
---ckpt ./pretrained_models/WaffleIron-48-256__60cm-baseline-nuscenes/ckpt_last.pth \
+--path_dataset /path/to/nuscenes/ \
+--config ./configs/WaffleIron-48-384__nuscenes.yaml \
+--ckpt ./pretrained_models/WaffleIron-48-384__nuscenes/ckpt_last.pth \
 --result_folder ./predictions_nuscenes \
 --phase val \
 --num_workers 12
 ```
 or, if you want to use, e.g., 10 votes with test time augmentations,
 ```
 python eval_nuscenes.py \
---path_dataset $PATH_NUSCENES \
---config ./configs/WaffleIron-48-256__60cm-baseline-nuscenes.yaml \
---ckpt ./pretrained_models/WaffleIron-48-256__60cm-baseline-nuscenes/ckpt_last.pth \
+--path_dataset /path/to/nuscenes/ \
+--config ./configs/WaffleIron-48-384__nuscenes.yaml \
+--ckpt ./pretrained_models/WaffleIron-48-384__nuscenes/ckpt_last.pth \
 --result_folder ./predictions_nuscenes \
 --phase val \
 --num_workers 12 \
 --num_votes 10 \
---batch_size 5
+--batch_size 10
 ```
-You can reduce `batch_size` to 2 or 1 depending on the available memory.
+You can reduce `batch_size` to 5, 2 or 1 depending on the available memory.
 
 These predictions can be evaluated using the official nuScenes API as follows
 ```
@@ -119,18 +118,18 @@ python nuscenes-devkit/python-sdk/nuscenes/eval/lidarseg/evaluate.py \
 --result_path ./predictions_nuscenes \
 --eval_set val \
 --version v1.0-trainval \
---dataroot $PATH_NUSCENES \
+--dataroot /path/to/nuscenes/ \
 --verbose True  
 ```
 
 #### SemanticKITTI
 
-To evaluate the pre-trained model on SemanticKITTI, type
+To extract the prediction with the model trained on SemanticKITTI, type
 ```
 python eval_kitti.py \
---path_dataset $PATH_KITTI \
---ckpt ./pretrained_models/WaffleIron-48-256__40cm-BEV-cutmix-kitti/ckpt_last.pth \
---config ./configs/WaffleIron-48-256__40cm-BEV-cutmix-kitti.yaml \
+--path_dataset /path/to/kitti/ \
+--ckpt ./pretrained_models/WaffleIron-48-256__kitti/ckpt_last.pth \
+--config ./configs/WaffleIron-48-256__kitti.yaml \
 --result_folder ./predictions_kitti \
 --phase val \
 --num_workers 12
@@ -141,7 +140,7 @@ The predictions can be evaluated using the official APIs by typing
 git clone https://github.com/PRBonn/semantic-kitti-api.git
 cd semantic-kitti-api/
 python evaluate_semantics.py \
---dataset $PATH_KITTI/dataset \
+--dataset /path/to/kitti//dataset \
 --predictions ../predictions_kitti \
 --split valid
 ```
@@ -150,72 +149,67 @@ python evaluate_semantics.py \
 
 ### nuScenes
 
-To train a WaffleIron-48-256 backbone on nuScenes with 
-- 2D cells of 60 cm,
-- the baseline sequence of projections along the z-axis, then the y-axis, then the x-axis, etc., until the last layer,
-
-type
+To retrain the WaffleIron-48-384 backbone on nuScenes type
 ```
 python launch_train.py \
 --dataset nuscenes \
---path_dataset $PATH_NUSCENES \
---log_path ./logs/WaffleIron-48-256__60cm-baseline-nuscenes/ \
---config ./configs/WaffleIron-48-256__60cm-baseline-nuscenes.yaml \
---gpu 0 \
+--path_dataset /path/to/nuscenes/ \
+--log_path ./logs/WaffleIron-48-384__nuscenes/ \
+--config ./configs/WaffleIron-48-384__nuscenes.yaml \
+--multiprocessing-distributed \
 --fp16
 ```
 
-For example, with `--seed 1` as additional arguments in `launch_train.py`, I obtain 76.2 % in mIoU at the last 
-training epoch (using one Nvidia Tesla V100S-PCIE-32GB for training).
+We used the checkpoint at the *last* training epoch to report the results.
 
-Note: for multi-GPUs training, you can remove `--gpu 0` and the code will use all available GPUs using PyTorch DataParallel 
-for parallelism. You can add the argument `--multiprocessing-distributed` to use DistributedDataParallel instead.
+Note: for single-GPU training, you can remove `--multiprocessing-distributed` and add the argument `--gpu 0`.
 
 
 ### SemanticKITTI
 
-To retrain a WaffleIron-48-256 backbone on SemanticKITTI with
-- 2D cells of 40 cm,
-- projection along the z-axis at all layers,
-- **instance cutmix augmentations**,
-
-type
+To retrain the WaffleIron-48-256 backbone, type
 ```
 python launch_train.py \
 --dataset semantic_kitti \
---path_dataset $PATH_KITTI \
---log_path ./logs/WaffleIron-48-256__40cm-BEV-cutmix-kitti \
---config ./configs/WaffleIron-48-256__40cm-BEV-cutmix-kitti.yaml \
---fp16 \
---multiprocessing-distributed
+--path_dataset /path/to/kitti/ \
+--log_path ./logs/WaffleIron-48-256__kitti \
+--config ./configs/WaffleIron-48-256__kitti.yaml \
+--multiprocessing-distributed \
+--fp16
 ```
 
-The instances for cutmix augmentation are saved in `/tmp/semantic_kitti_instances/`. You can disable the instance 
-cutmix augmentations by editing the `yaml` config file to set `instance_cutmix` to `False`.
+At the beginning of the training, the instances for cutmix augmentation are saved in `/tmp/semantic_kitti_instances/`. If this process is interrupted before completion, please delete `/tmp/semantic_kitti_instances/` and relaunch training. You can disable the instance cutmix augmentations by editing the `yaml` config file to set `instance_cutmix` to `False`.
 
-For submission to the official benchmark on the test set of SemanticKITTI, we also trained the network on both the 
-val and train sets (argument `--trainval` in `launch_train.py`), used the checkpoint at the last epoch and 10 test 
-time augmentations during inference.
+For submission to the official benchmark on the test set of SemanticKITTI, we trained the network on both the val and train sets (argument `--trainval` in `launch_train.py`), used the checkpoint at the last epoch and 12 test time augmentations during inference.
 
 
 ## Creating your own network
 
-### Config file
-
-You can refer to `./config/WaffleIron-template.yaml` where we describe the role of each parameter. 
-In particular, you can adjust `nb_channels` and `depth` to increase of decrease the capacity of WaffleIron.
-You can also adjust the memory required to train a network by adjusting `max_points` in `dataloader`, but a 
-too small value might impact the performance.
-
 ### Models
 
 The WaffleIron backbone is defined in `waffleiron/backbone.py` and can be imported in your project by typing
 ```python
 from waffleiron import WaffleIron
 ```
-It needs to be combined with a embedding layer to provide point tokens and a pointwise classification layer, as we do 
-in `waffleiron/segmenter.py`. You can define your own embedding and classification layers instead.
+It needs to be combined with a embedding layer to provide point tokens and a pointwise classification layer, as we do in `waffleiron/segmenter.py`. You can define your own embedding and classification layers instead.
+
 
+## Preliminary version
+
+To access the preliminary trained models and the corresponding code, you can clone version v0.1.1 of the code.
+```
+git clone -b v0.1.1 https://github.com/valeoai/WaffleIron
+cd WaffleIron/
+pip install -e ./
+```
+
+The corresponding pretrained models are available at:
+```
+wget https://github.com/valeoai/WaffleIron/files/10294734/pretrained_nuscenes.tar.gz
+tar -xvzf pretrained_nuscenes.tar.gz
+wget https://github.com/valeoai/WaffleIron/files/10294735/pretrained_kitti.tar.gz
+tar -xvzf pretrained_kitti.tar.gz
+```
 
 ## Acknowledgements
 We thank the authors of 

diff --git a/...leIron-48-256__40cm-BEV-cutmix-kitti.yaml → configs/WaffleIron-48-256__kitti.yaml b/...leIron-48-256__40cm-BEV-cutmix-kitti.yaml → configs/WaffleIron-48-256__kitti.yaml
@@ -1,51 +1,57 @@
-waffleiron: # Architecture of the backbone
-  nb_channels: 256  # Define F = the feature size = width of the WaffleIron
-  depth: 48         # Define L = the depth on the network
-  fov_xyz:          # Define the FOV in meters
-    - - -50 # min value on x-axis: -50 m
-      - -50 # min value on y-axis: -50 m
-      - -5  # min value on z-axis: -5 m
-    - - 50  # max value on x-axis: 50 m
-      - 50  # max value on y-axis: 50 m
-      - 3   # max value on z-axis: 5 m
-  dim_proj:         # Define the sequence of projection (which is then repeated sequentially until \ell = L)
-    - 2  # Project along the z axis at \ell = 1 (and then the same at all layer)
-  grids_size:       # Define here the size of the 2D grids
-    - [250, 250] # At \ell = 1, project along z, ie on (x, y) with FOV [-50, 50] on both axes: size [250, 250] -> resolution 40cm
-
-classif: # Architecture of the classifcation layer, after WaffleIron
-  nb_class: 19      # Number of classes on nuscenes (after removing the ignore class)
-
-embedding: # Architecture of the embedding layer, before WaffleIron
-  input_feat:       # List of features on each point
-    - "intensity"
-    - "height"
-    - "radius"
-  size_input: 3     # Input feature size on each point
-  neighbors: 16     # Neighborhood for embedding layer
-  voxel_size: 0.1   # Voxel size for downsampling point cloud in pre-processing
-
-dataloader:
-  batch_size: 4
-  num_workers: 12
-  max_points: 20000
-
-augmentations:
-  rotation_z: null
-  flip_xy: null
-  scale:
-    - [0, 1, 2]
-    - 0.1
-  instance_cutmix: True
-
-loss:
-  lovasz: 1.0
-
-optim:
-  lr: .001
-  weight_decay: 0.003
-
-scheduler:
-  min_lr: 0.00001
-  max_epoch: 45
-  epoch_warmup: 4
+waffleiron: # Architecture of the backbone
+  nb_channels: 256  # Define F = the feature size = width of the WaffleIron
+  depth: 48         # Define L = the depth on the network
+  fov_xyz:          # Define the FOV in meters
+    - - -50 # min value on x-axis: -50 m
+      - -50 # min value on y-axis: -50 m
+      - -3  # min value on z-axis: -5 m
+    - - 50  # max value on x-axis: 50 m
+      - 50  # max value on y-axis: 50 m
+      - 2   # max value on z-axis: 5 m
+  dim_proj:         # Define the sequence of projection (which is then repeated sequentially until \ell = L)
+    - 2  # Project along the z axis at \ell = 1 (and then the same at all layer)
+    - 1  # At \ell = 2, project along y
+    - 0  # At \ell = 1, project along x
+  grids_size:       # Define here the size of the 2D grids
+    - [250, 250]    
+    - [250,  12]    
+    - [250,  12]    
+  drop: 0.2
+
+classif: # Architecture of the classifcation layer, after WaffleIron
+  nb_class: 19      # Number of classes on nuscenes (after removing the ignore class)
+
+embedding: # Architecture of the embedding layer, before WaffleIron
+  input_feat:       # List of features on each point
+    - "intensity"
+    - "xyz"
+    - "radius"
+  size_input: 5     # Input feature size on each point
+  neighbors: 16     # Neighborhood for embedding layer
+  voxel_size: 0.1   # Voxel size for downsampling point cloud in pre-processing
+
+dataloader:
+  batch_size: 4
+  num_workers: 12
+  max_points: 20000
+
+augmentations:
+  rotation: 
+    - [2, 6]
+  flip_xy: null
+  scale:
+    - [4, 5, 6, 7]
+    - 0.1
+  instance_cutmix: True # Will apply Cutmix *and* Polarmix
+
+loss:
+  lovasz: 1.0
+
+optim:
+  lr: .001
+  weight_decay: 0.003
+
+scheduler:
+  min_lr: 0.00001
+  max_epoch: 45
+  epoch_warmup: 4