[ICCV'23 Oral] Unmasking Anomalies in Road-Scene Segmentation

[arXiv]

Mask2Anomaly.mp4

Run our demo using Colab:

Installation

Please follow the Installation Instruction to set up the codebase.

Datasets

We have three different sets of dataset used for training, ood-fine-tuning, and anomaly inference. Please follow the below steps to set-up each set.

• Inlier Dataset(Cityscapes/Streethazard): consists of only inlier classes that can be prepared by following the same structure as given here.
• Outlier Supervision Dataset(MS-COCO): helps fine-tune the model pre-trained on the inlier dataset on ood-objects. The outlier dataset is created by using this script and then changing the cfg.MODEL.MASK_FORMER.ANOMALY_FILEPATH accordingly.
• Anomaly Dataset (validation): can be downloaded using this link. Please unzip the file and place it preferably in the dataset folder.

Training and Inference

• The training of the model takes place in two stages:

  • [Stage1] Training of inlier dataset performed using the command: Cityscapes Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file configs/cityscapes/semantic-segmentation/anomaly_train.yaml. Streethazard Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file 'configs/streethazard/streethazard_training.yaml'.
  • [Stage2] We fine-tune the weights of the model using following command: Cityscapes Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file configs/cityscapes/semantic-segmentation/anomaly_ft.yaml. Streethazard Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file 'configs/streethazard/streethazard_ft.yaml' .

• During inference, we use the final-weights obtained after fine-tuning: CUDA_VISIBLE_DEVICES=1 python anomaly_utils/anomaly_inference.py --input '/path-to-images-files/*.jpg' --config-file '/path-to-anomaly-inference-config/anomaly_inference.yaml'

• We provide all the commands for training, ood-fine-tune, and anomaly inference in run.sh and corresponding config files at /configs/cityscapes /semantic-segmentation/.

• To perform anomaly segmentation using pre-trained models, download the model from shared Google Drive link and then change the model weight path in /configs/cityscapes/semantic-segmentation/anomaly_inference.yaml.

  • Single Pretained model for SMIYC, Fishyscapes, LostAndFound, and Road Anomaly: Link
  • StreetHazard: Link

Docker Image

• We provide a singularity image similar to docker that provides anomaly output without needing library installation/GPUs.
• Install Singularity following the instructions. Link
• Download the .sif image from link
• Run the command singularity run --bind {input-path-to-image-datset}:/input,{output-path-to-save-segmentation-maps}:/output mask2former.sif

License

Shield:

The majority of Mask2Anomaly is licensed under a MIT License.

However portions of the project are available under separate license terms: Mask2Former and Swin-Transformer-Semantic-Segmentation is licensed under the MIT license, Deformable-DETR is licensed under the Apache-2.0 License.

Acknowledgement

We thank the authors of the codebases mentioned below, which helped build the repository.

• Meta-OOD
• PEBEL
• Mask2Former