MossVLN: Memory-Observation Synergistic Planning for Continuous Vision-Language Navigation

Data Preparation

Scenes: Matterport3D

Instructions copied from VLN-CE:

Matterport3D (MP3D) scene reconstructions are used. The official Matterport3D download script (download_mp.py) can be accessed by following the instructions on their project webpage. The scene data can then be downloaded:

# requires running with python 2.7
python download_mp.py --task habitat -o data/scene_datasets/mp3d/

Extract such that it has the form scene_datasets/mp3d/{scene}/{scene}.glb. There should be 90 scenes. Place the scene_datasets folder in data/.

Data and Trained Weights

• Waypoint Predictor: data/wp_pred/check_cwp_bestdist*

  • For R2R-CE, data/wp_pred/check_cwp_bestdist_hfov90 [link].

• Processed data, pre-trained weight, fine-tuned weight [link].

  unzip MossVLN.zip    # file/fold structure has been organized

overall, files and folds are organized as follows:

MossVLN
├── data
│   ├── datasets
│   ├── logs
│   ├── scene_datasets
│   └── wp_pred
└── pretrained
    └── Moss

Setup

Installation

Follow the Habitat Installation Guide to install habitat-lab and habitat-sim. We use version v0.1.7 in our experiments, same as in the VLN-CE, please refer to the VLN-CE page for more details. In brief:

1.Create a virtual environment. We develop this project with Python 3.6.

conda env create -f environment.yaml

2.Install habitat-sim for a machine with multiple GPUs or without an attached display (i.e. a cluster):

conda install -c aihabitat -c conda-forge habitat-sim=0.1.7 headless

3.Clone this repository and install all requirements for habitat-lab, VLN-CE and our experiments. Note that we specify gym==0.21.0 because its latest version is not compatible with habitat-lab-v0.1.7.

git clone git@github.com:OpenMICG/MossVLN.git
cd MossVLN
python -m pip install -r requirements.txt
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

4.Clone a stable habitat-lab version from the github repository and install. The command below will install the core of Habitat Lab as well as the habitat_baselines.

git clone --branch v0.1.7 git@github.com:facebookresearch/habitat-lab.git
cd habitat-lab
python setup.py develop --all # install habitat and habitat_baselines

Running

Finetuning and Evaluation

Use main.bash for Training/Evaluation/Inference with a single GPU or with multiple GPUs on a single node. Simply adjust the arguments of the bash scripts:

# for R2R-CE
CUDA_VISIBLE_DEVICES=0 bash run_r2r/main.bash train 2333  # training
CUDA_VISIBLE_DEVICES=0 bash run_r2r/main.bash eval  2333  # evaluation
CUDA_VISIBLE_DEVICES=0 bash run_r2r/main.bash infer 2333  # inference

Acknowledge

Our implementations are partially inspired by ETPNav , CWP and DUET.

Thanks for their great works!