Dataset Processing - Add support for groups, articulated scenes, episodes.

[0mdc]

Motivation and Context

Motivations:

• We need to support articulated scenes in Unity.
• HSSD is massive. In WebGL, we only want to download assets we need, not include the entire dataset with the WebGL app.
• The unit of work is an episode set. It captures all dependencies for a "project", along with all paths.

Context:

• The unity_dataset_processing.py script is used to convert/decimate Habitat datasets into a format usable by Unity.
• The premise is that Unity has a clone of the data folder. A relative path from the data folder resolves both in Habitat and Unity (or any other external engine, like Blender).

This changeset entirely refactors the Habitat -> Unity dataset processing pipeline such as:

• An episode set is supplied as input instead of a list of scenes.
• Datasets are automatically gathered (not hard-coded).
• Articulated scenes are supported.
• A metadata.json file is produced, containing groups.
  • Each group contains the list of assets they depend on. Currently, groups match 1-1 with the concept of a scene.
  • This file is consumed by external asset pipelines (e.g. Unity) to determine how assets should be packaged.
  • The default "local" group hints that the asset should be packaged locally (along with the build).
  • Grouped assets hint that they will be used together and should be batched when downloading from a remote location.
• Refactoring. This will enable the dataset pipeline to be driven by hydra config.
  • This will be a future refactoring that will allow each "project" to define how to process this data.
  • Decimation level, included datasets and other parameters will be configurable per-dataset via config.
  • More target engines may be included (e.g. Blender).

How Has This Been Tested

Tested with existing episode sets and new articulated scenes.

Types of changes

• [Refactoring]
• [Development]

Checklist

• My code follows the code style of this project.
• I have updated the documentation if required.
• I have read the CONTRIBUTING document.
• I have completed my CLA (see CONTRIBUTING)
• I have added tests to cover my changes if required.

[aclegg3]

This seems functional for now and an improvement over existing process, so approving.

However, I do wonder if we could make this more generic by using MetadataMediator to load the SceneDataset and then enumerate the contents instead of directly querying the configs. I know that scene_instance access would be troubling, but the resulting config values would be more accurate since there is an override and defaults hierarchy built in.

Interested to get @jturner65's opinion on this.

[0mdc]

Great point @aclegg3 !
Beware that the magnum dependency used in this library requires all batteries to be included, therefore we direct users to create a separate environment from habitat. The magnum included with habitat_sim does not include things like basis compression.

It is possible to arrange the script differently, but I'm concerned about complicating its usage.