Digital Twins for Stone Masonry Walls (dtsmw - Tested on Ubuntu LTS 18.04 - 64GB RAM)

This repository contains code for creating geometrical digital twins for stone masonry walls using artificial intelligence, specifically computer vision and machine learning. This includes 3D registration algorithms based on information extracted from image features and SfM that use non-linear least squares. The full description of the methodology can be found in the paper Image based digital twinning for stone masonry walls by Pantoja-Rosero et. al, (2022).

How to use it?

1. Clone repository

Clone repository in your local machine. All codes related with method are inside the src directory.

2. Download data

Example input data can be downloaded from Data and Data. The first contains the input data necessary to reproduce the example 1 published in the paper. Please create a folder data/ inside the repository folder and inside this folder extract the .zip file given in the link. The second link contains the image data and the input file to run SfM pipeline from meshroom.

As alternative a temporary link is given. Data can be downloaded from Data. Extract the folder data/ and place it inside the repository folder. If the link is not available, please contact the fist author of the paper.

2a. Repository directory

The repository directory should look as:

dt_smw
└───src
└───data

3. Environment

Create a conda environment and install python packages. At the terminal in the repository location.

conda create -n dt_sme python==3.8

conda activate dt_sme

pip install -r requirements.txt

4. Using main method

The main function of the methodology which is in charge of creating the digital twin for stone masonry walls is placed in the file main.py. Test it with one of the examples provided as follows:

python p5_dt_ex_4-1_synthetic.py

We provide in total three examples: p5_dt_ex_4-1_synthetic.py, p5_dt_ex_4-3_dry.py and p5_dt_ex_4-4_mortar.py.

5. Creating your own digital twin

The methodology requires as input the output of the photogrametry pipeline by the openc source meshroom software. To check the files required please refer to one of the examples data located in data folder. Create a python file (your_twin.py) following the content of one of the provided example files example_....py. Then run it similarly as in the previous step.

python your_twin.py

5.1. Additional meshroom params

When running the Meshroom pipeline considere the next:

• Graph: Add two nodes ConvertSfMFormat comming from StructureFromMotion node. The Meshroom's graph should look as:

• The fist ConvertSfMFormat node should output a json file with all the SfM information. The node configurations should be like:

• The second ConvertSfMFormat node should output a .ply file with just the structure of SfM. The node configurations should be like:

• For feature extraction we recoment use sift and akaze descriptors with desnsity and quality high as the next image. The codes are meant to use sift and/or akaze feeatures. If more features are detected, the SfM reconstruction might improve but the extra features are ignored during the computation of transformation matrix T for registration.

5.2. SfM command line with meshroom

We provide a script which will run the SfM pipeline necessary to run our algorithm. For this follow the next:

a) Download the Meshroom binary file, renamed as Meshroom and place it inside the src\ folder as follows:

dt_smw
└───src
  └───Meshroom
    └───aliceVision
    └───lib
    └───qtPlugins
    └───CHANGES.md
    └───COPYING.md
    └───LICENSE-MPL2.md
    └───Meshroom
    └───meshroom_batch
    └───meshroom_compute
    └───README.md

b) Create a folder named your_twin inside data/ folder with your image data with the following structure:

dt_smw
└───data
  └───your_twin
    └───layers
      └───layer_00
        └───images
      └───layer_01
        └───images
      ...
      └───layer_xx
        └───images
   └───stones
      └───stone_00
        └───images
      └───stone_01
        └───images
      ...
      └───stone_xx
        └───images

c) Inside the file src/run_meshroom_full_dataset.py change the name of the data_folder to your_twin and save the file

d) Being at the src folder with the dt_smw environment actived run:

python run_meshroom_full_dataset.py

6. Results

The results will be saved inside results folder. This contain a domain.json file with the transformation matrices to register src (stones or layers) to dst (layers o wall) models. If flags activated, folders for registered point cloud, meshes and/or textured meshes are created. Take into account that the src 3D models have to be placed inside the respective data folder.

6.a Final repository directory

The repository directory after runing the medothology looks as:

dt_smw
└───src
└───data
  └───your_twin
└───results
  └───your_twin
    └───domain.json
    └───point_cloud
    └───textured_mesh
    └───mesh