Mariana Aguiar, Victor Alves, Jan Egger, Jianning Li and Christina Gsaxner
This project is the result of Mariana Aguiar ([email protected]), Victor Alves ([email protected]) and Jan Egger ([email protected]) work, having been developed as part of Mariana Aguiar's master thesis in Biomedical Engineering, Medical Informatics Branch from University of Minho.
The evolution of modern medicine and the technologies of implant design and manufacturing have allowed the number of cranioplasties performed to increase greatly, improving the quality of life of patients who have to undergo cranial reconstruction surgery. The need to generate cranial implants adapted to the individuality of each human skull as quickly and efficiently as possible has led in recent years to the search for methods that are as automated as possible and do not compromise the quality and integrity of the implant.
The most traditional and popular method in recent years has been to use CAD/CAM technologies, in which patient-specific cranial implants can be reliably generated. However, the process is time-consuming, and the available software has high associated costs that require skilled professionals for implant design, not allowing it to be available in all clinical settings. Due to the growing interest and success of the application of Deep Learning to 3D reconstruction, recently several works have been proposed on the application of these architectures to cranial reconstruction, allowing to obtain high-quality cranial implants automatically, not only reducing the cost of the process but also enabling its availability in a larger number of clinical contexts.
The work proposed in this thesis to automatically reconstruct a defected skull uses as its basis the Mesh R-CNN framework by Gkioxari et al. (2019) - https://github.com/facebookresearch/meshrcnn. A similar approach was proposed to explore an end-to-end method to perform skull reconstruction based on the CT images of the patient and output a high-resolution 3D cranial implant.
The dataset used was part of the AutoImplant Cranial Implant Design Challenge (https://autoimplant.grand-challenge.org/) provided by Jan Egger and Jianning Li, consisting of 200 pairs of healthy and artificially defective skulls. Since our system combines 2D perception with 3D prediction, it was necessary to generate from the NRRD the following data: a skull mesh, a voxelized skull, a 2D image from the skull and the camera parameters that allowed obtaining that image, the skull bounding box that enclose the skull shape, and the skull binary mask.
The meshes were obtained through the application of the Lewiner Marching Cubes algorithm that searches for the skull surfaces and generates a skull triangle mesh. The voxelization of the skull was performed with the use of a voxelizer tool (https://www.patrickmin.com/binvox/) that rasterizes both the surface and internal voxels into a binary voxel grid. The 2D images of the skull were taken using Meshlab (https://www.meshlab.net/), where the camera parameters for each image were also annotated. It’s important to note here that although the software gave us these camera parameters, at training time we had to use ideal camera parameters since the parameters provided by the software completely misaligned the 2D images with the 3D models. The skull mask generation consisted of a simple thresholding operation, and the skull bounding boxes was calculated by finding minimum and maximum coordinates in the x and y axis of visible points in the skull.
Even though all the required data was previously acquired and prepared, a dataset registration was necessary for the data to be compatible with the Detectron2 framework. The chosen standard representation for the dataset was COCO (Common Objects in Context) format, a well-known format for instance detection. The format consists of a list of dictionaries, where each dictionary contains the paths to the data, camera parameters information and other information such as the format of the bounding boxes, the category label and how many objects exist in the image.
After creating the dataset, we registered it into the DatasetCatalog of Detectron2 for both the training and testing data. The training and testing configurations of the DL networks of the system combine Detectron2’s default configurations well known for its results, and configurations defined by us to customize them into our context, which are merged and freezed at training time.
After registering the skull dataset in Detectron2’s DatasetCatalog and configuring the training and testing parameters of the network, it’s time to load the data from the annotations to be fed into the network. Since the network needs to consume the actual meshes, images, masks, and bboxes and not their paths, the Detectron2 framework offers a class called DatasetMapper that not only loads the data but also allows the user to perform data augmentation. The DatasetMapper maps each of the element from the previously registered annotations and loads the data into memory, apply some transformations to the data and transforms their corresponding annotations to maintain their alignment, and then converts the annotations into an Instances object, the Detectron2 ground truth structure object of the framework.
The proposed system integrates different neural networks with different aims divided into two main stages: a voxel stage followed by a mesh refinement stage. First, the voxel stage inputs the 2D skull image into a backbone feature extractor network CNN that outputs a feature map for the given image. Then these feature maps are inputted in a Region Proposal Network (RPN) to generate multiple ROIs where the output is classified as either having or not the object. Then an RoI Align layer is applied to extract a small feature map from each RoI, pixel-wise aligning the extracted features with the input image to preserve the spatial locations of each object. The resulting reduced feature maps serve as input for four parallel network branches: (i) a box detection branch, (ii) a mask branch, (iii) a Z depth extent branch, and (iv) a voxel branch. The first network predict a bbox and the label for each skull, the second a mask for each skull, the third network the depth extent of the skull object in the image, and the fourth network predicts a binary voxel grid for each skull.
The output of the voxel branch is then fed into an algorithm, named Cubify, that replaces each occupied voxel with a cuboid triangle mesh, generating a coarse 3D skull mesh. Since the 3D mesh representing each skull based on its 2D image has a coarse shape, a mesh refinement stage is applied to convert the low-resolution mesh into a high-resolution mesh. This stage refines the vertex positions by inputting the 3D skull mesh three times through mesh refinement stages. Each mesh refinement stage is comprised of three operations: vert align that extracts the 2D skull image features for the mesh vertices, propagating the acquired information through a series of graph convolutions blocks, and then the vertex positions are updated by a vertex refinement operation.
The methods implemented in this thesis to build a 3D system able to perform cranial reconstruction involved working with different libraries and frameworks.The below figure illustrates an overview of the computational resources and for what stage of the development each tool was applied.