Skip to content

Latest commit

 

History

History
executable file
·
39 lines (36 loc) · 2.54 KB

notes.md

File metadata and controls

executable file
·
39 lines (36 loc) · 2.54 KB
Raw

Discussion

  • Moderation: The weight from moderators tell us how much we should trust the feature (information) from body image or part (head/hands) crops, the animation figure shows the visualization of the weights: the lighter color means it trust body more.
  • Face from SMPL-X model: In PIXIE, we use a common shape variable for the body and the face. The shape space of SMPL-X is learned from 3800 real full body scans and captures the correlation between body and face shape. In the samples script we support the option of predicting the shape of the full body from a face-only image. If you are only interested in getting accurate face shape, we suggest to try other face-specifc work, such as DECA.
  • Model conversion: If you want to convert the predicted SMPL-X body to a different body model, e.g. SMPL, please take a look here

Limitations:

  • Cropping matters: Even though we already did a lot of data augmentations durining training, the results will still vary a bit due to cropping differences for the input images. For simplicity, we chose Faster-RCNN as a person detector, while in the paper we used OpenPose keypoints to compute a bounding box.
  • Perspective projection: We use a scaled-orthographic/weak-perspective camera model in PIXIE, which does not work well for images with strong perpective distortion.
  • Misalignment issue: For regression works that output model (SMPL/SMPL-X) parameters, misalignment with the person in the image is always an issue. This can be improved by using PIXIE results as initialization for an optimization method like SMPLify-X to refine the pose. Note that, the moderator weight could also be utilized as a confidence measure during optimization.
  • Speed: The main bottleneck of PIXIE is the need for three separate encoders for the body, head and hand images. Changing the current backbone, i.e. Resnet50 or HRNet, to a lighter one, like MobileNet, should accelerate inference at the cost of performance. We will attempt to provide different options when we release the training code.
  • Texture: Similar to DECA, we rely upon the Basel face model for our albedo space. Its lack of ethnic diversity in the albedo causes the model to often compensate for skin tone with lighting.