Issue with MediaPipe Pose: Incorrect Hand Landmarks for Occluded Hands in Multi-Person Scenarios

[Ramprasath1997]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Web & IOS

MediaPipe Tasks SDK version

0.10.0

Task name (e.g. Image classification, Gesture recognition etc.)

pose recognition

Programming Language and version (e.g. C++, Python, Java)

Javascript

Describe the actual behavior

When using MediaPipe Pose to track multiple people, if one person hides their hand (e.g., by placing it behind their back or out of the camera's view), the system continues to generate hand landmarks for that person. These generated landmarks do not correspond to any actual hand position and appear to be "hallucinated" or incorrect.

Describe the expected behaviour

When one person hides their hand (e.g., by placing it behind their back or out of the camera's view), MediaPipe Pose should not generate any hand landmarks for that person. The system should accurately reflect the current state of visible hands, ensuring that only the hands that are actually present in the camera view are tracked.

Standalone code/steps you may have used to try to get what you need

I attempted to implement custom logic to check for hand visibility by analyzing the landmark positions. However, the model still generates hallucinated landmarks for the hidden hand, which does not reflect the actual state of visibility.

Other info / Complete Logs

If Person A raises their hand while Person B hides their hand (e.g., behind their back), MediaPipe Pose continues to generate hand landmarks for Person B, despite their hand being occluded.
The generated landmarks for Person B do not correspond to any real hand position and appear to be hallucinated, often aligning incorrectly with Person A's hand or floating in space.
This behavior can lead to inaccurate gesture recognition and confusion in applications that depend on precise hand tracking.

When one person hides their hand (e.g., by placing it behind their back or out of the camera's view), MediaPipe Pose should not generate any hand landmarks for that person.
The system should accurately reflect the current state of visible hands, ensuring that only the hands that are actually present in the camera view are tracked.

[kuaashish]

Hi @Ramprasath1997,

Currently, this issue is outside the scope of the MediaPipe Pose Landmarker, as it can not detect occlusions. A similar issue #4868 has already been reported. However, we are considering this as a feature request and will share it with our team for potential future implementation.

Thank you!!

[kuaashish]

Hi @Ramprasath1997,

Could you please share the complete standalone code you are using, as outlined in our documentation? We will look into issue. Alternatively, you can refer to the documentation for further details. We will check this with our team, but we cannot commit to a specific timeline at this moment. We will provide an update as soon as we have more information.

Thank you!!