What to document about Camera Input: hands

[zepumph]

@terracoda and I are finding that we are asking quite often what information will need to be given about camera input to our users. We will add stuff here as it comes up.

• Hidden under cameraInput=hands
• Load time warning (couple second lag on startup)
• Single hand use
• Only supported for the hands interaction
• showVideo can help to calibrate where you are interacting with the webcam.
• Options in the input preferences
• O hand gesture + usage with voicing.
• Not supported offline

[zepumph]

Tagging @emily-phet and @BLFiedler if they have any other items to communicate.

[emily-phet]

Potentially - tips or nuances related to practical use scenarios, such as:

• An explicit indicator that there are two likely usage scenarios - the vertical and horizontal hand movement.
• If there are any likely collaborative scenarios that don't work well. i can't recall if two people standing next to each other can each contribute a hand or not...I think perhaps they can if their other hands aren't picked up on...?

[terracoda]

Yes, two people can each hide a hand use their shown hand to operate the sim.

[brettfiedler]

I think we considered restricting handedness via built in MediaPipe, but did not implement it, so either participant can contribute either of their hands. I believe it just looks for whichever hand is left/right-most.

• showVideo can help to calibrate where you are interacting with the webcam.

^^ Just adding some clarification on what calibration means in bullet list below.

Also adding point about performance on low-spec devices.

Updated list:

• Hidden under cameraInput=hands
• Load time warning (couple second lag on startup)
• Single-hand use
• Only supported for the hands interaction (not other sim interactions)
• showVideo can help to calibrate where you are interacting with the webcam.
  • Calibration means setting the range you can move hands up and down by adjusting distance between person and webcam.
• Options in the input preferences
• O hand gesture + usage with voicing.
• Not supported offline
• An explicit indicator that there are two likely usage scenarios - the vertical and horizontal hand movement.
• Collaborative scenarios with up to two hands in the detection window (no handedness, any two hands)
• Performance on different platforms (low-spec CPUs and GPUs will introduce delay/lag, so movements need to be slower)

[brettfiedler]

Where would we like to document this? Parts of this should find their way into the teacher tips, but only points relevant to how to integrate into a teacher's classroom (so the collaborative scenarios is a good point and what priority the sim makes of hands in view, as well as performance issues)

[terracoda]

Should we have a tip sheet explaining all of the above linked in the teacher tips?

[zepumph]

Where would we like to document this?

No strong preference, it seems like something that should be supplementary to the sim, so teacher tips, the research page, maybe both? A blog post somewhere by one of us about our "exciting new research"?

[emily-phet]

I think we should do a few things:

1. make a very short video (similar to the sound primer") that goes with this sim. That way any other place we provide info can refer to the video, simplifying some of the text.
2. A short section in the teacher tips with a bullet list that mentions all of these things. Since it's a new feature I think all of these are relevant, and if we keep it short and sweet and refer to the video I don't think it needs to overwhelm the teacher tips doc.
3. An email announcement when the sim is published that announces the Voicing feature, and as a slightly less emphasized but still present mention of the computer vision feature, with a reference to the video and teacher tips.

@BLFiedler - we can chat about the best plan for the video - the new video person may be able to be involved.

[zepumph]

All the above sounds good! This is not blocking but should probably move forward as we continue towards publicaion on this sim.

[zepumph]

We also want to update voicing (though it doesn't exactly relate to this issue).

• Noting that [safari] Voicing/SpeechSynthesis is buggy in multiple iframes utterance-queue#92 exists, and safari + multiple voicing sims in iframes behaves tremendously poorly.

[zepumph]

Over in #527 we realized that a list of camera input features that we don't currently have implemented seems prudent for if we ever come back to this issue. For now, I'll just put it in this comment:

We punted on this, but for next time we are in RAP:

• Voicing stops when only one hand is used with ?cameraInput=hands #527: Voicing + one-handed interaction.
• Support with LoL? Camera Input doesn't work with LoL #531

[brettfiedler]

RaP teacher tips needs to be updated with this. I'll tackle this tomorrow or next week at the latest

[terracoda]

Unless they are already compete, I can take on items 2 and 3 in #512 (comment)

[terracoda]

@BLFiedler, please re-assign to me if you would like me to take care of items 2 and 3.

[brettfiedler]

This is what I have for the teacher tips for RaP right now. It's certainly not short, but RaPs teacher tips are pretty long anyway. If we'd like to add anything to the website to describe the feature I can move some of this information.

image text:

Camera Input: Hands

• If a camera (e.g, webcam) is available on your device, you can enable a feature to detect your own hand positions to move the hands in the Play Area. This feature leverages the computer vision capabilities of the open-source MediaPipe model. It does not collect any data/images.
• Add the query parameter ?cameraInputs=hands to the end of simulation URL.
• The feature requires two hands to be present in the detection window. This can be from two individuals for cooperative exploration!
• Default settings assume the camera is facing you. If your camera is placed elsewhere (e.g., above) or you feel the direction of simulation hand movement does not match your own, see additional options in the Input tab of the Preferences menu.
• This feature interacts with the Voicing feature. If Voicing is enabled, you can silence the current voiced statement by bringing your index finger and thumb together.

Tips and Disclaimers (underlined)

• The simulation will take longer to load the necessary resources for realtime hand detection.
• This feature is resource intensive and may have poor performance on some devices.
• This feature does not work offline.
• The vertical range of the simulation hands are mapped to the upper and lower bounds of your camera window. Try backing away from your camera for more range of motion. Add the query parameter &showVideo to the simulation URL for a resizable camera preview for troubleshooting.

[terracoda]

This looks great!
I have one addition for the last bullet:

• This feature interacts with the Voicing feature. If Voicing is enabled, you can silence the current voiced statement by bringing your index finger and thumb together, forming an O-like gesture with your fingers.

[terracoda]

I can see how all that text will look on the prototypes page.

[terracoda]

Ok, I added some of this to the website. I think we will still need all the details in the Teacher Tips.

[brettfiedler]

Teacher tips updated and uploaded to sim page and google drive.

Linking this to the media update issue to accomplish documentation of Camera Input: Hands on the main page.

phetsims/a11y-research#181