Alternative Input support

[pixelzoom]

From 4/13/21 Q2 planning meeting:

Investigate adding keyboard navigation for the 1.0 release.

Milestones:
start = 4/1/21
end = 5/15/21

Desig doc: Fourier: Making Waves - Interactive Description Design

[pixelzoom]

4/9/21 Zoom discussion with @jbphet:

• Keyboard navigation mainly involves specify an order in which Nodes will be traversed.

• It's unlikely that the default traversal order is what we will want.

• Specifying traversal order involves creating an array of Nodes, where the order of the array determines the order in which Nodes will be traversed.

• Traversal order should be considered when making implementation decisions. Depending on the structure of the code, it may be easy or difficult to create the traversal array.

4/12/21 Zoom discussion with @arouinfar:

• Some common UI components likely lack a11y design. For example, is traversal implemented for keypad?

• Sim-specific UI components will need an a11y design. For example, how should we treat an Amplitude slider + NumberDisplay? As one component? As separate components?

• Fourier is more than 1/2 way through implementation, and the code has been implemented with no consideration (or knowledge) of keyboard navigation requirements. It's possible that big changes will be needed.

• Are we too far along in the 1.0 release process to do this?

• Let's discuss at design meeting.

[pixelzoom]

4/15/21 design meeting:

@Kathy would like to proceed with:

• Include only interactive UI components in the traversal order
• Ignore the amplitude NumberDisplays and keypad

Start with the Discrete screen.

Wave Game is going to be a little challenging. Components from vegas (level selection, reward dialog,...) and will need to be instrumented.

Next steps:

• Specify the traversal order. @arouinfar will create the associated design document and do a quick draft. Then we'll have a sim-team design meeting to discuss.
• Identify common-code UI components that need work (e.g. vegas for Wave Game), assign and prioritize.

[pixelzoom]

4/22/21 design meeting @kathy-phet @arouinfar @jessegreenberg @pixelzoom

Add supportsInteractiveDescription=true to play around with what's already supported for keyboard navigation. Traversal order defaults to scenegraph order.

@arouinfar is going to put all of this in the appropriate design document.

---

Discrete screen

Traversal order:

• Amplitude sliders
• Control panel top to bottom
• Measurement tools
• Charts
• Play/Pause
• Reset All

Keyboard navigation options for amplitude sliders need to be added. @arouinfar will specify.

Measurement tools will need keyboard navigation. Look at ISLCRulerNode (used in coulombs-law) for example of 2-step grab and drag (seems cumbersome). The other alternative is tab to tool, then move it around. @arouinfar will discuss with @terracoda.

Info dialog needs some work to set focus to close button when it opens.

---

Wave Game screen

@arouinfar will spec this out.

Next button is a question. Should we move focus to Next button automatically? (It's a bit rude and not a best practice for web accessibility.) Keyboard shortcut for Next button?

Reward dialog: start focus with "Keep Going" button, then "New Level", then "X' close button.

[pixelzoom]

I'm guessing (but not entirely certain) that we'll want to specify these options for the amplitude sliders. Here they are, shown with their defaults as they appear in AccessibleValueHandler.js:

          // keyboard steps for various keys/interactions
          keyboardStep: ( rangeProperty.get().max - rangeProperty.get().min ) / 20,
          shiftKeyboardStep: ( rangeProperty.get().max - rangeProperty.get().min ) / 100,
          pageKeyboardStep: ( rangeProperty.get().max - rangeProperty.get().min ) / 10,

shiftKeyboardStep is finer grain while the Shift key is pressed.
pageKeyboardStep is coarser grain while PageUp or PageDown is pressed.

[pixelzoom]

Slack post to a11y channel:

Chris Malley 3:08 PM
Fourier: Making Waves will support keyboard navigation for the 1.0 release, see #53. We had a design meeting last Thursday, and I have a rough idea of the desired traversal order. (@arouinfar will be doing a full spec.) Can someone point me in the right direction for implementing this feature? Who to talk to, examples, documentation, etc. Thanks!

Sam Reid 3:14 PM
I implemented limited keyboard support for Waves Intro in phetsims/wave-interference#412, with help from @zepumph. The first 2 commits took us a long way toward the goal:
phetsims/wave-interference@b54e4ae
phetsims/waves-intro@b20a94e

Michael Kauzmann 3:16 PM
I would say the best doc would be https://github.com/phetsims/phet-info/blob/master/doc/interactive-description-technical-guide.md, which alt input is a subset of interactive description. Most of everything down through the section "Add alternative-input input listeners" will be helpful, you can ignore stuff about "UtteranceQueue" and alerts. It will have you start by understanding the PDOM over in http://localhost:8080/scenery/doc/accessibility.html.

Chris Malley 3:17 PM
Thanks. Any exemplars that I should refer to? Or would that be Waves Intro?

Michael Kauzmann 3:18 PM
Best/most recent alt input would probably be Molecules and Light, and Ratio and Proportion. I haven't looked at Waves Intro personally, but I bet it's got the goods too.

[pixelzoom]

We have missed the 5/15/21 milestone. I have not investigated how to add keyboard navigation, or whether we have sim-specific structural problems that will make it difficult to assemble the desired order.

We are currently waiting for @arouinfar to produce the design document, and discuss how to handle the measurement tools with @terracoda.

[terracoda]

For the record, @arouinfar and @terracoda met on May 3rd, 2021.

I shared with @arouinfar why we have a two-step drag interaction for some draggable objects like the balloons in BASE, the Chemistry book in Friction, and the Bar Magnet in FL. And why we don't for the Both Hands interaction in Ratio and Proportion.

[pixelzoom]

Thanks for the update @terracoda. That info wasn't passed on to me, and I don't see it mentioned here. Can @terracoda or @arouinfar please summarize what you discussed, and which interaction is appropriate for this sim?

[terracoda]

Sure, the balloon, book and magnet objects are the main interactions for the sim. If a learner cannot explore using that object, they cannot use the sim for anything useful.

Browser support for Web Standards has improved, and continues to do so, but when I designed BASE, aria-roledescription, which is very useful for custom interactions, was not well-supported.

I had no ability as a designer to make the Yellow Balloon interaction sound interesting and intuitive to move as a straight-up custom interaction. When focus was moved to the balloon, screen reader users would hear something like this:

Yellow Balloon, web application, grabbable, group with 2 items, Use W, A, S, and D keys to drag balloon.

Not fun, not engaging, not intuitive, not usable!

A button is a very intuitive interaction. Upon focus, hearing:

Grab Yellow Balloon, button

is very intuitive. And once grabbed and in Application Mode, the user hears further instructions (appropriate to device platform) on how to move the balloon.

This pattern works great for screen reader users, but doesn't work for visual keyboard users because the balloon does not look like a button. We then designed the visual hint to support visual keyboard users. The interaction is very intuitive for everyone once it is learned. It usually takes one go at it to learn it, so the hints go away until the sim is reset.

For Ratio and Proportion
Two native HTML sliders precede the custom Both Hands interaction, and browser support for aria-roledescription is now good. Interviews with blind users showed us that blind users could easily operate Both Hands using the custom bi-manual input. No grab or activation button needed for the Both Hands interaction.

The Rulers in CL and GFLB
Note that in Coulomb's Law, our first sim with Alternative Input that was without Interactive Description, the ruler has no grab button. You can move it with the arrow keys on focus.

In Gravity Force Lab, however, which has Interactive Description, the ruler is a grab then drag interaction, similar to the balloon. This ruler, however, is not a primary interaction. We had to do extra work in the interactive description design to make screen users understand that it was not needed for productive exploration. The ruler's name in GFL is "Measure Distance Ruler" instead of "Grab Ruler". And the help text is, "If needed, grab ruler to measure distance between centers of spheres. Once grabbed, use keyboard shortcuts to move ruler." In this sim, the value for distance is provided through the Move m1 and Move m2 slider interactions. For non-visual users there is no specific need to grab the ruler.

For Fourier Making Waves, @arouinfar and you will find the right balance. Reach out when/if you have questions.

[pixelzoom]

5/20/21 design meeting: @arouinfar @kathy-phet @pixelzoom

@arouinfar is working on the doc here: https://docs.google.com/document/d/1wOdmPMD704u4OLl9avI9tl2jpTvsAAcZwMaNfnr0qSs/edit?pli=1#heading=h.qbkcpoz1vzlo

This is probably ready for me to start working with.

For now, treat measurement tools like Coulomb's Law. Can revisit when we do interactive descriptions.

[arouinfar]

Thanks for the summary @terracoda! For now, I think it makes the most sense to skip the grab button. When it comes time to design the descriptions, we can reevaluate.

@pixelzoom I updated the focus order in the linked table to include the erase button and the updated layout on the Game screen.

We also need to specify step sizes for the sliders and NumberSpinners. I reviewed the mass controls in GFL (NumberControl) and GFLB (NumberPicker) for comparison. In GFL, the standard step size for alt input is not the same as constrainValue used on the slider. I've chosen to do the same here too, otherwise, it would take 60 key presses to traverse the entire slider range.

Amplitude sliders:

• Standard step (arrow keys): 0.1
• Smaller step (shift + arrows): 0.01
• Bigger step (page up/down or fn up/down): 0.25

NumberSpinners:

• Bigger step (page up/down or fn up/down): 2

I don't know if NumberSpinner has support for a bigger step like NumberPicker. If it does, great. If not, I'm fine with deferring until we get to a sim with a NumberSpinner with a larger range.

[pixelzoom]

• @arouinfar Is it worth the time to put this information in the design doc? Alternative Input support #53 (comment)
  Tracking in Update a11y design doc? #85.

[pixelzoom]

@arouinfar I see this in the design doc, in the Focus Order column of Table: Discrete Screen Outline:

Each of the visible amplitude sliders
Amplitude {{#}}, slider
Erase, button

I don't know how to interpret this. Please provide more description/clarification. And we're skipping the amplitude NumberDisplays for keyboard nav, correct?

EDIT: It might be easier to just check this in master, to see if I interpreted this correctly.

[pixelzoom]

@arouinfar Looking at the code for other sims that support keyboard navigation, it looks like we need to decide what to connect to the "play area" vs "control area". For example, in greenhouse-effect MicroScreenView.js:

    // PDOM - the accessible order for the control area contents
    this.pdomPlayAreaNode.pdomOrder = [ this.observationWindow, clipRectangle, windowFrameNode, photonEmissionControlPanel, moleculeControlPanel ];
    this.pdomControlAreaNode.pdomOrder = [ timeControlNode, showLightSpectrumButton, resetAllButton ];

I don't see this information specified in the design doc.

Raising priority to high.

[pixelzoom]

@arouinfar I recall a discussion where we speculated that we could ignore "play area" vs "control area" for now, because we're not doing interactive descriptions. Well, each screen has 2 top-level Nodes, named pdomPlayAreaNode and pdomControlAreaNode. And traveral order can only be set on those Nodes, not on the screen's ScreenView (top-level Node). So I can either put everything under (e.g.) pdomPlayAreaNode for now, or we can take a stab at this now so we don't have to redo later. I recommend the latter.

EDIT: Since I'm creating my own root Node in my ScreenViews (screenViewRootNode), I was able to get away with ignoring pdomPlayAreaNode and pdomControlAreaNode for now. But maybe still worth taking a look?

[pixelzoom]

@jessegreenberg pointed out that we've been using "interactive description" incorrectly during this design process. The definition is at https://github.com/phetsims/phet-info/blob/master/doc/interactive-description-technical-guide.md#what-does-interactive-description-mean, and it includes 3 components:

• Alternative Input (including keyboard)
• State Description
• Responsive Description

Relevant discussion with @jessegreenberg on Slack:

Chris Malley 4:32 PM
So in design meetings, we’ve been saying “Fourier will not support interactive description”. What we should have been saying is “Fourier will not support state description and responsive description”. Is that correct?

Jesse Greenberg 4:32 PM
Yep! That is correct.

Chris Malley 4:34 PM
Great, I understand now. But I think the use of “description” in so many places is going to be a continual source of confusion. Especially lumping alternative input into “description”.

Jesse Greenberg 4:40 PM
Heard, and agree that is a confusing aspect of the term. It has been brought up with the broader a11y team before and the decision was to continue using it since it is a good fit for the full feature set.

[pixelzoom]

Re:

• Do UI components need to have "accessible names" for Fourier 1.0 release? ...

I discussed with @jessegreenberg on Zoom.

accessibleName is the option to use. innerContent is older style, and could be replaced by accessibleName.

He doesn't feel like this would be a big win. The sim will not be promoted publically as having "labels only" (there's no icon for that). And for things like eraser button and info button, it's not clear whether to "label" or "describe".

[terracoda]

Hmm, I feel there could be some big wins actually by adding accessibleName. That is, if the implementation time is not particularly heavy.

• Team win 1 - the team starts thinking about accessible object names as they are designing the visual sim. I think this is actually a huge win. @arouinfar is already thinking about accessleNames for this sim.
• Team win 2 - devs get to understand more of the API options and if there are changes needed in the API for accessibleName
• Team win 3 - we all get to see the challenges in designing object names. It's not always obvious what to call something, as has already been pointed out.
• Learner win - for sims that have sonification, adding accessibleNames adds meaning to the sounds for some users with vision impairments who might be using a screen reader.

[terracoda]

Of course, I don't want adding accessible names to derail anything about adding alternative input. I just want to add some context to the benefits.

[pixelzoom]

@terracoda I appreciate the enthusiasm and definitely agree that there could be "wins" to adding accessibleName. But we are definitely jeopardizing milestones. We've gone from "no a11y support" to "keyboard nav for common-code UI components" to "full keyboard nav with custom UI components", and are now considering adding more --- all without changing the milestones for this sim. Something similar has also happened for UI sounds.

I'll happily implement whatever PhET wants. But we definitely need to pause, do a sanity check, decide what features are essential for 1.0, and reset milestones.

[pixelzoom]

@jessegreenberg and I also discussed slider issues today, specifically the constrainValue problem in phetsims/sun#698. @jessegreenberg questioned why shiftKeyboardStep would be smaller than the mouse/touch step size that is implemented via constrainValue. I explained that the keypad is used to input values in 0.01 increments, and we want to avoid having to use the keypad when setting amplitudes via keyboard input. So this is a case where its desirable to have shiftKeyboardStep be smaller than constrainValue step size. (EDIT: phetsims/sun#698 is resolved, so "step size" options are no longer conflicting.)

If we did want to add keypad to keyboard navigation... If you open the keypad using a mouse/touch, you then tab around in the keypad, and it appears to be usable. So we could revisit adding amplitude NumberDisplays to the pdomOrder. Issues with that:

• Adding NumberDisplays to the pdomOrder will mean 11 more elements to traverse before getting to the control panel.
• Where do we put NumberDisplays in the pdomOrder?
• NumberDisplay does not support keyboard input. We’d probably need to identify as 'button', and add an onclick handler.
• Having shiftKeyboardStep:0.01 for sliders still seems like it’s very desirable, even if you can get to the keypad.

• Discuss keypads again with @arouinfar and @kathy-phet.

DECISION: No traversal for NumberDisplay or Keypad. Eventually would like to type numbers into Keypad.

[pixelzoom]

For 6/10/2021 design meeting, discuss the outstanding checklist items in these comments:

• Alternative Input support #53 (comment)
• Alternative Input support #53 (comment)
• Alternative Input support #53 (comment)

[pixelzoom]

6/10/21 Fourier design meeting: @arouinfar @kathy-phet @jessegreenberg @KatieWoe @pixelzoom

Notes and decisions were added to above checklist items, see "DECISION" comments.

Summary:

• In general, no work will be done on features related to screen reader for 1.0 release
• No accessibleNames for 1.0 release
• No play area and control area for 1.0 release, see Specify "play area" and "control area" for each screen #76
• Slider is working well enough for 1.0 release, additional work will be discussed by @kathy-phet and @jessegreenberg for Q3 goals

[pixelzoom]

Alternative input support is done for Discrete and Wave Game screens.

@arouinfar Remaining tasks for Continuous screen:

• Specify keyboard steps for sliders -- see Specify keyboard steps for sliders in the Wave Packet screen. #83
• Specify pdomOrder (I'll start with "similar to Discrete screen") -- see Specify traversal order (pdomOrder) for the Wave Packet screen #84

[terracoda]

I am not sure if NumberDisplay is the same component as I've seen in the Arithmetic simulation, but I think it might be. I think it is an interface component that allows a learner to touch or click numbers in order to enter values into a textbox in the sim.

I agree if a learner can get to those discrete values right on the slider using the keyboard (or some alternative input method), they do not need to have access to the NumberDisplay component. That just needs to be made clear in the Teacher Tips for the sim. Making that clear will help all learners. Mouse users might switch to using the keyboard when they need a precise number. I think of NumberDisplay as an accessibility feature for learners using mouse and touch who would otherwise have a hard time entering exact values.

In the future, should we want to make NumberDisplay available to other input methods, I think it would make sense that the component is a navigable region or group, that is thoughtfully designed to not exponentially increase the number of Tab stops in the PDOM.

[pixelzoom]

@terracoda said:

I am not sure if NumberDisplay is the same component as I've seen in the Arithmetic simulation, but I think it might be. I think it is an interface component that allows a learner to touch or click numbers in order to enter values into a textbox in the sim.

NumberDisplay is a display only. It is not interactive, has no support for any kind of input. In Fourier, I've added a mouse/touch listener (PressListener) that open a keypad. There is currently no support for this in common-code, and it must be done on a sim-specific basic. Adding keyboard support to open a keypad would involve more sim-specific work, as described in #53 (comment):

NumberDisplay does not support keyboard input. We’d probably need to identify as 'button', and add an onclick handler.

I do not think that input support should be added to NumberDisplay - it is useful to have a UI component that is a display only, without the overhead of input handling. I do think it would be useful to have a common-code subclass of NumberDisplay that adds support for opening a keypad. But there is currently no proposal or GitHub issue for such a component.

[pixelzoom]

I do not think that input support should be added to NumberDisplay - it is useful to have a UI component that is a display only, without the overhead of input handling.

Thinking about this more, I've changed my mind - an option to NumberDisplay would be preferrable to a subclass. NumberDisplay is (or should be) used as a subcomponet on many other UI components: NumberControl, NumberSpinner,... It would be nice to have keypad entry as an option for those controls, probably via the nested numberDisplayOptions. And making keypad entry an option to NumberDisplay would make that easier. Having to choose between NumberDisplay and (e.g.) InteractiveNumberDisplay would be more complicated programmng, and make it more difficult to mutate this feature (turn if on and off), which is something that I suspect would be desirable via PhET-iO.

[pixelzoom]

I've converted all remaining checklist items to individual GitHub issues.

I've moved the general discussion about NumberDisplay input handling to phetsims/scenery-phet#678.

So this issue can be closed.