Screen reader architecture

[francipvb]

Hello all,

I found the project and I started to code on it. After it, I found the PR #120 and I had to destroy all my work.

Is there a place with some documentation about the screen reader architecture itself?

I mean:

• How events are catched (obviously I found this).
• How events should be processed?
• What are the purpose of the traits defined in the traits module (I assume somewhat like NVDA touch and NVDA+enter command actions, but not sure) to open touchscreen support later?
• What about screen reader events, like focus changes? I found an enumeration for screen reader events but it has only a few members. Maybe these events can be categorized like at-spi events itself.

I inferred some of these questions myself, but I am not sure the direction you (project authors) want to go with it, so I am posting this before any big code effort.

Another consideration are screen reader speech-related behavior, like how messages are structured, how about braille output, etc.

I thought about just adding more impls for the screen reader state struct with commands for notifying the focused object, caret/line/etc changed, text changes, etc.

From there, in the apply_all function, match on the screen reader event and applying the corresponding to these other functions.

Another alternative, to group every object-tree related state, is to add an additional struct to hold cache, event and focus history and active applications history and implement the screen reader event processing there, passing the required structs such the speech backend.

Another thing to consider is to extract the speech synthesis to an additional struct to be able to implement, for example, direct calls to espeak-ng later, same for braille.

The last question is about addons. What they could do? How is event processing related to them? What is the programming languages you plan to allow?

Thank you for your effort. I hope I can help you at least a little.

[francipvb]

I browsed to the official home page and found ansuers for these questions.

[TTWNO]

Hi there,

I'm just going to bed here in Canada. I promise I'll get t9 this in the morning.

[TTWNO]

Hi, @francipvb let me explain what I'm attempting to do right now with the mother-of-all-refactors branch.

We were running into problems testing the state and reactions of various components of the screen reader, since, just to test how a function works, we needed to pass a full ScreenReaderState structure into the function, followed by checking which pieces of state were changed. In an attempt to make Odilia more testable and modular, I'm attempting to create an architecture that takes in minimal state as required for the event, and returns a list of operations that can be checked tested against. These operations could be "say X", or "focus user on Y".

I'm very comitted to making Odilia easy to test, and so each function simply should only take the state absolutely necessary, and then we can test using that.
Unfortunately, this opened us up to a bunch of cache invalidation problems, which I've been trying to solve in the past week or two, but I don't have a degree or anything fancy, and this is a very hard problem. (Some say there are only two hard problems in software development: naming things, and cache invalidation).

I think I'm close, but just a bit more patience we would be there. Odilia has never handled more than 10-ish AT-SPI events, so once the architectural decisions are figured out, it should be only 5-10 hours of work for me to actually implement it.

If you have any opinions here about how to use the Rust type system and concurrency to our advantage, please let me know!

The last question is about addons. What they could do? How is event processing related to them? What is the programming languages you plan to allow?

They will be able to listen to, intercept and modify various stages of the event processing pipeline. So they can discard/modify events coming from AT-SPI, they could discard or modify OdiliaCommand variants once they are created, etc. I personally not sure about languages because we can either use DBus as a communication mechanism with addons (which would be external processes) or we could use WASM plugins, which would prioritize performance. I'm on the fence about which one to implement.

Addons are not here yet, I just want to build a system which will have easy hook-in points for addons.

how about braille output

Braille is not supported right now. There's no stable liblouis or brltty Rust port/bindings. This will have to be considered later.

How events are catched

Events are not cached, only accessible objects denoted by the struct AccessiblePrimitive (key) and CacheItem (value).

What are the purpose of the traits defined in the traits module (I assume somewhat like NVDA touch and NVDA+enter command actions, but not sure) to open touchscreen support later?

Those are my attempt at creating a type-system based guarantee of who is accessing what data at which point in the pipeline. It is not about anything external or touchscreen-support realted.

What about screen reader events, like focus changes? I found an enumeration for screen reader events but it has only a few members. Maybe these events can be categorized like at-spi events itself.

There are three enums to pay attention to:

1. ScreenReaderEvent this is a command from user input. "I would like to structurally navigate to the next link", "open Odilia settings", "disable learning mode", etc. We don't support very many of these right now because we don't have any way to receive key inputs (I'm working on this accessibility portal issues that should solve this, but this is still in early stages. I'm expecting an implementation by the end of the year at the latest.
2. OdiliaCommand is an internal action that Odilia must perform, generally a very small "atomic operation" like "add X to cache", "say Y", or "move focus to Z". This is distinctly different from what the user asks for, since one user action may constitute multiple operations that Odilia must perform. For example, structural navigation moves both focus and the caret.
3. atspi_common::events::Event is an event from the Linux accessibility infrastructure: "A new element is on this webpage", "the progress bar has changed its value", "the user has moved their cursor". This is not a direct interaction from the user, AT-SPI is simply making us aware that the user has already done something. Based on this, we might speak some text, or tell the user the role of the object they're focused on, or add something to the cache (all OdiliaCommands).

I think this answers most of your question. Let me know if you need any clarifications. I'm always happy to help!

[francipvb]

Hello,

Regarding naming things, I often find the same issue.

Ok, to summarize my understanding I will put the event handling pipeline in a numbered list:

1. The event processor subscribes itself to the at-spi registry.
2. When an event arrives, the event is processed and a list of commands is returned from it.
3. The command list is sent to the screen reader internal process.
4. Commands are processed sequentially and internally by the screen reader core.
5. Step 2-4 are repeated.
6. Event processing loop is scheduled to be stoppped.

I see your point, but the main issue here are the screen reader concepts. I propose these concepts:

• OdiliaGesture: an input action coming from the user.
• OdiliaObject: an abstraction on top of the At-SPI accessible objects with implementations to get various properties and query and perform actions. This structure can store a reference to a native At-SPI object.
• OdiliaCommand: an order sent to the screen reader (like say "x"). This enum should reference only OdiliaObjects.
• OdiliaEvent: an abstraction on top of At-SPI events to abstract them in logic ways. For example, FocusChanged. This can be an enum and every referenced object should be an OdiliaObject.
• OdiliaAction: an action available from the object. Not sure if we can send orders to accessible objects. For example this can be activate, expand, collapse and so on.

I got inspiration from the NVDA code itself. They organize the code in classes, we have to use other structures.

Note these concepts can even be simply traits, for example OdiliaObject, whose structures could be populated by other accessibility toolkits.

About cache, At-SPI events could be used to permanently update the cache, and there is a state change that notify about at-spi object destruction so we can remove it from the cache.

If you want, I can code these concepts in a branch in my own clone.

[francipvb]

Regarding event processing, we can structure the system in two parts, event emitters and event handlers. Event emitters could be, for instance, At-SPI registry. Event handler can be the screen reader itself, or an addon. I saw in the home page that you want to setup a channel for events and from there we can direct the events to handlers. We must implement a way to notify the screen reader to stop processing the event or continue with it, also we have to ensure the handlers are called in proper order, based in priority.

Another idea I have is to return an handler per OdiliaObject from a factory. This way we can have granular control and we can ensure an object's events are handled by a well defined handler. If the handler does not handle a particular event, the event is sent to the default handler whose purpose is to implement default behavior. The handler also has the option to process the event and also send it to the default handler.

This way we have a good extensibility structure for addons and included scripts to handle events.

[francipvb]

The last thing I missed is the following:

Screen reader state should be split in traits. Not sure if the type system allows it, but we can define odilia modules as traits, for example OdiliaSpeech, OdiliaBraille, OdiliaInput, OdiliaFocus, etc. The issue could be how to request these modules from an addon/handler/etc.

[TTWNO]

I propose these concepts:

I like your 5 concepts. My only concern is OdiliaEvent. Ideally, it would not be necessary to abstract over AT-SPI.

We must implement a way to notify the screen reader to stop processing the event or continue with it, also we have to ensure the handlers are called in proper order, based in priority.

I'm basically trying to do this with Mutex<_>es and RwLock<_>s; manually implementing which events can proceed in parallel vs sequential based on priority is much harder than allowing the Rust type system to ensure that we can not violate the constraints of the cache system.

The good news is that outside of caching, it's pretty easy to keep state consistent. Caching is by far the hardest thing to get right.

Another idea I have is to return an handler per OdiliaObject from a factory. This way we can have granular control and we can ensure an object's events are handled by a well defined handler. If the handler does not handle a particular event, the event is sent to the default handler whose purpose is to implement default behavior. The handler also has the option to process the event and also send it to the default handler.

I'm not sure if I understand what you're proposing. Could you rephrase it? The factory pattern is pretty rare in Rust because it requires explicit dynamic dispatch: Box<dyn Trait>.

Screen reader state should be split in traits.

I'm not sure what advantage this has? Traits can not be accessed by third-party plugins anyway: whether via WASM or DBus. Even if they were able to be exported and implemented, what functionality does it add for us? Again, it is possible I'm just misunderstanding.

If you want, I can code these concepts in a branch in my own clone.

I'd be interested to see where you're going with this. And I could absolutely use some help getting the architecture nailed down. Whether you want to discuss it more or just go off and see what you can whip up as a proof of concept, either is totally good with me! Whatever you are able to contribute would be extremely valuable.

[francipvb]

I propose these concepts:

I like your 5 concepts. My only concern is OdiliaEvent. Ideally, it would not be necessary to abstract over AT-SPI.

We must implement a way to notify the screen reader to stop processing the event or continue with it, also we have to ensure the handlers are called in proper order, based in priority.

I'm basically trying to do this with Mutex<_>es and RwLock<_>s; manually implementing which events can proceed in parallel vs sequential based on priority is much harder than allowing the Rust type system to ensure that we can not violate the constraints of the cache system.

The good news is that outside of caching, it's pretty easy to keep state consistent. Caching is by far the hardest thing to get right.

A simpler approach for event handling is just to check the return value of the event handler who processed it. An enum value can determine if the event should continue in the processing pipeline or not. Parallel processing of events are not a must in screen reading context. If you have to do some parallel or secondary work on it, just send to a new spun up task.

Another idea I have is to return an handler per OdiliaObject from a factory. This way we can have granular control and we can ensure an object's events are handled by a well defined handler. If the handler does not handle a particular event, the event is sent to the default handler whose purpose is to implement default behavior. The handler also has the option to process the event and also send it to the default handler.

I'm not sure if I understand what you're proposing. Could you rephrase it? The factory pattern is pretty rare in Rust because it requires explicit dynamic dispatch: Box<dyn Trait>.

In fact I don't have a good idea about this.

All user interface events require a particular behavior from the screen reader, so here are a difficulty in defining an architecture for extensions to be able to handle every particular event with all of the data while maintaining the screen reader behavior if the extension wants it.

My original idea was to link in some way a particular OdiliaObject to a particular event handler whose purpose is to handle all events related to the object. The first case is where the only event handler available is the default one (implemented in the core), and the second is where there is a event handler chain, where the first event should handle the event and either forward it to another handler in the chain or stop processing it by returning a result or an error.

An addon should register an OdiliaEventHandlerFactory whose single method takes an OdiliaObject and returns an event handler for the object. The factory has access to the next handlers chain and can wrap the handlers one on top of another to make an event process pipeline for an object.

Another alternative, more advanced, is to process the OdiliaObject itself and return an wrapped one if desired, with custom implementations of events, properties, new roles, user-defined states and custom behavior for events like when you read the entire object.

In this case we call top to bottom a list of object wrappers, having an wrapped odilia object to store in the cache.

Screen reader state should be split in traits.

I'm not sure what advantage this has? Traits can not be accessed by third-party plugins anyway: whether via WASM or DBus. Even if they were able to be exported and implemented, what functionality does it add for us? Again, it is possible I'm just misunderstanding.

I thought about making the state into a trait to be able to code a proxy, enabling us to make a proxy for every screen-reader core methods access, enabling us to add security capabilities in the future.

If you want, I can code these concepts in a branch in my own clone.

I'd be interested to see where you're going with this. And I could absolutely use some help getting the architecture nailed down. Whether you want to discuss it more or just go off and see what you can whip up as a proof of concept, either is totally good with me! Whatever you are able to contribute would be extremely valuable.

I will try to code my ideas as I can express these better in code than in human words... At least the type system and some skeletons.

[francipvb]

Hello, Here is my proposal as code. Note that this is only to show up my idea, not to be a final form:

Concepts branch

[TTWNO]

I apologize. This code is over my head. I don't understand, for example, why screen reader state would be stored as a HashSet, and why there would be a "previous" and "next" hash of the state?

I do see how you joined events into the necessary data to make decisions, so CaretChanged has both a previous and next position. That's smart; I like that.

[francipvb]

The hash sets are to be able to take a diff and act according to it. The states are object states, not screen-reader states. In some situations, these states could be a low-level way to handle some events, like collapse and expand.

[francipvb]

Hello,

Another idea is to use actix framework to implement the various parts and avoid manual implementation of event listening and processing loop, also for other systems like braille and speech.

Implementing a system is very simple, just implement the actor trait and add logic for starting and stopping the particular system. Also you implement the message handler trait to listen for messages from other systems.

[francipvb]

In fact, actix project has two parts:

• actix-web: The http/websocket/etc high-level framework.
• actix: This is the foundation of actix-web, it is an actor pattern implementation.

In escense, actix does the dirty work for us, allowing inter-actor communication to send, for example, events.

About tokio, I don't know very much about integration between actix and tokio, but it has it's own way to spawn async processing.

[francipvb]

Just a demo on what can be achieved with actix without too much overhead:
francipvb@e910044

I just implemented the CTRL+C handler from tokio. If you look carefully you'll see that the only way to interact from async code is by sending messages. This is similar to a raw stream but with responses implemented.

[TTWNO]

Just took a look at this and it looks interesting; I think it adds a bit of conceptual complexity just by taking in an entire framework for how to handle tasks, when we have a mostly working dispatch system already.

I have some more thoughts I'm going to write in a separate comment, not a reply.

[francipvb]

Maybe I am wrong... But if there are performance issues, work related to dispatching events can be left in the main thread while other tasks could be dropped to other threads. Actix adds some abstractions for message interchanges between them.

Other concern is about extensibility, with a well known system we can have more developers and they won't need to learn the dispatching system implemented.

I missed to thank you and your collavorators for this project, I thought about starting a screen reader for linux but I didn't have time to think carefully about it.

[TTWNO]

But if there are performance issues, work related to dispatching events can be left in the main thread while other tasks could be dropped to other threads. Actix adds some abstractions for message interchanges between them.

It shouldn't be a problem to worry about message passing between different tasks. Each event should be handled on its own task and communicate with state via shared memory (or at least, that was my plan).

Other concern is about extensibility, with a well known system we can have more developers and they won't need to learn the dispatching system implemented.

Possibly. I haven't seen enough to know for sure whether this adds anything worthwhile. For example, is middleware easy to implement here? I'll need to take a look into it later. Also, again, since actix uses its own runtime, we'd have to get upstream support from zbus.

I missed to thank you and your collaborators for this project, I thought about starting a screen reader for linux but I didn't have time to think carefully about it.

Thanks! It's been fun so far! Glad to hear it's making your contributions a bit easier.

[francipvb]

Hello again,

What about your at-spi packages? are the enumerations, interfaces and so generated automatically?

Another thing is about examples, I just try to run focused-async-std and focused-tokio examples, but neither of these work.

[TTWNO]

What about your at-spi packages? are the enumerations, interfaces and so generated automatically?

They are not generated automatically, since there is no IDL for these at the moment. I should propose this to at-spi2-core.

Another thing is about examples, I just try to run focused-async-std and focused-tokio examples, but neither of these work.

Strange, explain what you mean by "they don't work"?

[TTWNO]

Also, sorry for the time between responses. I'm not great with GH response times unless I get pinged in Matrix.