STORY: As a Monarch handler developer, I want to receive text-only preprocessor responses to followup queries on an entire graphic, so that we can provide responses to queries made directly from the Monarch.

[jeffbl]

Future work items will cover:

• selection of subregions of the graphic
• responding with tactile layer(s)
• paying attention to previous followup queries

The goal of this story is to create the preprocessor to be able to serve the simplest possible followup query response. The user scenario is:

• user has a graphic loaded in the IMAGE app on the Monarch
• user verbalizes a question about the graphic as a whole
• stt is used to create a followup query, that is sent to the server
• followup preprocessor sends original graphic to LMM, along with user query
• text response is returned to orchestrator, for consumption by handler
• completed rendering is sent to Monarch, which uses TTS to read response

Technical plan:

• create new preprocessor text-followup
• create new preprocessor schema text-followup.schema.json

The text-followup preprocessor will be pretty much identical to the graphic-caption preprocessor at first, but will diverge as we add additional features like paying attention to previous queries.

@JRegimbal @VenissaCarolQuadros

QUESTION: It could be combined with graphic-caption, where it works as it does currently if there is no followup content in the request, but then pay attention to that content if it exists. Would that be better long-term?

QUESTION: Downsides to a generic text.schema.json for preprocessors? otherwise text-followup.schema.json? ANSWER FROM @JRegimbal : text-followup (modified above)