More work on llm-as-judge phrase endpointing

[kwindla]

Some proposed/possible additions to the "natural conversation" phrase endpointing:

• change in the pipeline/elements to try to pass all frames in the expected sequence
• change the pipeline architecture to do as much as possible in parallel (for latency)
• iterated on system prompt for the llm judge
• concatenate user messages for the llm judge -- this seems to be necessary for good results, in my testing
• temporarily disabled idle timeout to make it easier to test
• not-quite-finished interruption handling logic

The llm-as-judge performance is better in this version, I think. Latency is also lower.

There are at least two bugs to fix:

1. The bot doesn't talk when prompted initially with the pipeline is started with anLLMMessagesFrame. That's easy to fix.
2. Multiple inferences can cause the TTS to speak over itself. We either need to fix this with proper interruption handling, or move the TTS inference out of the parallel pipelines so that we're not doing TTS inference greedily. That would add a little bit of latency, but might be the right thing from a cost perspective, anyway.

[aconchillo]

Thanks! Yes, I remember doing this in the gated aggregator.

[manish-baghel]

@kwindla @aconchillo
Thanks for the great example.
Just curious, have you thought about utilizing InterimTranscriptionFrames from stt and feed those to a Completeness pipeline which can directly push frames containing the complete/identified phrase instead of outputting YES/NO?

[markbackman]

This is a nice improvement upon the work @aconchillo did. I think we should merge it and continue to improve the functionality and prompting as we learn more.