Voice-Based Form Filling Component for Flutter to Enhance Digital Access to Public Services

[Taherabharmal]

Description

In light of the digital transformation of public services, this project endeavours to create a UI component tailored for Flutter applications. The main objective is to address the challenge faced by many citizens in completing digital forms by developing a voice-based form-filling solution. This component, when integrated into a Flutter form, will automatically extract existing form elements and conversationally engage users to facilitate form completion. By offering a generic voice-based form-filling capability, the project aims to significantly enhance the user experience of Flutter-based apps. Given the increasing popularity of Flutter for building mobile applications used by citizens and frontline workers, such innovation has the potential to streamline interactions with digital services and improve accessibility for all users.

Goals

Design and develop a voice-based form-filling component specifically tailored for Flutter applications on DIGIT.
Create a conversational interface that engages users to facilitate form completion using voice commands.
The interface should allow the user to hear as well as see the voice prompts and recorded responses.

Expected Outcome

• Design and build UI elements to activate voice based form filling
• Implement logic to extract existing form elements from Flutter forms and dynamically generate conversational prompts for users.
• Design and build of Interface where users can see written versions of the voice prompts and recorded responses. Interface should provide feedback to users during the form-filling process, including confirmation messages, error notifications, and prompts for clarification.
• Design and build of a backend service or API that can process speech input and convert it into text. This could involve utilizing existing speech recognition APIs (must be open source)
• Integration/build of LLM to understand and interpret user’s input -Acceess to Open AI or alternate LLM can be provided
• Thorough testing and debugging to ensure reliability, accuracy, and performance of the component across different devices and platforms.
• Optimization of the frontend code for efficiency, minimizing resource usage and maximizing responsiveness during voice interactions.

Acceptance Criteria

Voice-Based Form-Filling Component Development:

• The voice-based form-filling component must be designed and developed
• The component should work on Flutter applications on DIGIT.
• The component should seamlessly integrate into existing Flutter forms, allowing users to interact with forms using voice commands.
• Users should be able to activate the voice-based form-filling feature through a designated UI element.

Conversational Interface:

• The component must feature a conversational interface that engages users during the form-filling process.
• Users should be provided with clear and concise voice prompts to guide them through the form completion process.
• The interface must allow users to both hear and see the voice prompts and recorded responses.

Form Element Extraction and Dynamic Prompt Generation:

• The component should automatically extract existing form elements from Flutter forms.
• Dynamic conversational prompts must be generated based on the extracted form elements to facilitate user interaction.

User Interface Design:

• A user interface must be designed and built to display written versions of the voice prompts and recorded responses.
• The interface should provide feedback to users during the form-filling process, including confirmation messages, error notifications, and prompts for clarification.

Backend Speech Processing Service:

• A backend service or API must be implemented to process speech input and convert it into text.
• The backend service/API should utilize open-source speech recognition APIs to achieve speech-to-text conversion.

Integration with NLP Service:

• Integration with an NLP service or API is required to understand and interpret user input.
• The NLP service/API must be capable of analyzing user utterances and extracting relevant intents and entities.

Thorough Testing and Debugging:

• Comprehensive testing must be conducted to ensure the reliability, accuracy, and performance of the voice-based form-filling component.
• Testing should cover different devices, platforms, and scenarios to identify and address any issues or bugs.

Frontend Optimization:

• The frontend code must be optimized for efficiency to minimize resource usage and maximize responsiveness during voice interactions.
• Measures should be taken to ensure smooth performance across various devices and screen sizes.

Out of Scope:

• The component should be able to identify the selected language of the user and provide voice prompts accordingly

Implementation Details

• Libraries/3rd Party tools may be used but must be open-source
• DIGIT uses Java Spring Boot in the backend and Flutter for the frontend for mobile applications
• Staging environment of DIGIT will be provided and an existing form from a product may be used.

Mockups / Wireframes

• To be created by the selected contributor with support from DIGIT product manager

Product Name
DIGIT

Project Name
Voice-Based Form Filling Component for Flutter

Organization Name:
eGovernments Foundation

Domain
Public Services

Tech Skills Needed:
Flutter, Java Spring Boot

Mentor(s)
Ramkrishna Sahoo

Complexity
High

Category
Feature

Sub Category
API, Frontend, Backend, LLM

[AieshaShetty]

Hi @Taherabharmal, I would like to work on this project. My tech stack includes core Java, Spring Boot, and JPA. Is there any smaller issue to work on this first?

[sbhamita]

Hi @Taherabharmal, I would like to work on this project. My tech stack includes core Java, C++,DSA,HTML. Is there any smaller issue to work on this first?

[Raj0408]

Hi @Taherabharmal
I want to contribute to this project for flutter and machine learning.
My tech stack is Flutter, Django, and machine learning.

if there is any issue related to fronted or machine learning please give me a chance.

[AbhimanyuSamagra]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

[renuka010]

Hi @Taherabharmal,
I would like to contribute to this project. I am proficient in Java, Spring boot and also have experience creating LLM pipelines. I would like to know more information and guidelines on how to get involved with this project.

[abhi87374]

Hello @Taherabharmal, I'm new to Flutter development & had some experience in it this year , eager to learn more and contribute to this project. The detailed description you provided gives me a good understanding of what needs to be achieved with the Voice-Based Form Filling Component. While I may be a beginner, I'm enthusiastic about gaining hands-on experience and growing my skills. If there are any smaller tasks or issues suitable for someone at my level, I'd appreciate the opportunity to start there and gradually ramp up. Thank you for considering me for this project!

[goyal02aditi]

Hello Team
I am a computer science student with proficiency in flutter and good knowledge in backend. I found the project Voice-Based Form Filling Component for Flutter to Enhance Digital Access to Public Services very innovative and strongly aligning with my capabilities. I would love to work on it and make some meaningful contributions.
Before moving forward, I have a couple of questions since I'm participating in C4GT’s DMP event for the first time and wish to familiarize myself more on how things work on the other end :-

How many teammates can I expect working on a project?
Will the backend and frontend be tackled by different set of people or everyone is expected to work on both aspects?
Would there be a separate platform to communicate with my team and the mentors, apart from the common discord portal?
Whom should I connect with regarding the technical queries about the project details?
Thanks for your consideration, looking forward to be part of the team asap!
Warm regards
Aditi Goyal

[pinki639]

hey there @renuka010 pinki bhati is this side i want to work on this project , i have good understanding of core and advanced java

[VIDHIGUPTA17]

Subject: Request to Contribute to Voice-Based Form Filling Component Project

Dear Ramkrishna Sahoo,

I hope this message finds you well. My name is Vidhi Gupta, and I am interested in contributing to the "Voice-Based Form Filling Component for Flutter to Enhance Digital Access to Public Services" project under eGovernments Foundation.

I have a strong background in Flutter development and API integration, and I am excited about the potential impact of this project. I am eager to bring my skills to the team and contribute to enhancing digital accessibility.

Could you please provide more details on how I can start contributing and any specific areas where my expertise might be most beneficial?

Thank you for considering my request. I look forward to the opportunity to discuss how I can contribute to this meaningful project.

[Ramkrishna-egov]

Issue Link [https://github.com/egovernments/egov-rnd/issues/20]

Weekly Goals

Week 1

• Research and development over digit component
• Look for libraries that can help us
• Create UI components to handle voice commands and form submission.

Week 2

• Research on methods to achieve voice integration on dummy UI
• Try talk back feature
• Semantic widget integration

Week 3

• Flutter TTS and speech-to-text package integration
• Add voice command functionality to fill form fields
• Use DigitTextField from digit_components package
• Create UI components to handle voice commands and form submission

Week 4

• Implement form validation and snack bar notifications
• Use the Reactive Text Form field in code along with form control
• Transition DigitTextField to DigitTextFormField
• Add more Digit Components enabled with voice commands such as DigitReactiveDropDown

Week 5

• First, implement it on the Flutter Material 3 widget and then transition it to DigitReactiveDropDown
• Add voice command in Digit Component (DigitCheckBoxTile) (reactive checkbox)
• Use "agreed" as a keyword to mark the tick in the checkbox

Week 6

• Research and Choose the Best LLM Model
  Compare Different LLM Models:
  - Evaluate models like OpenAI's GPT-4, GPT-3.5, and other alternatives.

    **Consider Factors:**
    - Look at accuracy, response time, cost, ease of integration, and available features.
  
    **Select the Best Model:**
  
    - Choose the model that best fits your needs based on the evaluation.
  

• Documentation on the finalized approach.

Week 7

• Create an analysis of Research in a summarized manner
• Make PPT to showcase the work to the technical council
• Add a dummy apk of current version using speech_to_text in PPT

[EuclidStellar]

Issue Link [https://github.com/https://github.com/egovernments/egov-rnd/issues/20]

Weekly Learnings & Updates

Week 1

1. Research and Development on Digit Component:
   Explored the digit_components package to understand its functionality and how it can be leveraged for form fields.
   Identified the key components that could be integrated with voice commands.

2. Library Exploration:
   Investigated libraries such as speech_to_text and flutter_tts for integrating voice commands.
   Evaluated the pros and cons of different libraries to determine the best fit for the project.

3. UI Components for Voice Commands:
   Created basic UI components to handle voice commands and form submission.
   Developed initial prototypes to test the integration of voice commands with form fields.

Week 2

1. Voice Integration on Dummy UI:
   Implemented basic voice integration on a dummy UI to test the feasibility.
   Ensured the app can listen for voice commands and update the UI accordingly.

2. TalkBack Feature Exploration:
   Experimented with TalkBack as an alternative to text-to-speech for providing audio feedback to users.
   Compared the user experience and effectiveness between TalkBack and flutter_tts.

3. Semantic Widget Integration:
   Integrated semantic widgets to improve accessibility and provide better screen reader support.
   Ensured that the app is accessible to users with visual impairments.

Week 3

1. Flutter TTS and Speech-to-Text Integration:
   Successfully integrated flutter_tts and speech_to_text packages into the project.
   Implemented voice command functionality to fill form fields.

2. DigitTextField from Digit Components:
   Incorporated DigitTextField from the digit_components package to manage form fields.
   Created UI components to handle voice commands and form submission.

Week 4

1. Form Validation and Snack Bar Notifications:
   Implemented form validation to ensure user input meets required criteria.
   Provided user feedback through a snackbar notification upon successful form submission.

2. Reactive Text Form Field:
   Used the Reactive Text Form field in code along with form control.
   Transitioned DigitTextField to DigitTextFormField.

3. Digit Components with Voice Commands:
   Added more Digit Components enabled with voice commands such as DigitReactiveDropDown.

Week 5

1. Flutter Material 3 Widget Implementation:
   Implemented voice command functionality on Flutter Material 3 widgets.
   Transitioned this functionality to DigitReactiveDropDown.

2. Voice Command in Digit Component (DigitCheckBoxTile):
   Added voice command functionality to DigitCheckBoxTile (reactive checkbox).
   Used "agreed" as a keyword to mark the tick in the checkbox.

Week 6