Evaluate Performance and Accuracy of AI Models

[jvJUCA]

• Conduct rigorous evaluations to assess the performance and accuracy of AI models in analyzing user responses compared to human-coded annotations.
• Measure metrics such as precision, recall, and F1 score to quantify the effectiveness of AI-driven response categorization and sentiment analysis.
• Benchmark AI models against state-of-the-art approaches and baseline algorithms to identify areas for improvement and optimization.
• Solicit feedback from researchers and domain experts to validate the relevance and usefulness of AI-generated insights in real-world usability testing scenarios.

[kindler-king]

I would also be interested to take up on this task, given my past experience running A/B tests and evaluation studies on ML models.

I feel that it is also essential to compare the combination of feature engineering and data pre-processing with SOTA ML/NLP techniques to get the best possible result.

The evaluation study needs to be well documented for future work, and can serve as the baseline for choosing models we want to integrate.

I am well connected to researchers being one myself ( check out some of my past work here: https://scholar.google.com/citations?user=vsyHmL0AAAAJ&hl=en)

Would love to discuss this in more detail, and a sample user response or any sample datasets would go a long way in understanding the way forward in going about this study.

[marcgc21]

@jvJUCA this issue is related to the eye-tracking repository?

[kindler-king]

@marcgc21 Do you know where a good place to start would be?
If I could get my hands on some human-coded annotations and user-response data, then I would have an idea where to start.
Even if we don't have access to data, if I know exactly what kind of data we might need for this comparative analysis, I could look online for some open-access datasets for gold-standard responses and collect user-responses from the community.

Looking forward to some more information on this issue, so that I can start working on it :D
Thanks a lot,

[jvJUCA]

Hi everyone, this issue is related to sentiment analysis, it will be used on the webcam record on a successfully made test.

[kindler-king]

@jvJUCA That sounds great, do you think we could gain access to a sample video recording as a part of the test set?

I feel for starters we could use a gold standard VIDEO dataset, such as CREMA-D Or RAVDESS as shown here
https://paperswithcode.com/datasets?task=video-emotion-recognition&mod=audio

Then it's a matter of testing the SOTA methods on the dataset of choice and finding the best options, luckily we can get some metrics from the relevant papers citing the dataset and it would be even simpler if they have made their pretrained models available for open access.

I would love to work on this as it's closely related to the other data extraction issue.