evaluate results from elastic search as well

[SubhadityaMukherjee]

No description provided.

[SubhadityaMukherjee]

I did evaluate elastic search with the queries that do not rely on filtering but it was not fair because that is not really the strength of the LLM.
Evaluating it with the structured query remains to be done.

[PGijsbers]

but it was not fair

I don't want to be a hard-ass here, but the user doesn't care what's "fair". If the user isn't happy with the service, we can't say "Oh, but you see, we have someone in the back-end personally reading your query, going to the paper archives to read through the datasets, and then returning a number. It's a very different approach than conventional fast and reliable software, so you shouldn't have the same expectations."

That's a wild exaggeration of course, but it drives the point home. Right now the LLM prototype is a standalone prototype so it's fine. But for normal integration we need to compare to conventional methods so we can make an informed decision about what the user experience will be like.

[SubhadityaMukherjee]

I just meant that we havent done the evaluation yet ): and that the one that we have right now was not fair because we did not compare it to the elastic search yet. So we cannot be certain how much better it is.
Not that it is not fair to compare them.

Just that comparing the keyword search between the RAG and elastic search led to slightly better results for the RAG. But elastic search is not able to say find the appropriate filters and the latter was not evaluated yet.

[PGijsbers]

okay clear :) just wanted to make sure, sorry if it came off the wrong way