GenAI use-case evaluation with semantic kernel and prompt flow

[tamer-abdulghani]

Hello,

We are building a genai use-case with the help of azure OpenAI and semantic kernel.
Basically, writing financial reports.

Our use-case involve constructing a large prompt (> 20K tokens) that contains various configuration such as:
Report parameters, year, month, input data, meta prompt, etc.

Now, we are struggling to evaluate our prompts and setting up GitHub pipeline for evaluation, as there is no clear guidance on how to achieve this with Semantic Kernel .

We have tried prompt flow SDK but it's not straightforward to implement this, specially we are not using SK planner.

We are wondering what would be the best practices for achieving the evaluation pipeline ?

[sophialagerkranspandey]

We are working on a sample that should be released in a few weeks on eShopSupport that will answer this.

[alesspanms]

Hi @sophialagerkranspandey, has already been released mentioned sample? If not, are there a branch containing some work in progress documetatuon or code?

[sophialagerkranspandey]

There will be a blog post shared out soon on this work as well: https://github.com/dotnet/eShopSupport