chore(llmobs): implement skeleton code for ragas faithfulness evaluator

[lievan]

This PR implements EvaluatorRunner, a periodic service for LLM Obs that runs evaluations on batches of finished spans.

We add _DD_LLMOBS_EVALUATORS to detect which evaluators should be enabled. Right now, the only supported evaluation is ragas_faithfulness

Within the trace processor, spans events—after being enqueued to the span writer—are enqueued to the evaluator runner.

On each call of periodic() we run a list of evaluators of the batch of finished spans. An evaluator is defined as a function that takes a span as an argument and returns an evaluation metric.

Right now, the faithfulness function is a dummy function that just returns an eval metric with score label 1. In a future PR we will implement the actual faithfulness evaluation.

Intended Usage

_DD_LLMOBS_EVALUATORS="ragas_faithfulness,ragas_.." DD_LLMOBS_ENABLED=true python3 app.py

! No user facing changes for this pr !

No changelog since this PR only implements the internal skeleton code necessary for RAGAS evaluation integration. The environment variable to enable the ragas evaluator service is hidden (_DD_LLMOBS_RAGAS_FAITHFULNESS_ENABLED) and will be made public when we implement an actual faithfulness function.

(Full e2e poc, which contains some differences)

See #10431 for an idea of what the full e2e implementation of the ragas integration looks like.

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

ddtrace/llmobs/_evaluators/ragas/faithfulness.py                        @DataDog/ml-observability
ddtrace/llmobs/_evaluators/runner.py                                    @DataDog/ml-observability
tests/llmobs/test_llmobs_evaluator_runner.py                            @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_trace_processor.py                                      @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/conftest.py                                                @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

[datadog-dd-trace-py-rkomorn]

Datadog Report

Branch report: evan.li/ragas-skeleton
Commit report: be893a1
Test service: dd-trace-py

✅ 0 Failed, 100 Passed, 850 Skipped, 1m 26.57s Total duration (13m 4.6s time saved)

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-09-20 20:32:45

Comparing candidate commit 04d202e in PR branch evan.li/ragas-skeleton with baseline commit 33daba9 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 356 metrics, 48 unstable metrics.

[Yun-Kim]

If we're going to have evaluators be a part of the trace processor, we're going to need to ensure we do the same thing on _child_after_fork() as we do for the span writer, i.e. something like

self._evaluator = self._evaluator.recreate()
self._trace_processor._evaluator = self._evaluator