docs(evals): Updated docs for evals #1512
Conversation
|
|
||
| You can see the details of your evaluation run in this page, including original input, extracted context and metrics (if any). | ||
|
|
||
| <!-- TODO, more convincing conclusion here? --> |
There was a problem hiding this comment.
@mjchristy thoughts?
| You can also provide custom extractors to be used in `eval:extractData` and | ||
| `eval:flow` commands. Custom extractors allow you to override the default | ||
| extraction logic giving you more power in creating datasets and evaluating them. | ||
| <!-- TOOD: Any caveats on where this approach does not work (ES5 or something?) --> |
| // ... | ||
| }); | ||
| ``` | ||
| 1. **Inference-based evaluation**: In this type of evaluation, a system is run on a collection of pre-determined inputs and the corresponding outputs are assessed for quality. |
There was a problem hiding this comment.
"A system is run on" sounds strange to me. Maybe we just say, "This type of evaluation is run against a collection of..." or similar
There was a problem hiding this comment.
s/1./*
Numbered lists are really more for sequences (i.e., procedures, etc.).
| ```posix-terminal | ||
| npm install @genkit-ai/evaluator @genkit-ai/vertexai | ||
| ``` | ||
| 2. **Raw evaluation**: This type of evaluation directly assesses the quality of inputs without any inference. This approach typically is used with automated evaluation using metrics. All required fields for evaluation (`context`, `output`) must be present in the input dataset. This is useful when you have data coming from an external source (eg: collected from your production traces) and you simply want to have an objective measurement of the quality of the collected data. |
There was a problem hiding this comment.
Do you mean quality of "outputs" here?
There was a problem hiding this comment.
Hmm, no? Because the inputs contain the outputs ;)
I can see how this can be confusing though.
Maliciousness is not RAG specific. |
| // ... | ||
| }); | ||
| ``` | ||
| 1. **Inference-based evaluation**: In this type of evaluation, a system is run on a collection of pre-determined inputs and the corresponding outputs are assessed for quality. |
There was a problem hiding this comment.
s/1./*
Numbered lists are really more for sequences (i.e., procedures, etc.).
| ```posix-terminal | ||
| npm install @genkit-ai/evaluator @genkit-ai/vertexai | ||
| ``` | ||
| 2. **Raw evaluation**: This type of evaluation directly assesses the quality of inputs without any inference. This approach typically is used with automated evaluation using metrics. All required fields for evaluation (`context`, `output`) must be present in the input dataset. This is useful when you have data coming from an external source (eg: collected from your production traces) and you simply want to have an objective measurement of the quality of the collected data. |
There was a problem hiding this comment.
s/2./*
(see comment on L17)
Checklist (if applicable):