Context Recall returning NaN when using GPT-4 models

[FranciscoAlves00]

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
When using any gpt-4 model as an evaluator, the context recall metric returns an NaN result and the following warning for almost every single question:
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'

I have tried this with my own dataset, as well as following the instructions in https://docs.ragas.io/en/stable/getstarted/evaluation.html simply changing the evaluator to any of the GPT-4 models (gpt-4-0125-preview, gpt-4-1106-preview and gpt-4). From the 10 questions in the testset, I got on average 9 NaN results for that metric. The other metrics work correctly.

Ragas version: 0.1.5
Python version: 3.10

Code to Reproduce
Follow the code in https://docs.ragas.io/en/stable/getstarted/evaluation.html simply changing the evaluator to any of the GPT-4 models (gpt-4-0125-preview, gpt-4-1106-preview and gpt-4).

Error trace
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'

[shahules786]

Hey, can you please share some data points that I can use to reproduce the issue?
I'll raise a fix - this is mostly an issue related to the JSON formating which we are working on.

[FranciscoAlves00]

from ragas.metrics import (
answer_relevancy,
faithfulness,
context_recall,
context_precision,
answer_correctness,
context_relevancy
)
from ragas import evaluate
from langchain.chat_models import ChatOpenAI
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
amnesty_qa

gpt4 = ChatOpenAI(model_name="gpt-4-0125-preview")
#gpt4 = ChatOpenAI(model_name="gpt-4")

result = evaluate(
#experiment_dataset,
amnesty_qa["eval"],
metrics=[
context_precision,
faithfulness,
answer_relevancy,
context_recall,
context_relevancy,
answer_correctness
],
llm=gpt4
)

result
df = result.to_pandas()

df.head(10)

Running this code from your website I am getting 9/10 values of NaN for context recall:
recall_error.json

[FranciscoAlves00]

I would like to add it works better with the gpt-4 simple model and works almost perfectly with the gpt-3.5 models. But I need to run the evaluation with the gpt4 models.
Moreover, I have tried installing previous RAGAS versions and it still returns the same problems. Which is very odd, since yesterday I was being able to run the evaluations correctly.

[abhinavkashyapcrayon]

Hi, I think this is still relevant. Context Precision and context recall return Nan for Gpt-4o models