TypeError(unsupported operand type(s) for +=: 'dict' and 'dict')

[jinnoh47]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
KeyboardInterrupt: Exception raised in Job[1]: TypeError(unsupported operand type(s) for +=: 'dict' and 'dict')
I'm assuming there is a bug in evaluation mertic

Ragas version: 0.1.20
Python version: 3.11.3 | packaged by conda-forge

Code to Reproduce
Share code to reproduce the issue
https://gist.github.com/jinnoh47/f58dae6aabb007bd4b5a356807ec5695

Error trace

---

KeyboardInterrupt Traceback (most recent call last)
Cell In[25], line 54
52 data_point = gpt_llm_dataset[i]
53 try:
---> 54 result = evaluate(Dataset.from_dict({
55 "question": [data_point["question"]],
56 "contexts": [data_point["contexts"]],
57 "answer": [data_point["answer"]],
58 "ground_truth": [data_point["ground_truth"]]
59 }), samoo_llm_metrics)
60 results.append(result)
61 except Exception as e:

File /opt/conda/lib/python3.11/site-packages/ragas/_analytics.py:129, in track_was_completed..wrapper(*args, **kwargs)
126 @wraps(func)
127 def wrapper(*args: P.args, **kwargs: P.kwargs) -> t.Any:
128 track(IsCompleteEvent(event_type=func.name, is_completed=False))
--> 129 result = func(*args, **kwargs)
130 track(IsCompleteEvent(event_type=func.name, is_completed=True))
132 return result

File /opt/conda/lib/python3.11/site-packages/ragas/evaluation.py:295, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, run_config, token_usage_parser, raise_exceptions, column_map, show_progress)
292 scores = []
293 try:
294 # get the results
--> 295 results = executor.results()
296 if results == []:
297 raise ExceptionInRunner()

File /opt/conda/lib/python3.11/site-packages/ragas/executor.py:118, in Executor.results(self)
114 results.append(r)
116 return results
--> 118 results = asyncio.run(_aresults())
119 sorted_results = sorted(results, key=lambda x: x[0])
120 return [r[1] for r in sorted_results]

File /opt/conda/lib/python3.11/site-packages/nest_asyncio.py:35, in _patch_asyncio..run(main, debug)
33 task = asyncio.ensure_future(main)
34 try:
---> 35 return loop.run_until_complete(task)
36 finally:
37 if not task.done():

File /opt/conda/lib/python3.11/site-packages/nest_asyncio.py:84, in _patch_loop..run_until_complete(self, future)
82 f._log_destroy_pending = False
83 while not f.done():
---> 84 self._run_once()
85 if self._stopping:
86 break

File /opt/conda/lib/python3.11/site-packages/nest_asyncio.py:107, in _patch_loop.._run_once(self)
100 heappop(scheduled)
102 timeout = (
103 0 if ready or self._stopping
104 else min(max(
105 scheduled[0]._when - self.time(), 0), 86400) if scheduled
106 else None)
--> 107 event_list = self._selector.select(timeout)
108 self._process_events(event_list)
110 end_time = self.time() + self._clock_resolution

File /opt/conda/lib/python3.11/selectors.py:468, in EpollSelector.select(self, timeout)
466 ready = []
467 try:
--> 468 fd_event_list = self._selector.poll(timeout, max_ev)
469 except InterruptedError:
470 return ready

KeyboardInterrupt:
Exception raised in Job[1]: TypeError(unsupported operand type(s) for +=: 'dict' and 'dict')

Expected behavior

This code is expected to evaluate the dataset using the specified metrics (faithfulness, answer_relevancy, answer_correctness) and handle any errors that occur during the evaluation process. The results should be appended to the results list.

Additional context
Add any other context about the problem here.

[dosubot]

The TypeError you're encountering, "unsupported operand type(s) for +=: 'dict' and 'dict'," typically occurs when there is an attempt to use the += operator on two dictionaries, which is not supported in Python. This issue might be related to how results are being aggregated or returned within the evaluate function or the metrics being used.

A similar issue was discussed in the Ragas repository, where a TypeError involving dictionary operations was caused by inconsistent value types for the same key in the dictionaries being merged ^[1]. In your case, it seems like the error might be occurring during the evaluation process when trying to append results to a list.

To resolve this, you should check the part of your code where dictionaries are being combined or aggregated. Ensure that you are not using the += operator with dictionaries. Instead, you can use the update() method to merge dictionaries, like so:

dict1.update(dict2)

This will add the key-value pairs from dict2 into dict1.

Additionally, ensure that the data types for each key in the dataset dictionary passed to the evaluate function are as expected:

• question: list[str]
• contexts: list[list[str]]
• answer: list[str]
• ground_truth: list[list[str]] ^[2].

If the issue persists, you might want to review the specific metrics being used and how they handle the results.

To continue talking to Dosu, mention @dosu.

[dosubot]

Could you provide more details about how the evaluate function is being used in your code? Specifically, it would be helpful to see how the results from the evaluate function are being handled or aggregated. Additionally, if you have any custom metrics or modifications to the evaluate function, please share those as well. This information will help in identifying the source of the issue.

To continue talking to Dosu, mention @dosu.

[dosubot]

Could you describe the specific part of the code where the evaluate function is called and how its results are being processed? Additionally, if there are any custom metrics or configurations being used with the evaluate function, please provide details on those as well. This information will help in diagnosing the issue more effectively.

To continue talking to Dosu, mention @dosu.

[dosubot]

The TypeError you're encountering might be due to how the results from the evaluate function are being aggregated. Since you're using the evaluate function from the Ragas library, which returns a Result object containing scores in a dataset format, ensure that you're handling these results correctly.

Instead of using += to combine results, you should append each Result object to a list or convert it to a dictionary using the to_pandas method and then handle it accordingly. Here's a general approach:

1. Append Results: If you're storing results in a list, simply append each Result object:

   results.append(result)

2. Convert to DataFrame: If you need to manipulate the results further, convert them to a pandas DataFrame:

   df = result.to_pandas()

3. Ensure Correct Data Types: Make sure the data types for each key in the dataset dictionary passed to the evaluate function are as expected:

   • question: list[str]
   • contexts: list[list[str]]
   • answer: list[str]
   • ground_truth: list[list[str]]

By following these steps, you should be able to handle the results without encountering the TypeError. If the issue persists, double-check the logic where results are being processed to ensure no unsupported operations are being performed on dictionaries.

To continue talking to Dosu, mention @dosu.