-
Notifications
You must be signed in to change notification settings - Fork 668
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can't generate testdataset, always connection error and event loops is closed. #1381
Comments
@KylinMountain Hello there! I'm here to help you with any bugs, questions, or contributions you may have. Let's get to the bottom of this together! I found some similar issues that might help you resolve the "Event loop is closed" error and connection issues:
To resolve the "Event loop is closed" error in your specific case, you can modify your code to ensure the event loop is properly managed. Here is an example: import asyncio
from ragas.testset.generator import TestsetGenerator
async def main():
generator_llm = ChatOpenAI(model_name="phi3.5:latest", openai_api_base="http://localhost:11434/v1", temperature=0.7)
critic_llm = ChatOpenAI(model_name="phi3.5:latest", openai_api_base="http://localhost:11434/v1", temperature=0.7)
hf_embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-base-zh-v1.5", model_kwargs={"device": "mps"})
generator = TestsetGenerator.from_langchain(
generator_llm,
critic_llm,
hf_embeddings,
run_config=RunConfig(max_workers=1, seed=42, max_retries=3, log_tenacity=True, timeout=180),
)
testset = await generator.generate_with_langchain_docs(
documents=documents,
test_size=10,
distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25},
run_config=RunConfig(max_workers=1, seed=42, max_retries=1, log_tenacity=True),
with_debugging_logs=True,
is_async=True
)
ds = testset.to_dataset()
ds.save_to_disk("./activity_testset")
if __name__ == "__main__":
asyncio.run(main()) This should help manage the event loop correctly and avoid the "Event loop is closed" error [1][2][3][4]. |
@shahules786 Do you know why? I am always getting this error, it's upset. |
hey @KylinMountain that is a bummer. We have a new version of testset generation that is out - will you consider trying that out? you will have to install it from source though |
@jjmachan I would like to try, is there any instructions how to use new version? Is it same with current api? |
Adding this helped in my case. |
[x] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
LLM is started by ollama, so there's no connection issue and it is able to extract key phrase, but it always failed in generating.
Ragas version: latest source code.
Python version: 3.10
Code to Reproduce
Error trace
Expected behavior
Should be able to generate testdataset.
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: