Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TapeAgents export for experiment results #238

Open
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

ollmer
Copy link
Contributor

@ollmer ollmer commented Nov 4, 2024

This PR introduces a simple compatibility layer for exporting traces of the BrowserGym experiments in a format compatible with the Tapes of the TapeAgents framework.
What does it do:

  • convert a single trace of the experiment into a single Tape
  • decomposes each StepInfo step of the experiment into 3 steps of the tape agents: observation, thought, and action.
  • attempts to preserve as much metadata as possible for the step and for the whole trace.
  • parses action in the simplest possible way as a string with a single Python function call
  • if the think field is present in the agent info and not empty - it creates the thought step of the tape from it.
  • method to save the tape in json format

The MVP is ready but not yet tested.

Next steps in this PR:

  • test conversion over the real traces
  • collect feedback from the BrowserGym team
  • test tape loading on the TapeAgents side and rendering it with the tape browser

cc @recursix.

@ollmer ollmer changed the title [WIP] TapeAgents compatibility layer for experiment results TapeAgents export for experiment results Nov 8, 2024
@ollmer ollmer marked this pull request as ready for review November 8, 2024 15:49
@ollmer
Copy link
Contributor Author

ollmer commented Nov 8, 2024

I've simplified this PR down to <100 lines of changes in a single file and added support of tape_segments inside the agent_info, which could be produced by TapeAgent in the AgentLab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants