Need for resetting in run som scripts

[sanjari-orb]

Hi, I am looking at the scripts run_reddit_som.sh, run_shopping_som.sh, run_classifieds_som.sh. IIUC, they all involve creating batches of indices and the docker gets reset between each of these batches.

visualwebarena/scripts/run_reddit_som.sh

Line 21 in b56b6d8

bash scripts/reset_reddit.sh

However, I think more than one example with require_reset: True can occur in every batch based on the raw config JSON files (eg: https://github.com/web-arena-x/visualwebarena/blob/main/config_files/vwa/test_classifieds.raw.json).

If that is the case, what is the point of resetting and how are we ensuring correctness of the run scripts?

[kohjingyu]

For Classifieds we implemented the per example reset, but there's no way at present to do it fast enough for shopping/reddit (about 2 mins for a full reset). Resetting the environment after every batch is kind of a compromise between resetting only at the end of each run (like WebArena recommends, which can sometimes lead to intermediate examples not working properly, e.g., if the cart is full by example 50) and resetting after every single example (which takes too long).

Hope that helps!

[sanjari-orb]

I am not sure I follow, why are we resetting when require_reset=False instead of True? (

visualwebarena/browser_env/envs.py

Line 155 in b56b6d8

if instance_config.get("require_reset", False):

)

Is there some measure of how much variance in the benchmark can accumulate in this batched reset setting? Does the batch size you selected ensure that the all the examples will always work? (as opposed to not resetting at all)