Race condition when running alpaca eval?

[GeorgiosSmyrnis]

Alpaca eval seems to save annotations under json files in eval/chat_benchmarks/alpaca_eval/src/alpaca_eval/evaluators_configs - this leads to a race condition if multiple processes try to use the same annotator at once.

This issue can be circumvented by serializing evaluation, but posting this here in case there is a better way.