Merge pull request #238 from Elfsong/mercury

Add a new dataset Mercury

README.md

@@ -38,6 +38,7 @@ Below are the features and tasks of this framework:
     - [SantaCoder-FIM](https://huggingface.co/datasets/bigcode/santacoder-fim-task) for evaluating FIM on **Python** code using Exact Match. Further details are described in [SantaCoder](https://arxiv.org/abs/2301.03988). Includes two tasks:
         - `StarCoderFIM`: which uses the default FIM tokens `"<fim_prefix>", "<fim_middle>", "<fim_suffix>"`, and
         - `SantaCoderFIM`: which uses SantaCoder FIM tokens `"<fim-prefix>", "<fim-middle>", "<fim-suffix>"`
+    - [Mercury](https://huggingface.co/datasets/Elfsong/Mercury) for evaluating computational efficiency of **Python** code generation.
 
 More details about each task can be found in  the documentation in [`docs/README.md`](https://github.com/bigcode-project/bigcode-evaluation-harness/blob/main/docs/README.md).
 ## Setup

bigcode_eval/tasks/__init__.py

@@ -5,7 +5,7 @@
                concode, ds1000, gsm, humaneval, humanevalplus, humanevalpack,
                instruct_humaneval, instruct_wizard_humaneval, mbpp, mbppplus,
                multiple, parity, python_bugs, quixbugs, recode, santacoder_fim,
-               studenteval)
+               studenteval, mercury)
 
 TASK_REGISTRY = {
     **apps.create_all_tasks(),
@@ -30,6 +30,7 @@
     **recode.create_all_tasks(),
     **santacoder_fim.create_all_tasks(),
     "studenteval": studenteval.StudentEval,
+    "mercury": mercury.Mercury,
 }
 
 ALL_TASKS = sorted(list(TASK_REGISTRY))