diff --git a/README.md b/README.md index 68bd7a3..0891a19 100644 --- a/README.md +++ b/README.md @@ -44,17 +44,17 @@ For more details, please visit our [leaderboard]() (Coming Soon). ## ⏬ Download -You can download the dataset from the [HuggingFace Page](https://huggingface.co/datasets/OpenDFM/MULTI-Benchmark). Current [version](https://huggingface.co/datasets/OpenDFM/MULTI-Benchmark/blob/main/MULTI_v1.2.2_20240212_release.zip) is `v1.2.2`. Unzip the files and put them under `data`. +You can simply download data using the following command: -``` -wget https://huggingface.co/datasets/OpenDFM/MULTI-Benchmark/resolve/main/MULTI_v1.2.2_20240212_release.zip -unzip MULTI_v1.2.2_20240212_release.zip -d ./data/ +```shell +cd eval +python download_data.py ``` -The structure of `data` should be something like: +The structure of `./data` should be something like: ``` -data +./data ├── images # folder containing images ├── problem_v1.2.2_20240212_release.json # MULTI ├── knowledge_v1.2.2_20240212_release.json # MULTI-Extend diff --git a/README_zh.md b/README_zh.md index 6f303ca..566d663 100644 --- a/README_zh.md +++ b/README_zh.md @@ -44,17 +44,17 @@ ## ⏬ 下载 -您可以从[HuggingFace页面](https://huggingface.co/datasets/OpenDFM/MULTI-Benchmark)下载数据集。最新[版本](https://huggingface.co/datasets/OpenDFM/MULTI-Benchmark/blob/main/MULTI_v1.2.2_20240212_release.zip)为`v1.2.2`。解压文件并将它们放置在`data`下。 +您只需使用以下命令即可下载数据: -``` -wget https://huggingface.co/datasets/OpenDFM/MULTI-Benchmark/resolve/main/MULTI_v1.2.2_20240212_release.zip -unzip MULTI_v1.2.2_20240212_release.zip -d ./data/ +```shell +cd eval +python download_data.py ``` -`data` 的结构应该如下所示: +`./data` 的结构应该如下所示: ``` -data +./data ├── images # 包含图片的文件夹 ├── problem_v1.2.2_20240212_release.json # MULTI ├── knowledge_v1.2.2_20240212_release.json # MULTI-Extend diff --git a/data/README.md b/data/README.md deleted file mode 100644 index 720a699..0000000 --- a/data/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# this README will be deprecated soon - -# Overview - -This document will guide you to fetch our benchmark. - -## Use Google Drive - -We directly provide our file in this [link](). This includes a `problem_v1.2.0_20231217.json` containing the benchmark data, and a `knowledge_v1.2.0_20231217.json` containing the knowledge file. - -## Use HuggingFace - -We also provide our dataset on huggingface. To match the huggingface format, there are some slight changes to the dataset. To get the identical data structure for evaluation, follow the following steps: - -```python -from datasets import load_dataset -# load data from huggingface -hf_data = load_dataset("") # TODO: huggingface path here -data, knowledge_data = hf_data["data"], hf_data["kn"] -data = {question["problem_id"]: question for question in data} -knowledge = {kn["knowledge"]: kn for kn in knowledge_data} - -import json -with open("xmulti_benchmark.json", "w", encoding="utf-8") as fw: # whatever filename - json.dump(data, fw, ensure_ascii=False, indent=4) -with open("xmulti_knowledge.json", "w", encoding="utf-8") as fw: # whatever filename - json.dump(knowledge, fw, ensure_ascii=False, indent=4) - -``` - -and then the output file contains the original format of data. \ No newline at end of file diff --git a/eval/download_data.py b/eval/download_data.py new file mode 100644 index 0000000..6dd301b --- /dev/null +++ b/eval/download_data.py @@ -0,0 +1,15 @@ +from datasets import load_dataset +import os +import shutil + +if os.path.exists("../cache"): + shutil.rmtree("../cache") +os.makedirs("../cache") + +load_dataset("OpenDFM/MULTI-Benchmark", cache_dir="../cache") + +random_string = os.listdir("../cache/downloads/extracted")[0] + +shutil.copytree(f"../cache/downloads/extracted/{random_string}/", "../data/", dirs_exist_ok=True) + +shutil.rmtree("../cache")