LLM-based evaluation

[Monstertail]

Two evaluations:
1.end-to-end evaluation: by human
2. component evaluation:
2.1 idea;
2.2 paper (draft)
2.3 review.

[Monstertail]

[End-to-end eval]
run end to end pipeline and eval by human experts;
[things to release]
paper profile;
agent profile;
env;
[Human eval] @chengzr01
prompt/criteria alignment? For prompt, select three good/bad papers to see the difference.

[Monstertail]

[Discussion about evaluation methods]

1. Component eval (evaluate each stage/components like idea/paper/review) and pipeline eval( evaluate the entire research pipeline)
2. Component eval-- 1) idea; 2) paper; 3) review; 4) discussion(?)
3. Details of eval for each component:
   idea. e.g. 10 dimensions to get an overall score: (novelty, practical, ...). Maybe input trends of research ideas for novelty evaluation from--1) real world; 2) env in research town.
   paper draft（including 'title' and 'abstract', and previous step "idea"）
   review：（data from the real world as inputs）compare with open review by ranking consistency.

[chengzr01]

Here are some references for writing prompts based on top-tier conferences in machine learning.

• NeurIPS https://neurips.cc/Conferences/2023/ReviewerGuidelines
• ICLR https://iclr.cc/Conferences/2024/ReviewerGuide
• ICML https://icml.cc/Conferences/2024/ReviewerInstructions

[Monstertail]

[Discusssion on 29 May]

1. Why did we introduce the LLM evaluator?-> automatic evaluation framework, and we hope it is somehow reliable.
2. What does human eval work for?-> make sure the alignment with LLM evaluator

[Monstertail]

[tasks to do]

1. idea prompt alignment ;
2. paper prompt alignment ;
3. review prompt design (including alignment );
4. micro-bench: idea/paper-> select 6 papers(3 good/3 bad); review-> open review ranking consistency.

[Monstertail]

Priority: 3(#105 )->4（#130 #112 ）->1/2 (#128)
@ft2023 @chengzr01

[Monstertail]

3. review prompt design: single paper --- {idea},{trend},{paper}, {review}. {regulation}. Instruction-- verify scores of 4 reviewers .

[chengzr01]

I have included some criteria in #79.

[Monstertail]

[Discussion on 29 May] Jinwei/Zirui

1. (To sell evaluators) we should add LLM evaluator for reviews, and then compare with human evaluation for the review process.
2. (To sell environment) mico-bench is a case study to prove our research pipeline( review phase) is effective. Our reviewers could provide relatively fair reviews. @ft2023