Where does this leaderboard originate from?

[zhimin-z]

Checking https://leaderboard.tabbyml.com, I found there isn't any specification/documentation about this leaderboard.

Is any repository, paper, report, or blog relevant?

[zhimin-z]

For instance, which benchmark dataset is employed for the evaluation? How is instruction-based fine-tuning configured for this evaluation?

[wsxiaoys]

Hey - the leaderboard come from an intern project running amazon cceval data on certain models, the original data / code can be found at https://github.com/TabbyML/tabby/pull/642/files for StarCoder-1B / StarCoder-3B vs WizardCoder-1B / WizardCoder-3B

The overview of method can be found in this blog post: https://tabby.tabbyml.com/blog/2023/11/23/coding-llm-leaderboard

[zhimin-z]

Thanks for your reply. Would you consider posting this blog on the leaderboard website? I found it hard to understand the leaderboard without this provided information. Thanks in advance XD

[zhimin-z]

BTW, what is the difference between the first and second leaderboards? I found they are actually referring to the same post...It seems all three leaderboards are released out of this paper.