Skip to content

Issues: stanford-crfm/helm

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

108 Open 979 Closed
108 Open 979 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Is there any way to load GGUF models?
#3141 opened Nov 7, 2024 by InAnYan
Support few-shot chain-of-thought in GPQA / MMLU
#3088 opened Oct 22, 2024 by yifanmai
Issue with running HEIM documentation Improvements or additions to documentation HEIM (Text2Image) user question
#3080 opened Oct 22, 2024 by sudhir-mcw
2
flake8 in Python 3.12 produces a large number of errors
#3072 opened Oct 18, 2024 by yifanmai
Optimum Intel OpenVino fails with segmentation fault bug Something isn't working models
#3066 opened Oct 16, 2024 by yifanmai
2
How to use this package when I have the pompts and images across models HEIM (Text2Image) user question
#3062 opened Oct 14, 2024 by snehith3195
1
Not able to install install-heim-extras.sh for heim leaderboard HEIM (Text2Image) user question
#3060 opened Oct 11, 2024 by snehith3195
1
Proposal: Multi-turn support
#3059 opened Oct 10, 2024 by yifanmai
Proposal: User-provided custom adapters
#3058 opened Oct 10, 2024 by yifanmai
HatefulMemesScenario get_instances returning error bug Something isn't working HEIM (Text2Image) methodology Evaluation methodology user question
#3056 opened Oct 10, 2024 by dxwu2
3
Add new model (upstage/solar-pro-preview-instruct)
#3040 opened Oct 7, 2024 by remy-rec
2
Add MMLU-Pro additions New models or scenarios good first issue Good for newcomers scenarios
#3018 opened Sep 24, 2024 by yifanmai
@siyagoel
Add amazon bedrock 3p models to the model configuration. user question
#2967 opened Sep 3, 2024 by subhaviv
2
Incorrect scoring due to answer format mismatch in MMLU evaluation user question
#2939 opened Aug 16, 2024 by DerryChan
1
Add Windows support
#2907 opened Aug 8, 2024 by yifanmai
Add sorting/filtering functionality to predictions pages frontend Frontend issues
#2887 opened Aug 1, 2024 by farzaank
HEIMHumanEvalScenario requires permissions to download data from codalab HEIM (Text2Image) user question
#2865 opened Jul 30, 2024 by slymane
1
pip needs to be temporarily downgraded to 24.1.2
#2855 opened Jul 30, 2024 by yifanmai
Sort by correctness functionality on Predictions page in front end
#2783 opened Jun 29, 2024 by farzaank
@farzaank
Annotator issues
#2697 opened May 31, 2024 by yifanmai
4 tasks
@farzaank
Don't set trust_remote_code=True by default in HuggingFaceClient
#2645 opened May 13, 2024 by yifanmai
Duplication of fields across *_schema.yaml files
#2631 opened May 8, 2024 by yifanmai
1 of 5 tasks
Fix instance level metrics being way too far to the right
#2619 opened May 7, 2024 by farzaank
@farzaank
2
helm-summarize gives extra weight to runs in multiple groups framework methodology Evaluation methodology p2 Priority 2 (Good to have for release)
#2541 opened Apr 2, 2024 by yifanmai
1
Add validation for schema YAML files help wanted
#2528 opened Mar 29, 2024 by yifanmai
8 tasks
ProTip! Add no:assignee to see everything that’s not assigned.