Leaderboard(Scandinavian) : Missing most results on DKHate

[x-tabdeveloping]

I think it has to do something with the fact that we fixed some model-loading issues and there are some conflict with the revisions. I will look into it.

[x-tabdeveloping]

It's the DKHateClassification scores that are missing.

[x-tabdeveloping]

Hmmm now that I'm looking at it in the current version of the Leaderboard on HF Spaces, the main branch has DKHateClassification, but the one on HF Spaces doesn't. Has it only been added recently to MTEB(Scandinavian) @KennethEnevoldsen ?

[x-tabdeveloping]

Oh no, I think I know what's going on.
Since all results were missing on DKHate, it didn't even get displayed in the table, so we didn't notice it was missing.
Now, since we fixed issues with result loading, it shows up, but we haven't run most models on it, so ti will probably have to be run...

[x-tabdeveloping]

I just checked and the results are indeed missing from our results repo.
@Muennighoff can we run DKHateClassification on these models?

models_missing_dkhate = ['Linq-Embed-Mistral',
 'GritLM-7B',
 'SFR-Embedding-2_R',
 'GritLM-8x7B',
 'gte-Qwen2-7B-instruct',
 'SFR-Embedding-Mistral',
 'e5-mistral-7b-instruct',
 'Cohere-embed-multilingual-v3.0',
 'text-embedding-3-large',
 'multilingual-e5-large-instruct',
 'gte-Qwen1.5-7B-instruct',
 'gte-Qwen2-1.5B-instruct',
 'voyage-multilingual-2',
 'bilingual-embedding-large',
 'Solon-embeddings-large-0.1',
 'jina-embeddings-v3',
 'voyage-large-2-instruct',
 'text-embedding-3-small',
 'voyage-3-lite',
 'bilingual-embedding-base',
 'Cohere-embed-multilingual-light-v3.0',
 'snowflake-arctic-embed-l-v2.0',
 'KaLM-embedding-multilingual-mini-v1',
 'stella_en_1.5B_v5',
 'voyage-3',
 'NV-Embed-v2',
 'bilingual-embedding-small',
 'KaLM-embedding-multilingual-mini-instruct-v1',
 'gte-multilingual-base',
 'NV-Embed-v1',
 'arabic-english-sts-matryoshka',
 'granite-embedding-278m-multilingual',
 'snowflake-arctic-embed-m-v2.0',
 'speed-embedding-7b-instruct',
 'Arabic-labse-Matryoshka',
 'Arabic-all-nli-triplet-Matryoshka',
 'Cohere-embed-english-v3.0',
 'paraphrase-multilingual-mpnet-base-v2',
 'LaBSE',
 'granite-embedding-107m-multilingual',
 'e5-large-v2',
 'paraphrase-multilingual-MiniLM-L12-v2',
 'e5-base-v2',
 'stella_en_400M_v5',
 'nomic-embed-text-v1',
 'sentence_croissant_alpha_v0.2',
 'mmlw-e5-large',
 'Arabic-MiniLM-L12-v2-all-nli-triplet',
 'STS-multilingual-mpnet-base-v2',
 'UAE-Large-V1',
 'sentence_croissant_alpha_v0.3',
 'bge-large-en-v1.5',
 'bge-base-en-v1.5',
 'mxbai-embed-large-v1',
 'sentence_croissant_alpha_v0.4',
 'stella-base-en-v2',
 'nomic-embed-text-v1-unsupervised',
 'e5-base-4k',
 'e5-small-v2',
 'nomic-embed-text-v1.5',
 'GIST-Embedding-v0',
 'snowflake-arctic-embed-l',
 'text2vec-base-multilingual',
 'gte-base',
 'gte-large',
 'gemma-2b-embeddings',
 'MedEmbed-small-v0.1',
 'NoInstruct-small-Embedding-v0',
 'mmlw-roberta-large',
 'bge-small-en-v1.5',
 'Cohere-embed-english-light-v3.0',
 'embedder-100p',
 'GIST-small-Embedding-v0',
 'GIST-large-Embedding-v0',
 'LaBSE-en-ru',
 'snowflake-arctic-embed-m',
 'granite-embedding-125m-english',
 'all-mpnet-base-v2',
 'gte-small',
 'snowflake-arctic-embed-s',
 'LaBSE-ru-turbo',
 'snowflake-arctic-embed-m-long',
 'mmlw-roberta-base',
 'Wartortle',
 'Bulbasaur',
 'all-MiniLM-L12-v2',
 'snowflake-arctic-embed-m-v1.5',
 'gte-micro-v4',
 'Ivysaur',
 'Venusaur',
 'KartonBERT-USE-base-v1',
 'GIST-all-MiniLM-L6-v2',
 'granite-embedding-30m-english',
 'Squirtle',
 'slx-v0.1',
 'jina-embedding-b-en-v1',
 'German_Semantic_STS_V2',
 'snowflake-arctic-embed-xs',
 'rubert-tiny2',
 'sgpt-bloom-7b1-msmarco',
 'ru-en-RoSBERTa',
 'st-polish-kartonberta-base-alpha-v1',
 'USER-base',
 'gte-micro',
 'potion-base-8M',
 'rubert-tiny-turbo',
 'rubert-tiny',
 'jina-embedding-s-en-v1',
 'rubert-base-cased-sentence',
 'potion-base-4M',
 'M2V_base_glove_subword',
 'M2V_base_output',
 'distilrubert-small-cased-conversational',
 'deberta-v1-base',
 'Arabic-mpnet-base-all-nli-triplet',
 'sbert_large_mt_nlu_ru',
 'rubert-base-cased',
 'potion-base-2M',
 'silma-embeddding-matryoshka-v0.1',
 'sbert_large_nlu_ru',
 'Marbert-all-nli-triplet-Matryoshka',
 'jina-embeddings-v2-small-en',
 'M2V_base_glove',
 'Arabert-all-nli-triplet-Matryoshka',
 'cai-lunaris-text-embeddings',
 'ternary-weight-embedding',
 'jina-embeddings-v2-base-en']

[KennethEnevoldsen]

Note that DKHate is a gated dataset, but since it has an open license we could simply reupload in MTEB (cc @Samoed I felt like we had functions for this?)

[Samoed]

Yes, it is in v2 branch

import mteb

task = mteb.get_task("DKHateClassification")
task.push_dataset_to_hub("mteb/DKHateClassification")