-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
4 changed files
with
2,909 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,276 @@ | ||
| <!DOCTYPE html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="UTF-8"> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | ||
| <meta http-equiv="X-UA-Compatible" content="ie=edge"> | ||
| <title>LM Pub Quiz</title> | ||
| <link rel="stylesheet" href="style.css"> | ||
| </head> | ||
| <body> | ||
| </body> | ||
| <head> | ||
| <meta charset="UTF-8"> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | ||
| <title>LM Pub Quiz</title> | ||
|
|
||
| <link rel="stylesheet" href="https://fonts.xz.style/serve/inter.css"> | ||
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@exampledev/[email protected]/new.min.css"> | ||
| <link rel="stylesheet" href="style.css"> | ||
| <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/font/bootstrap-icons.min.css"> | ||
| </head> | ||
| <body> | ||
| <header> | ||
| <h1>LM Pub Quiz</h1> | ||
| <h2 class="subtitle">Evaluating language models using multiple choice items</h2> | ||
| <nav> | ||
| <a href=""><i class="bi bi-git"></i> Library</a> / | ||
| <a href=""><i class="bi bi-database"></i> BEAR Dataset</a> / | ||
| <a href=""><i class="bi bi-file-earmark"></i> Paper</a> / | ||
| <a href=""><i class="bi bi-file-zip"></i> Raw Results</a> | ||
| </nav> | ||
| </header> | ||
|
|
||
| <section> | ||
| <figure class="shadow-box"> | ||
| <img src="./media/bear_evaluation_final.svg" width="100%" alt="Illustration of how LM Pub Quiz evaluates LMs."> | ||
| <figcaption>Illustration of how LM Pub Quiz evaluates LMs: Answers are ranked by the (pseudo) log-likelihoods of the textual statements derived from all of the answer options.</figcaption> | ||
| </figure> | ||
| </section> | ||
|
|
||
| <section class="shadow-box"> | ||
| <div style="text-align: right;"><span class="badge acl">Accepted at NAACL 2024</span></div> | ||
| <h2>BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models</h2> | ||
| <h3>Abstract</h3> | ||
| <p> | ||
| Knowledge probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Probing is an inexpensive way to compare LMs of different sizes and training configurations. However, previous approaches rely on the objective function used in pre-training LMs and are thus applicable only to masked or causal LMs. As a result, comparing different types of LMs becomes impossible. To address this, we propose an approach that uses an LM's inherent ability to estimate the log-likelihood of any given textual statement. We carefully design an evaluation dataset of 7,731 instances (40,916 in a larger variant) from which we produce alternative statements for each relational fact, one of which is correct. We then evaluate whether an LM correctly assigns the highest log-likelihood to the correct statement. Our experimental evaluation of 22 common LMs shows that our proposed framework, BEAR, can effectively probe for knowledge across different LM types. We release the BEAR datasets and an open-source framework that implements the probing approach to the research community to facilitate the evaluation and development of LMs. | ||
| </p> | ||
| <p> | ||
| <a href=""> | ||
| <button><i class="bi bi-file-earmark"></i> Read the Paper</button> | ||
| </a> | ||
| </p> | ||
| </section> | ||
| <section> | ||
| <figure class="shadow-box"> | ||
| <img src="./media/accuracy_by_model_size_bear.svg" width="100%" alt="Illustration of how LM Pub Quiz evaluates LMs." style="max-width: 600px;"> | ||
| <figcaption>Accuracy of various models on the BEAR dataset.</figcaption> | ||
| </figure> | ||
| </section> | ||
| <section class="shadow-box"> | ||
| <h2>Model Results</h2> | ||
| <p> | ||
| We evaluated 22 lanuages models (of various sizes, trained using different pretraining objectives, and of both causal and masked LM types) on the BEAR dataset. | ||
| </p> | ||
|
|
||
| <table border="1" class="dataframe"> | ||
| <thead> | ||
| <tr style="text-align: left;"> | ||
| <th></th> | ||
| <th>Type</th> | ||
| <th>Num Params</th> | ||
| <th>BEAR</th> | ||
| <th>BEAR<sub>1:1</sub></th> | ||
| <th>BEAR<sub>N:1</sub></th> | ||
| </tr> | ||
| <tr> | ||
| <th>Model</th> | ||
| <th></th> | ||
| <th></th> | ||
| <th></th> | ||
| <th></th> | ||
| <th></th> | ||
| </tr> | ||
| </thead> | ||
| <tbody> | ||
| <tr> | ||
| <th>Llama-2-13b-hf</th> | ||
| <td>CLM</td> | ||
| <td>13b</td> | ||
| <td> 66.9% ±<small> 1.0%</small></td> | ||
| <td> 66.5% ±<small> 1.6%</small></td> | ||
| <td> 67.0% ±<small> 1.1%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>Mistral-7B-v0.1</th> | ||
| <td>CLM</td> | ||
| <td>7.0b</td> | ||
| <td> 65.4% ±<small> 1.1%</small></td> | ||
| <td> 64.5% ±<small> 1.2%</small></td> | ||
| <td> 65.5% ±<small> 1.1%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>gemma-7b</th> | ||
| <td>CLM</td> | ||
| <td>7.0b</td> | ||
| <td> 63.7% ±<small> 1.3%</small></td> | ||
| <td> 63.5% ±<small> 0.7%</small></td> | ||
| <td> 63.8% ±<small> 1.4%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>Llama-2-7b-hf</th> | ||
| <td>CLM</td> | ||
| <td>7.0b</td> | ||
| <td> 62.4% ±<small> 1.3%</small></td> | ||
| <td> 62.2% ±<small> 1.1%</small></td> | ||
| <td> 62.4% ±<small> 1.3%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>gemma-2b</th> | ||
| <td>CLM</td> | ||
| <td>2.0b</td> | ||
| <td> 51.5% ±<small> 1.0%</small></td> | ||
| <td> 53.1% ±<small> 1.3%</small></td> | ||
| <td> 51.3% ±<small> 1.0%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-30b</th> | ||
| <td>CLM</td> | ||
| <td>30b</td> | ||
| <td> 47.9% ±<small> 0.5%</small></td> | ||
| <td> 45.8% ±<small> 1.0%</small></td> | ||
| <td> 48.2% ±<small> 0.6%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-13b</th> | ||
| <td>CLM</td> | ||
| <td>13b</td> | ||
| <td> 45.4% ±<small> 0.8%</small></td> | ||
| <td> 43.5% ±<small> 2.1%</small></td> | ||
| <td> 45.7% ±<small> 0.6%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-6.7b</th> | ||
| <td>CLM</td> | ||
| <td>6.7b</td> | ||
| <td> 43.8% ±<small> 1.1%</small></td> | ||
| <td> 42.5% ±<small> 1.0%</small></td> | ||
| <td> 43.9% ±<small> 1.2%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-2.7b</th> | ||
| <td>CLM</td> | ||
| <td>2.7b</td> | ||
| <td> 37.3% ±<small> 0.9%</small></td> | ||
| <td> 35.6% ±<small> 0.7%</small></td> | ||
| <td> 37.5% ±<small> 1.0%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-1.3b</th> | ||
| <td>CLM</td> | ||
| <td>1.3b</td> | ||
| <td> 31.5% ±<small> 0.8%</small></td> | ||
| <td> 31.3% ±<small> 0.6%</small></td> | ||
| <td> 31.5% ±<small> 0.9%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>gpt2-xl</th> | ||
| <td>CLM</td> | ||
| <td>1.6b</td> | ||
| <td> 26.2% ±<small> 0.7%</small></td> | ||
| <td> 24.1% ±<small> 1.6%</small></td> | ||
| <td> 26.5% ±<small> 0.6%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>gpt2-large</th> | ||
| <td>CLM</td> | ||
| <td>812M</td> | ||
| <td> 22.2% ±<small> 0.6%</small></td> | ||
| <td> 20.1% ±<small> 1.8%</small></td> | ||
| <td> 22.5% ±<small> 0.5%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>roberta-large</th> | ||
| <td>MLM</td> | ||
| <td>355M</td> | ||
| <td> 21.5% ±<small> 0.8%</small></td> | ||
| <td> 22.0% ±<small> 1.1%</small></td> | ||
| <td> 21.5% ±<small> 0.8%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>bert-large-cased</th> | ||
| <td>MLM</td> | ||
| <td>335M</td> | ||
| <td> 19.9% ±<small> 0.5%</small></td> | ||
| <td> 16.6% ±<small> 1.0%</small></td> | ||
| <td> 20.3% ±<small> 0.5%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-350m</th> | ||
| <td>CLM</td> | ||
| <td>350M</td> | ||
| <td> 19.6% ±<small> 0.6%</small></td> | ||
| <td> 18.6% ±<small> 1.2%</small></td> | ||
| <td> 19.7% ±<small> 0.6%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>gpt2-medium</th> | ||
| <td>CLM</td> | ||
| <td>355M</td> | ||
| <td> 19.0% ±<small> 0.8%</small></td> | ||
| <td> 16.0% ±<small> 2.6%</small></td> | ||
| <td> 19.4% ±<small> 0.6%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>bert-base-cased</th> | ||
| <td>MLM</td> | ||
| <td>109M</td> | ||
| <td> 18.4% ±<small> 0.4%</small></td> | ||
| <td> 15.0% ±<small> 1.1%</small></td> | ||
| <td> 18.8% ±<small> 0.4%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>roberta-base</th> | ||
| <td>MLM</td> | ||
| <td>125M</td> | ||
| <td> 16.4% ±<small> 0.7%</small></td> | ||
| <td> 15.8% ±<small> 1.8%</small></td> | ||
| <td> 16.5% ±<small> 0.8%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>opt-125m</th> | ||
| <td>CLM</td> | ||
| <td>125M</td> | ||
| <td> 16.4% ±<small> 0.5%</small></td> | ||
| <td> 14.0% ±<small> 1.3%</small></td> | ||
| <td> 16.7% ±<small> 0.4%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>xlm-roberta-large</th> | ||
| <td>MLM</td> | ||
| <td>561M</td> | ||
| <td> 14.3% ±<small> 0.3%</small></td> | ||
| <td> 14.9% ±<small> 1.7%</small></td> | ||
| <td> 14.3% ±<small> 0.5%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>gpt2</th> | ||
| <td>CLM</td> | ||
| <td>137M</td> | ||
| <td> 13.5% ±<small> 0.8%</small></td> | ||
| <td>  9.4% ±<small> 2.1%</small></td> | ||
| <td> 14.0% ±<small> 0.7%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>xlm-roberta-base</th> | ||
| <td>MLM</td> | ||
| <td>279M</td> | ||
| <td> 11.4% ±<small> 0.2%</small></td> | ||
| <td> 11.4% ±<small> 1.1%</small></td> | ||
| <td> 11.4% ±<small> 0.2%</small></td> | ||
| </tr> | ||
| <tr> | ||
| <th>Random Baseline</th> | ||
| <td>-</td> | ||
| <td>-</td> | ||
| <td>  4.7%</td> | ||
| <td>  1.7%</td> | ||
| <td>  5.1%</td> | ||
| </tr> | ||
| </tbody> | ||
| </table> | ||
| </section> | ||
| <section class="shadow-box"> | ||
| <h2>Citation</h2> | ||
| <p>When using the dataset or library, please cite the following paper:</p> | ||
| <pre>@inproceedings{wiland-ploner-akbik-2024-bear, | ||
| title = "BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models", | ||
| author = "Wiland, Jacek and Ploner, Max and Akbik, Alan", | ||
| booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024", | ||
| year = "2024", | ||
| publisher = "Association for Computational Linguistics", | ||
| }</pre> | ||
| </section> | ||
| </body> | ||
| </html> |
Oops, something went wrong.