Code

[DRAFT] Quac Benchmark - ASET#333

Code

Draft

lauritowal wants to merge 18 commits intoUKGovernmentBEIS:mainUKGovernmentBEIS/inspect_ai:mainfrom lauritowal:quaclauritowal/inspect_ai:quacCopy head branch name to clipboard

+330

Commits

Commits on Sep 1, 2024

• experimenting with dataset

  lauritowal

  committed
  3e7093f

  Copy full SHA for 3e7093f

• format dataset for inspect

  lauritowal

  committed
  feb38dc

  Copy full SHA for feb38dc

• add comments

  lauritowal

  committed
  49acfa8

  Copy full SHA for 49acfa8

• fix joint target answers

  lauritowal

  committed
  f9fc742

  Copy full SHA for f9fc742

• use f1 scorer from drop benchmark

  lauritowal

  committed
  b67ef05

  Copy full SHA for b67ef05

• add comment from paper about evaluation

  lauritowal

  committed
  cb5f937

  Copy full SHA for cb5f937

• add totods

  lauritowal

  authored
  21490ff

  Copy full SHA for 21490ff

Commits on Sep 5, 2024

• make custom f1 implementation more similar to paper description

  lauritowal

  committed
  d542385

  Copy full SHA for d542385

• add test for quac

  lauritowal

  committed
  7925a66

  Copy full SHA for 7925a66

• add human f1 score aiming to filter out states under 0.4

  lauritowal

  committed
  dc7a6d0

  Copy full SHA for dc7a6d0

• add Readme + formatting

  lauritowal

  committed
  29472f0

  Copy full SHA for 29472f0

• add original name to readme

  lauritowal

  committed
  0979951

  Copy full SHA for 0979951

• remove test folder for now

  lauritowal

  committed
  905dc0a

  Copy full SHA for 905dc0a

• remove unused file

  lauritowal

  committed
  8d385a6

  Copy full SHA for 8d385a6

• cleanup ruff

  lauritowal

  committed
  f0c81f0

  Copy full SHA for f0c81f0

• remove wrong import

  lauritowal

  committed
  8b5d39c

  Copy full SHA for 8b5d39c

• add docs

  lauritowal

  committed
  56a997b

  Copy full SHA for 56a997b

Commits on Sep 6, 2024

• Update README.md

  Show description for 9b798fa
  lauritowal

  authored
  9b798fa

  Copy full SHA for 9b798fa