Code

[WIP] MLM Training Objective#680

Code

Draft

prady-saligram wants to merge 7 commits intostanford-crfm:mainstanford-crfm/levanter:mainfrom JulienDarve:roberta-trainingJulienDarve/levanter:roberta-trainingCopy head branch name to clipboard

+1,249-32

Commits

Commits on Jul 30, 2024

• Implements dynamic masking objective

  prady-saligram

  committed
  8f7402e

  Copy full SHA for 8f7402e

• Implements dynamic masked dataset

  prady-saligram

  committed
  670b053

  Copy full SHA for 670b053

• Reintroduced accidentally deleted CausalLMDataset class

  prady-saligram

  committed
  42f5404

  Copy full SHA for 42f5404

Commits on Aug 5, 2024

• [WIP] Re-implements MLM training objective

  prady-saligram

  committed
  53fd8d2

  Copy full SHA for 53fd8d2

Commits on Aug 6, 2024

• Adds error handling and reverts LmExample class to original

  prady-saligram

  committed
  dcd45b2

  Copy full SHA for dcd45b2

Commits on Aug 26, 2024

• Sets RobertaConfig as model architecture and creates default config file

  prady-saligram

  committed
  027b176

  Copy full SHA for 027b176

Commits on Sep 1, 2024

• Adds compute_loss to roberta and changes positional ids to begin from 0

  prady-saligram

  committed
  399e08c

  Copy full SHA for 399e08c