Code

Fix tokenizer note overlaps#7

Code

Merged

loubbrad merged 25 commits intoEleutherAI:mainEleutherAI/aria-amt:mainfrom loubbrad:fix_tokenizerloubbrad/aria-amt:fix_tokenizerCopy head branch name to clipboardFeb 21, 2024

+166-86

Commits

Commits on Feb 17, 2024

• skeleton

  loubbrad

  committed
  a701f95

• convert train script from aria - not tested

  loubbrad

  committed
  035f7be

Commits on Feb 18, 2024

• msg

  loubbrad

  committed
  7b558e3

• add cli

  loubbrad

  committed
  ec7358c

• fix

  loubbrad

  committed
  4dfb1f8

• fix

  loubbrad

  committed
  4dd7092

• fix log

  loubbrad

  committed
  f8fa148

• fix dataset padding

  loubbrad

  committed
  d9b2c6c

• add bos tok

  loubbrad

  committed
  ba398f4

• remove mp test

  loubbrad

  committed
  c30e977

Commits on Feb 19, 2024

• fix config

  

  Louis Bradshaw

  committed
  0fc1e26

• fix modelconfig

  loubbrad

  committed
  c097f75

• Merge branch 'train' of github.com:loubbrad/aria-amt into main

  

  Louis Bradshaw

  committed
  161f251

• add infer

  loubbrad

  committed
  6f50990

• Merge branch 'train' of github.com:loubbrad/aria-amt into main

  

  Louis Bradshaw

  committed
  36ba72a

• add sample cli

  loubbrad

  committed
  f097257

• Merge branch 'train' of github.com:loubbrad/aria-amt into main

  

  Louis Bradshaw

  committed
  3efcd91

• fix

  loubbrad

  committed
  bd0d813

• Merge branch 'train' of github.com:loubbrad/aria-amt into main

  

  Louis Bradshaw

  committed
  00597ad

Commits on Feb 20, 2024

• implement training and inference

  

  Louis Bradshaw

  committed
  748a27d

• fix format

  

  Louis Bradshaw

  committed
  f01a99c

• Merge branch 'train' into main

  

  Louis Bradshaw

  committed
  4881af4

• Merge branch 'main' of github.com:loubbrad/aria-amt into main

  

  Louis Bradshaw

  committed
  0b0ae1b

Commits on Feb 21, 2024

• fix tokenizer

  loubbrad

  committed
  fccb2f8

• fix train

  loubbrad

  committed
  59260fa