Code

TransformerEngine Integration#1282

Code

Merged

Quentin-Anthony merged 30 commits intoEleutherAI:mainEleutherAI/gpt-neox:mainfrom aurelion-source:teaurelion-source/gpt-neox:teCopy head branch name to clipboardDec 19, 2024

+968-171

Commits

Commits on Sep 15, 2024

• Implemented ColumnParallelLinear with Transformer-Engine

  aurelion-source

  committed
  5382c8a

• Implemented RowParallelLinear with Transformer-Engine

  aurelion-source

  committed
  fa887b7

• Implemented LayerNormMLP with Transformer-Engine

  aurelion-source

  committed
  0a6f140

Commits on Sep 16, 2024

• Implemented MultiheadAttention with Transformer-Engine

  aurelion-source

  committed
  5cba717

• Cleaned up transformer.py

  aurelion-source

  committed
  94e552c

• Cleaned up neox_args

  aurelion-source

  committed
  40e1019

• Cleaned up neox_args

  aurelion-source

  committed
  885e72c

Commits on Sep 26, 2024

• - Fixed TE_MHA and added rope support

  Show description for fe8f22a
  aurelion-source

  committed
  fe8f22a

• Fixed mixed files.

  aurelion-source

  committed
  ee42a31

Commits on Sep 27, 2024

• Implemented ColumnParallelLinear with Transformer-Engine

  aurelion-source

  committed
  8961dd7

• Implemented RowParallelLinear with Transformer-Engine

  aurelion-source

  committed
  5162d54

• Implemented LayerNormMLP with Transformer-Engine

  aurelion-source

  committed
  3cad89c

• Implemented MultiheadAttention with Transformer-Engine

  aurelion-source

  committed
  eedb6c2

• Cleaned up transformer.py

  aurelion-source

  committed
  36ad680

• Cleaned up neox_args

  aurelion-source

  committed
  6963103

• Cleaned up neox_args

  aurelion-source

  committed
  afc9c92

• - Fixed TE_MHA and added rope support

  Show description for a0e7acd
  aurelion-source

  committed
  a0e7acd

• Fixed mixed files.

  aurelion-source

  committed
  0b4bdc5

• Merge branch 'te' of github.com:aurelion-source/gpt-neox into te

  aurelion-source

  committed
  d559be9

• Changed get_linear name

  aurelion-source

  committed
  bb76510

Commits on Oct 1, 2024

• Added rng tracker to lnmlp and placed rope in te_mha init instead of forward

  aurelion-source

  committed
  43cf4ee

• Updated fp8 arguments to te_fp8

  aurelion-source

  committed
  42716f2

• Added EAI copyright

  aurelion-source

  committed
  b3255e6

Commits on Oct 8, 2024

• Merge branch 'main' into te

  Quentin-Anthony

  authored
  afeff03

• precommit

  Quentin-Anthony

  committed
  98f0388

• add sample TE config

  Quentin-Anthony

  committed
  7e7dbfb

• add te to readme

  Quentin-Anthony

  committed
  5757be6

• remove pip install prefix from reqs file

  Quentin-Anthony

  committed
  9ea3dcf

Commits on Oct 16, 2024

• Force TE pytorch in requirements file

  aurelion-source

  committed
  f3e40e9

Commits on Dec 19, 2024

• Merge remote-tracking branch 'upstream/main' into te

  Quentin-Anthony

  committed
  7bf9c44