Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPT3 MoE #4

Open
wants to merge 53 commits into
base: main
Choose a base branch
from
Open

GPT3 MoE #4

wants to merge 53 commits into from

Commits on Jun 26, 2024

  1. XGLM work in progress: Causal Attention and Positional Embeddings work

    @AleHD
    AleHD committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    7aa3940 View commit details
    Browse the repository at this point in the history

  2. WIP: GPT arch almost done, hf->nt converters working perfectly for no… 

    …n-distributed inference
    @AleHD
    AleHD committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    78dd53c View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Added hf2nt frontend + tested training

    @AleHD
    AleHD committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    a74c71a View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Added nt2hf conversion + tests :)

    @AleHD
    AleHD committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    04eaef9 View commit details
    Browse the repository at this point in the history

  2. precommit

    @AleHD
    AleHD committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    138da5f View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. Merge pull request #1 from swiss-ai/gpt 

    GPT-3 and XGLM implementation
    @negar-foroutan
    negar-foroutan authored Jul 15, 2024
    Configuration menu
    Copy the full SHA
    35c43f7 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Added MultilingualNanoset Config

    @TJ-Solergibert
    TJ-Solergibert committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    0485fd6 View commit details
    Browse the repository at this point in the history

  2. Added MultilingualNanoset

    @TJ-Solergibert
    TJ-Solergibert committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    539832a View commit details
    Browse the repository at this point in the history

  3. Added Language token

    @TJ-Solergibert
    TJ-Solergibert committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    d9f0670 View commit details
    Browse the repository at this point in the history

  4. Forgot the trainer ups

    @TJ-Solergibert
    TJ-Solergibert committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    efe8720 View commit details
    Browse the repository at this point in the history

  5. Fix minor errors. Everything works

    @TJ-Solergibert
    TJ-Solergibert committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    25ad39b View commit details
    Browse the repository at this point in the history

  6. Updated config file with GPT2 tokenized datasets in RCP

    @TJ-Solergibert
    TJ-Solergibert committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    d91f9e1 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. Before lunch

    @TJ-Solergibert
    TJ-Solergibert committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    d0c14e3 View commit details
    Browse the repository at this point in the history

  2. After lunch

    @TJ-Solergibert
    TJ-Solergibert committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    9cfc5ea View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. Ready

    @TJ-Solergibert
    TJ-Solergibert committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    eed7bce View commit details
    Browse the repository at this point in the history

  2. Merge pull request #2 from TJ-Solergibert/multilingual_nanoset 

    Multilingual Nanoset
    @negar-foroutan
    negar-foroutan authored Jul 18, 2024
    Configuration menu
    Copy the full SHA
    da50231 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. start documenting moe setup

    Alexander Hagele committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    7a932f8 View commit details
    Browse the repository at this point in the history

  2. base moe file

    @haeggee
    haeggee committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    f08a05e View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. add todo

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    fa06c0d View commit details
    Browse the repository at this point in the history

  2. gpt3_moe basis

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    a9dba53 View commit details
    Browse the repository at this point in the history

  3. add nn.linear to init for moe router

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    2efffb8 View commit details
    Browse the repository at this point in the history

  4. changes to pipeline for backward through aux losses

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    57c58b7 View commit details
    Browse the repository at this point in the history

  5. correct block costs and flops

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    3967bee View commit details
    Browse the repository at this point in the history

  6. case of dict in pipelineblock

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    bcb94cc View commit details
    Browse the repository at this point in the history

  7. option for GLU or normal MLP

    @haeggee
    haeggee committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    91acdc0 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. init of linear layer in starcoder

    @haeggee
    haeggee committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    df3befc View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. potential bug in pipeline block

    @haeggee
    haeggee committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    6edce83 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. XGLM work in progress: Causal Attention and Positional Embeddings work

    @AleHD @negar-foroutan
    AleHD authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    7425167 View commit details
    Browse the repository at this point in the history

  2. WIP: GPT arch almost done, hf->nt converters working perfectly for no… 

    …n-distributed inference
    @AleHD @negar-foroutan
    AleHD authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    42695ba View commit details
    Browse the repository at this point in the history

  3. Added hf2nt frontend + tested training

    @AleHD @negar-foroutan
    AleHD authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    6294aad View commit details
    Browse the repository at this point in the history

  4. Added nt2hf conversion + tests :)

    @AleHD @negar-foroutan
    AleHD authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    b469ee9 View commit details
    Browse the repository at this point in the history

  5. precommit

    @AleHD @negar-foroutan
    AleHD authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    1b19ca2 View commit details
    Browse the repository at this point in the history

  6. Added MultilingualNanoset Config

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    c1fabac View commit details
    Browse the repository at this point in the history

  7. Added MultilingualNanoset

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    086b50d View commit details
    Browse the repository at this point in the history

  8. Added Language token

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    1fe7445 View commit details
    Browse the repository at this point in the history

  9. Forgot the trainer ups

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    fb6631a View commit details
    Browse the repository at this point in the history

  10. Fix minor errors. Everything works

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    a6eb1bd View commit details
    Browse the repository at this point in the history

  11. Updated config file with GPT2 tokenized datasets in RCP

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    ef3fac4 View commit details
    Browse the repository at this point in the history

  12. Before lunch

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    49294f1 View commit details
    Browse the repository at this point in the history

  13. After lunch

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    8a80e5a View commit details
    Browse the repository at this point in the history

  14. Ready

    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    0fa1971 View commit details
    Browse the repository at this point in the history

  15. Add multilingual validation (#3) 

    Add multilingual validation step.
    @TJ-Solergibert @negar-foroutan
    TJ-Solergibert authored and negar-foroutan committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    8b68126 View commit details
    Browse the repository at this point in the history

  16. correct logging of all losses

    @haeggee
    haeggee committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    d08c949 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. minor bug fix when using bias

    @haeggee
    haeggee committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    d14315f View commit details
    Browse the repository at this point in the history

  2. bias init in case of use for moe

    @haeggee
    haeggee committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    5dc67fe View commit details
    Browse the repository at this point in the history

  3. sparse upcycling converter

    @haeggee
    haeggee committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    fad3497 View commit details
    Browse the repository at this point in the history

  4. merge main into moe; also, adapt gpt3(-moe) for val logging

    @haeggee
    haeggee committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    100ebf4 View commit details
    Browse the repository at this point in the history

  5. add example config

    @haeggee
    haeggee committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    f31a1a3 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. small fixes

    @haeggee
    haeggee committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    a45dc35 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. fix for eval

    @haeggee
    haeggee committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    bb768bb View commit details
    Browse the repository at this point in the history

  2. lighteval fix for multiling

    @haeggee
    haeggee committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    f8e30b4 View commit details
    Browse the repository at this point in the history

  3. Update moe.md

    @haeggee
    haeggee authored Sep 6, 2024
    Configuration menu
    Copy the full SHA
    ac27ada View commit details
    Browse the repository at this point in the history

  4. Update moe.md

    @haeggee
    haeggee authored Sep 6, 2024
    Configuration menu
    Copy the full SHA
    328b8c2 View commit details
    Browse the repository at this point in the history