Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Implements Roberta Model #679

Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft

[WIP] Implements Roberta Model #679

wants to merge 32 commits into from

Commits on Jul 30, 2024

  1. [WIP] Implements Roberta Model

    JulienDarve committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    d6284ca View commit details
    Browse the repository at this point in the history

  2. Implements dynamic masking objective

    @prady-saligram
    prady-saligram committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    8f7402e View commit details
    Browse the repository at this point in the history

  3. Implements dynamic masked dataset

    @prady-saligram
    prady-saligram committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    670b053 View commit details
    Browse the repository at this point in the history

  4. Reintroduced accidentally deleted CausalLMDataset class

    @prady-saligram
    prady-saligram committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    42f5404 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. Everything works except stuck on the final method, 

    RobertaForMaskedLM
    JulienDarve committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    9ad06af View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. [WIP] Re-implements MLM training objective

    @prady-saligram
    prady-saligram committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    53fd8d2 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Adds error handling and reverts LmExample class to original

    @prady-saligram
    prady-saligram committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    dcd45b2 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Testing Modifications

    JulienDarve committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    6f21e0d View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2024

  1. Merge branch 'stanford-crfm:main' into roberta-model

    @prady-saligram
    prady-saligram authored Aug 26, 2024
    Configuration menu
    Copy the full SHA
    730d847 View commit details
    Browse the repository at this point in the history

  2. Sets RobertaConfig as model architecture and creates default config file

    @prady-saligram
    prady-saligram committed Aug 26, 2024
    Configuration menu
    Copy the full SHA
    027b176 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. Adds compute_loss to roberta and changes positional ids to begin from 0

    @prady-saligram
    prady-saligram committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    399e08c View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. Investingating precision loss over time within the model using output… 

    …_hidden_states implementation in jax model
    JulienDarve committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    cd4118c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'roberta-model' of https://github.com/JulienDarve/levanter… 

    … into roberta-model
    JulienDarve committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    96522f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Model can now successfully import weights from huggingface + made att… 

    …ention mask more robust
    JulienDarve committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8a732e5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'roberta-training' into roberta-model-copy-2

    JulienDarve committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    5f3d8a2 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. trial

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    6c105f5 View commit details
    Browse the repository at this point in the history

  2. update 1

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ab85079 View commit details
    Browse the repository at this point in the history

  3. update 2

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    5b97400 View commit details
    Browse the repository at this point in the history

  4. update 3

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    bd7d411 View commit details
    Browse the repository at this point in the history

  5. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b5d8e14 View commit details
    Browse the repository at this point in the history

  6. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8717c3f View commit details
    Browse the repository at this point in the history

  7. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    10c130c View commit details
    Browse the repository at this point in the history

  8. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    834d88d View commit details
    Browse the repository at this point in the history

  9. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    47fe23b View commit details
    Browse the repository at this point in the history

  10. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    fb5c55c View commit details
    Browse the repository at this point in the history

  11. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8594e79 View commit details
    Browse the repository at this point in the history

  12. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    3ae80d7 View commit details
    Browse the repository at this point in the history

  13. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    de93fc9 View commit details
    Browse the repository at this point in the history

  14. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    896af7d View commit details
    Browse the repository at this point in the history

  15. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0be9a83 View commit details
    Browse the repository at this point in the history

  16. update

    JulienDarve committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0c94a47 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. Training works!

    @JulienDarve
    JulienDarve committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    7ae681d View commit details
    Browse the repository at this point in the history