Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug Flash GPT2 #4

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Debug Flash GPT2 #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
05169df
wip
abhi-mosaic Oct 17, 2022
472c576
wip
abhi-mosaic Oct 17, 2022
40c55a4
wip
abhi-mosaic Oct 18, 2022
0326acd
upgrade trainer args
abhi-mosaic Oct 18, 2022
5b693b9
wip
abhi-mosaic Oct 18, 2022
ad1c028
no eos token, concat directly
abhi-mosaic Oct 19, 2022
215ce69
Merge branch 'main' into abhi/debug_flash_gpt2
abhi-mosaic Oct 20, 2022
e57cb6c
upgrade requirements.txt
abhi-mosaic Nov 1, 2022
13d742a
add tests
abhi-mosaic Nov 1, 2022
fd9c049
updates
abhi-mosaic Nov 1, 2022
b44ab07
add 1b final
abhi-mosaic Nov 1, 2022
d408f76
fix eval subset
abhi-mosaic Nov 1, 2022
a54dee5
typo
abhi-mosaic Nov 1, 2022
a3f9fb6
another typo
abhi-mosaic Nov 1, 2022
21dc5b9
fix wandb resumption
abhi-mosaic Nov 2, 2022
0842306
fix act ckpt
abhi-mosaic Nov 2, 2022
a137933
cleanup, add 5b
abhi-mosaic Nov 2, 2022
7df31d0
add 7b option
abhi-mosaic Nov 10, 2022
4507fd7
bugfixes
abhi-mosaic Nov 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 0 additions & 119 deletions llm/convert_c4.py
View file
Open in desktop

This file was deleted.

38 changes: 38 additions & 0 deletions llm/hf_configs/final/gpt-125m-biotok.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 28895,
"embd_pdrop": 0.1,
"eos_token_id": 28895,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": true,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.21.3",
"use_cache": false,
"vocab_size": 28896
}
38 changes: 38 additions & 0 deletions llm/hf_configs/final/gpt-1b-biotok.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 28895,
"embd_pdrop": 0.1,
"eos_token_id": 28895,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 2048,
"n_head": 16,
"n_inner": null,
"n_layer": 24,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": true,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.21.3",
"use_cache": false,
"vocab_size": 28896
}
38 changes: 38 additions & 0 deletions llm/hf_configs/final/gpt-3b-biotok.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 28895,
"embd_pdrop": 0.1,
"eos_token_id": 28895,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 2560,
"n_head": 20,
"n_inner": null,
"n_layer": 32,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": true,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.21.3",
"use_cache": false,
"vocab_size": 28896
}
38 changes: 38 additions & 0 deletions llm/hf_configs/final/gpt-5b-biotok.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 28895,
"embd_pdrop": 0.1,
"eos_token_id": 28895,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 4096,
"n_head": 32,
"n_inner": null,
"n_layer": 24,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": true,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.21.3",
"use_cache": false,
"vocab_size": 28896
}
38 changes: 38 additions & 0 deletions llm/hf_configs/final/gpt-7b-biotok.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 28895,
"embd_pdrop": 0.1,
"eos_token_id": 28895,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 4096,
"n_head": 32,
"n_inner": null,
"n_layer": 32,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": true,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.21.3",
"use_cache": false,
"vocab_size": 28896
}
0 llm/hf_configs/gpt-125m-ctx-1024.json → llm/hf_configs/old/gpt-125m-ctx-1024.json
View file
Open in desktop
File renamed without changes.
0 llm/hf_configs/gpt-1b-ctx-1024.json → llm/hf_configs/old/gpt-1b-ctx-1024.json
View file
Open in desktop
File renamed without changes.
0 llm/hf_configs/gpt-350m-ctx-1024.json → llm/hf_configs/old/gpt-350m-ctx-1024.json
View file
Open in desktop
File renamed without changes.
0 llm/hf_configs/gpt-3b-ctx-1024.json → llm/hf_configs/old/gpt-3b-ctx-1024.json
View file
Open in desktop
File renamed without changes.
0 llm/hf_configs/gpt-760m-ctx-1024.json → llm/hf_configs/old/gpt-760m-ctx-1024.json
View file
Open in desktop
File renamed without changes.
0 llm/hf_configs/gpt-7b-ctx-1024.json → llm/hf_configs/old/gpt-7b-ctx-1024.json
View file
Open in desktop
File renamed without changes.
0 llm/hf_configs/mistral_gpt2_small.json → llm/hf_configs/old/mistral_gpt2_small.json
View file
Open in desktop
File renamed without changes.
38 changes: 38 additions & 0 deletions llm/hf_configs/old/noflash-gpt-125m-ctx-1024.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.1,
"bos_token_id": 50256,
"embd_pdrop": 0.1,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.1,
"scale_attn_by_inverse_layer_idx": false,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.1,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.12.3",
"use_cache": false,
"vocab_size": 50257
}
0 llm/hf_configs/pubmed_gpt2_medium.json → llm/hf_configs/old/pubmed_gpt2_medium.json
View file
Open in desktop
File renamed without changes.
0 ...hf_configs/pubmed_mistral_gpt2_small.json → ...onfigs/old/pubmed_mistral_gpt2_small.json
View file
Open in desktop
File renamed without changes.
38 changes: 38 additions & 0 deletions llm/hf_configs/tests/gpt-125m-ctx-1024-no-dropout.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"activation_function": "gelu_new",
"architectures": [
"GPT2LMHeadModel"
],
"attn_pdrop": 0.0,
"bos_token_id": 50256,
"embd_pdrop": 0.0,
"eos_token_id": 50256,
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"model_type": "gpt2",
"n_ctx": 1024,
"n_embd": 768,
"n_head": 12,
"n_inner": null,
"n_layer": 12,
"n_positions": 1024,
"reorder_and_upcast_attn": false,
"resid_pdrop": 0.0,
"scale_attn_by_inverse_layer_idx": true,
"scale_attn_weights": true,
"summary_activation": null,
"summary_first_dropout": 0.0,
"summary_proj_to_labels": true,
"summary_type": "cls_index",
"summary_use_proj": true,
"task_specific_params": {
"text-generation": {
"do_sample": true,
"max_length": 50
}
},
"torch_dtype": "float32",
"transformers_version": "4.21.3",
"use_cache": false,
"vocab_size": 50257
}
Loading