llm.go

GPT-2 implementation written in go only using the standard library.

🪞 Quick start

Install python dependencies, output tokenized dataset

make setup

Run the training script:

make train

This will run go run ./cmd/traingpt2/main.go

Run the testing script:

make test

This will run go run ./cmd/testgpt2/main.go

Tokenize input text, the implementation of this is incorrect. Need to do pair matching not tries
Very slow, need to improve performance.
It runs in WASM but using WebGPU bindings might be fun.
More refactoring.
Running as CLI.

See LICENSE for more details.

Name		Name	Last commit message	Last commit date
Latest commit History 18 Commits
cmd		cmd
testdata/fuzz		testdata/fuzz
wasm		wasm
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
command.go		command.go
dataloader.go		dataloader.go
dataloader_test.go		dataloader_test.go
go.mod		go.mod
go.sum		go.sum
gpt.go		gpt.go
gpt_test.go		gpt_test.go
hf_loader.go		hf_loader.go
main_test.go		main_test.go
math.go		math.go
math32.go		math32.go
math_test.go		math_test.go
open.go		open.go
open_wasm.go		open_wasm.go
prepro_tinyshakespeare.py		prepro_tinyshakespeare.py
prepro_tinystories.py		prepro_tinystories.py
requirements.txt		requirements.txt
sampler.go		sampler.go
tensor.go		tensor.go
tokenizer.go		tokenizer.go
tokenizer_test.go		tokenizer_test.go
train_gpt2.py		train_gpt2.py
utils.go		utils.go