What it actually does
nanoGPT is a compact PyTorch codebase for training, finetuning and sampling GPT-2-style language models. It has a model definition, a training loop with DDP, gradient accumulation, mixed precision and cosine LR decay, scripts that tokenize datasets into uint16 .bin files, and a sampling script. There is no agent loop or tool use; it is a plain ML training library. The README notes the repo is deprecated in favour of nanochat.
- GPT-2 architecture (causal self-attention with flash attention or a manual fallback, MLP, pre-LN blocks, weight tying) with an autoregressive generate() using temperature and top-kmodel.py:29-76, 94-106, 118-193, 305-330
- Load OpenAI GPT-2 weights (gpt2 through gpt2-xl) from HuggingFace transformers, transposing Conv1D weightsmodel.py:206-261
- Training loop with gradient accumulation, GradScaler for fp16, grad clipping, cosine LR with warmup, periodic eval and checkpointing, optional wandbtrain.py:231-242, 255-333
- Multi-GPU and multi-node training via PyTorch DDP, launched through torchruntrain.py:82-101, 210-212
- Resume from a checkpoint or initialize from scratch or from a GPT-2 model for finetuning, with block-size croppingtrain.py:149-193; model.py:195-204
- Dataset preparation: OpenWebText GPT-2 BPE tokenization to a memmapped uint16 file, Shakespeare in BPE and char-level variantsdata/openwebtext/prepare.py:21-81; data/shakespeare/prepare.py; data/shakespeare_char/prepare.py
- Sampling script that supports resume checkpoints or pretrained GPT-2, file-based prompts, and meta.pkl char encodingssample.py:35-89
- Benchmarking and profiling with MFU estimation relative to A100 bf16 peakbench.py:60-117; model.py:289-303
- Config system that exec()s a config file and applies --key=value overrides to globalsconfigurator.py:20-47
README claims vs the code
| Simple, readable code: train.py is ~300-line training loop and model.py ~300-line GPT definition | Backed by code | train.py is 337 lines and model.py 331 lines, and both are plain, flat PyTorch. |
| Can optionally load the GPT-2 weights from OpenAI | Backed by code | model.py:206-261 GPT.from_pretrained via HuggingFace GPT2LMHeadModel. |
| train.py reproduces GPT-2 (124M) on OpenWebText on an 8xA100 40GB node in about 4 days (reaching ~2.85 loss) | Can't verify | config/train_gpt2.py and the DDP code exist, but the runtime and loss result depend on hardware and runs; the config comment says ~5 days while the README says ~4. |
| Supports finetuning pretrained GPT-2 checkpoints (e.g. on Shakespeare) | Backed by code | train.py:181-188 init_from gpt2*; config/finetune_shakespeare.py; data/shakespeare/prepare.py. |
| Supports multi-GPU and multi-node training with DDP | Backed by code | train.py:82-101, 210-212, with torchrun usage in the docstring. |
| torch.compile gives a noticeable speedup (250ms to 135ms per iter) | Can't verify | The code calls torch.compile (train.py:205-208), but the timings depend on hardware. |
| Baseline GPT-2 val losses on OpenWebText (e.g. 3.12 for 124M) | Can't verify | The eval configs exist (config/eval_gpt2*.py, eval_only in train.py:287), but the numbers need to be run. |
| Fastest repository for training/finetuning medium-sized GPTs | Can't verify | It uses flash attention, fused AdamW, compile and bf16 (model.py:62-64, 281-284), but "fastest" is a comparative claim that can't be shown from the code. |
Code findings
None of the code checks fired.
Who made it and how it got popular
Nothing unusual in the account, history or stars.
Scanned 26 files (24 text, 2 binary) at commit 3adf61e. "No red flags" means none of our checks fired, not that the code is guaranteed safe.
Vetted launch
Are you @karpathy? Log in to pump.fun with GitHub to claim your fees.
Builder's response
The maintainers haven't responded.
Check again
If the repo has new commits, or your level allows a deeper review than this one, a new check runs. Otherwise you'll see this report. Levels