Introduction

What this project is

This project is a small language model written from scratch in Python and NumPy. Its forward and backward passes are written by hand, with no automatic differentiation. This site documents its mathematics and its code.

The values below come from config.py:

• 2 transformer blocks.
• 2 attention heads, each of width 16.
• Width 32 through the model, and 128 inside the feed-forward step.
• Training windows of 16 tokens.

The vocabulary size is not a config value, because it depends on the corpus. Besides NumPy, data.py uses pandas to read the corpus and to hold the training windows.

The optimizer is the Adam class in optim.py. Training gives it a weight decay of 0.01, applied separately from the gradient, so it behaves as AdamW. See AdamW / Learning Rate / Clipping.

8 test files hold 64 test functions, which make 67 cases because parametrized tests run once per value. Checkpoints & Tests lists what they cover.

Credits

This project was inspired by the open-source how-llms-work repository and by this YouTube video.

It is an independent Python and NumPy implementation. The documentation describes only this repository's code.

How to read this site

The layout

Each page has three columns. The left column lists every section and page. The middle column is the page itself. The right column, "On this page", lists the headings of the current page and follows your position as you scroll.

To search concepts, equations, pages, files and functions, use the Search button at the top, or press Ctrl+K or /.

The sections

• Mathematics explains each operation the model uses, from notation and tokenization to the full model. Each one has its equation, a small worked example, its tensor shapes and its code.
• Training covers how gradients reach the parameters, how the optimizer updates them, and the training loop.
• Generation covers sampling a token and repeating that to produce text.
• Code shows every source file with its real lines, and covers checkpoints and the test suite. Testing, next to it, covers gradient checking.
• Reference collects every equation in one table, and records what the model leaves out and where the sources disagree.

A suggested reading order

1. How This LLM Works follows one training run from start to finish.
2. Architecture shows the model and its parameters.
3. Notation & Shapes defines the symbols that later pages use.
4. Read the rest of Mathematics in sidebar order, from Tokenization to Full Model.
5. Read Training in sidebar order, starting at How the Model Learns.
6. Read Generation, starting at Sampling.
7. Use Code Explorer and Mathematics Reference whenever you want to look something up.

Every page also has previous and next links at the bottom, which follow the sidebar.

From an equation to its code

Each equation has a Show code button. It highlights the lines that compute the equation and scrolls to them. A marked code line works the other way, and shows which equations it belongs to. Line numbers and GitHub links come from a pinned commit, and are generated from the code, not typed by hand.

Labels on this site

• Illustrative Example means the numbers are hand-picked to show how a calculation works, and they are not learned values.
• Actual Model Output means the numbers were produced by running this repository's code.
• Derived from implementation means the code implements this mathematics, and the reference PDF does not state it.

What this site does not cover

Not in This Model lists techniques this code does not use.

The PDF study guide and the code differ in places. Where they do, the site follows the code. Sources & Discrepancies records each difference.

All pages

A page that is not written yet appears as grey text, not a link.