Tokenization

The tokenizer turns text into the integer ids the model reads. It learns from the text itself, by repeatedly merging the most frequent adjacent pair of tokens.

What this code does

The method is called byte pair encoding, but this code works on characters, not bytes. list(word) splits a pre-token into Unicode characters (tokenizer.py:27). Nothing here touches UTF-8 bytes.

Feature	In this code
Starting tokens	Unicode characters of the text, not byte values
Characters not seen in training	Each maps to the unknown token, id 0
Case	Preserved, so `The` and `the` are distinct

Tokens this code does not have are on Not in This Model.

train.main passes 500 as the limit on merges (train.py:41). That is a maximum, because training stops early if no pair is left (tokenizer.py:40). The default in the function signature, max_merges=1000, is not used by training.

Source

The reference PDF differs from the code here, and this page follows the code. Its pattern lets a word carry its leading space, and its treatment of case differs too. In the code, a space is always its own pre-token and case is preserved. The full list is on Sources & Discrepancies.

The tokenizer has no backward pass. Training the merges only counts, and the model's gradients stop at the embedding table.

Actual Model Output

Everything else on this page is an Illustrative Example. This panel reads the tokenizer stored in a checkpoint.

The first merges in the checkpointActual Model OutputCommit 66da45b. Checkpoint model.npz sha256 74de2ec88a63. Seed 42, from config.py, since a checkpoint does not record its own. Read from checkpoints/model, an ignored local file, trained on data/corpus.csv (git-ignored, and not at the pinned commit), with training settings that by inference differ from config.py.500 merges and a vocabulary of 575 tokens, read from the checkpoint's tokenizer.

Merge	Pair	New token	Frequency
0	`h + e`	`he`	13,019
1	`t + he`	`the`	7,414
2	`i + n`	`in`	5,744
3	`a + n`	`an`	5,487
4	`o + u`	`ou`	4,313
5	`e + r`	`er`	4,126
6	`i + t`	`it`	3,612
7	`r + e`	`re`	3,462
8	`an + d`	`and`	3,426
9	`a + t`	`at`	3,311

Sentences from the corpus, encoded by that tokenizerActual Model OutputCommit 66da45b. Checkpoint model.npz sha256 74de2ec88a63. Seed 42, from config.py, since a checkpoint does not record its own. Read from checkpoints/model, an ignored local file, trained on data/corpus.csv (git-ignored, and not at the pinned commit), with training settings that by inference differ from config.py.

Sentence	Tokens	Ids
"It's the Rain, and I want to come in," said a soft, sad, little voice.	`"` `It` `'` `s` `␠` `the` `␠` `R` `ain` `,` `␠` `and` `␠` `I` `␠` `want` `␠` `to` `␠` `come` `␠` `in` `,` `"` `␠` `said` `␠` `a` `␠` `s` `of` `t` `,` `␠` `s` `ad` `,` `␠` `little` `␠` `voice` `.`	`41 254 5 6 7 76 7 42 239 15 7 83 7 13 7 407 7 85 7 305 7 77 15 41 7 123 7 8 7 6 97 12 15 7 6 103 15 7 136 7 386 32`
"No, you can't come in," the little Tulip said.	`"` `No` `,` `␠` `you` `␠` `can` `'` `t` `␠` `come` `␠` `in` `,` `"` `␠` `the` `␠` `little` `␠` `T` `u` `li` `p` `␠` `said` `.`	`41 538 15 7 114 7 292 5 12 7 305 7 77 15 41 7 76 7 136 7 1 17 110 26 7 123 32`
By and by she heard another little _tap, tap, tap_ on the window-pane.	`B` `y` `␠` `and` `␠` `by` `␠` `she` `␠` `heard` `␠` `another` `␠` `little` `␠` `_` `ta` `p` `,` `␠` `ta` `p` `,` `␠` `ta` `p` `_` `␠` `on` `␠` `the` `␠` `w` `ind` `ow` `-` `p` `an` `e` `.`	`44 28 7 83 7 280 7 118 7 402 7 539 7 136 7 40 203 26 15 7 203 26 15 7 203 26 40 7 90 7 76 7 29 190 99 45 26 78 3 32`

Where these numbers came from

Repository commit	`66da45bea72176fa5d2f8dba2ea1c7e597e30dd3`
`model.npz` sha256	`74de2ec88a63ceb131a27e03cc63070c6d8c4174cc78bc189195e9356e84550e`
`model.json` sha256	`f93002949162068d573aab9b38990b12aec7f32b468ec9eaaef986075df7d3b4`
`model.tok.json` sha256	`bf874ab4e4de2b61cb936f38635a7c589c5c0d29879947a5b45d13be2456b17d`
Corpus `data/corpus.csv` sha256	`c1bec80f274294e6ce989fcc63fd7022a206f415e7d3df85e82c410e56dc4f4f`
Tokenizer rebuilt from this corpus equals the checkpoint's	Yes
Model shape, from the checkpoint	`{"context_len":32,"emb_dim":32,"ff_dim":128,"ln_eps":0.00001,"num_heads":2,"num_layers":2,"vocab_size":575}`
`config.py`, when this was exported	`{"adam_eps":1e-8,"batch_size":16,"beta1":0.9,"beta2":0.999,"context_len":32,"corpus_column":"text","corpus_path":"data/corpus.csv","emb_dim":32,"epochs":20,"ff_dim":128,"grad_clip":1,"head_dim":16,"ln_eps":0.00001,"lr_max":0.001,"lr_min":0.0001,"max_new_tokens":50,"max_rows":2000,"num_heads":2,"num_layers":2,"num_merges":500,"seed":42,"seq_len":16,"stride":1,"temperature":0.8,"top_k":0,"top_p":0.9,"val_fraction":0.1,"warmup_steps":50,"weight_decay":0.01}`
Seed	`42`
Environment	`Python 3.14.4, NumPy 2.5.2, pandas 3.1.0`

Pre-tokens and word counts

What is it?

Pre-tokens and word countsImplemented

Tokenizer

A regular expression cuts the text into pre-tokens. Each one is a run of word characters, a single whitespace character, or a single other character. count_words returns how often each one occurs.

The pattern has three alternatives, tried in order:

\w+: one or more word characters. On Python strings these are Unicode letters and digits, plus the underscore.
\s: exactly one whitespace character. Two spaces make two pre-tokens.
[^\w\s]: exactly one character that is neither of those, such as punctuation.

Every character matches one of the three, so no character is dropped.

Why does the model need it?

Pre-tokens limit where merges can happen. Each pre-token has its own list of tokens, so no merge can join the end of one pre-token to the start of the next. The counts avoid repeating work, since each distinct pre-token is handled once and weighted by its count.

Whitespace and punctuation pre-tokens are always one character long, so they have no inner pair. As a result, every merge in this tokenizer happens inside a run of word characters, and a merged token never contains a space or punctuation.

Mathematics

How often the pre-token w occurs #

n_w = \left|\{\, i : p_i = w \,\}\right|

What does each symbol mean?

Symbol	Meaning	In code	Shape
$p_i$	The i-th match of PRE_TOKEN_RE in the text	`word`	string
$w$	A distinct pre-token	`word`	string
$n_w$	Its count	`counts[word]`	integer

Tiny numerical example

Pre-tokens and their countsIllustrative ExampleTiny texts chosen by hand, not the corpus. ␠ is a space and \n is a newline.

Text	Pre-tokens with their counts
`"low lower lowest"`	`low␠×␠1` `␠␠×␠2` `lower␠×␠1` `lowest␠×␠1`
`"Don't stop_1 now,\nok\t?"`	`Don␠×␠1` `'␠×␠1` `t␠×␠1` `␠␠×␠3` `stop_1␠×␠1` `now␠×␠1` `,␠×␠1` `\n␠×␠1` `ok␠×␠1` `\t␠×␠1` `?␠×␠1`

Tensor shapes

None. This step works on a string and a dict, before any array exists.

Actual implementation

tokenizer.pyPRE_TOKEN_REGitHub

4PRE_TOKEN_RE = re.compile(r"\w+|\s|[^\w\s]")

tokenizer.pycount_wordsGitHub

6def count_words(text):7    counts = {}8    for word in PRE_TOKEN_RE.findall(text):9        counts[word] = counts.get(word, 0) + 110    return counts

Code and math connection

Equation	Role	Lines	Code
How often the pre-token w occurs	Computes it	tokenizer.py:8	`for word in PRE_TOKEN_RE.findall(text):`
How often the pre-token w occurs	Computes it	tokenizer.py:9	`counts[word] = counts.get(word, 0) + 1`
How often the pre-token w occurs	Tests it	tests/test_tokenizer.py:11	`assert count_words("a a b!") == {"a": 2, " ": 2, "b": 1, "!": 1}`

Where is it used?

Source

tokenizer.py, lines 6-10, count_words

PDF reference

Section 4.1, Eq. 3, p. 4

Used by

Called by: tokenizer.train_bpe_on_text

Tested by: tests.test_tokenizer.test_count_words_splits_words_spaces_punctuation

Tests it: tests/test_tokenizer.py, line 11 (test_count_words_splits_words_spaces_punctuation), for How often the pre-token w occurs

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
How often the pre-token w occurs	Computes it	`tokenizer.py`	`count_words`	8	GitHub: tokenizer.py, line 8
How often the pre-token w occurs	Computes it	`tokenizer.py`	`count_words`	9	GitHub: tokenizer.py, line 9

Pair frequency and the best pair

What is it?

Pair frequency and the best pairImplemented

Tokenizer

At each step, train_bpe counts every adjacent pair of tokens inside every pre-token, weighted by how often that pre-token occurs. It picks the pair with the largest count. A tie goes to the pair it found first.

Each entry of the merge list records the pair, the merged string and the frequency (tokenizer.py:59).

Why does the model need it?

Merging the most frequent pair lets one token stand for characters that often occur together, so common words take fewer tokens. The list of merges is what the tokenizer learns from the text.

Mathematics

Pair frequency, weighted by word count #

f(a,b) = \sum_{w} n_w \,\bigl|\{\, i : s^{w}_{i} = a,\; s^{w}_{i+1} = b \,\}\bigr|

The best pair #

(a^*, b^*) = \operatorname*{arg\,max}_{(a,b)} f(a,b)

The argmax is not unique when pairs tie. The code loops over pair_freq in the order pairs were first found and only replaces the best pair on a strictly larger count (tokenizer.py:46), so the first maximum found wins. That order follows the order of first appearance of the pre-tokens in the text.

What does each symbol mean?

Symbol	Meaning	In code	Shape
$f(a,b)$	Frequency of the adjacent pair	`pair_freq[(a, b)]`	integer
$n_w$	Count of the pre-token w	`word_freqs[word]`	integer
$s^{w}$	The current split of w, a list of tokens	`word_splits[word]`	list of strings
$(a^, b^)$	The chosen pair, the first maximum found	`best_pair`	pair of strings

Tiny numerical example

Step through the training of a tiny text. Each step shows the word table, the pair counts, the chosen pair and the updated splits. In the first text, l + o and o + w tie on the first merge, so you can see the rule above decide.

BPE, one merge at a timeIllustrative ExampleA tiny text chosen by hand, not the corpus. These are not the merges of the real model, which trains on the corpus with up to 500 merges. A test compares these steps with the real train_bpe on the same text.

Text

Step 1 of 8: Start

Pre-tokenize the text, count each pre-token, and split each one into characters. Each pre-token keeps its own list, so a pair can only come from inside one pre-token.

Pre-token	Count	Current split
`low`	1	`l` `o` `w`
`␠`	2	`␠`
`lower`	1	`l` `o` `w` `e` `r`
`lowest`	1	`l` `o` `w` `e` `s` `t`

Tensor shapes

None. Training the merges uses dicts and lists of strings.

Actual implementation

tokenizer.pytrain_bpeGitHub

24def train_bpe(word_freqs, max_merges=1000):25    word_splits = {}26    for word in word_freqs:27        word_splits[word] = list(word)2829    merges = []3031    for step in range(max_merges):32        pair_freq = {}33        for word in word_splits:34            tokens = word_splits[word]35            weight = word_freqs[word]36            for i in range(len(tokens) - 1):37                key = (tokens[i], tokens[i + 1])38                pair_freq[key] = pair_freq.get(key, 0) + weight3940        if len(pair_freq) == 0:41            break4243        best_pair = None44        best_count = 045        for pair in pair_freq:46            if pair_freq[pair] > best_count:47                best_count = pair_freq[pair]48                best_pair = pair4950        if best_pair is None:51            break5253        a, b = best_pair54        merged = a + b5556        for word in word_splits:57            word_splits[word] = merge_tokens(word_splits[word], best_pair, merged)5859        merges.append({"pair": best_pair, "merged": merged, "frequency": best_count})6061    return merges

train_bpe_on_text joins the two steps so far.

tokenizer.pytrain_bpe_on_textGitHub

76def train_bpe_on_text(text, max_merges=1000):77    return train_bpe(count_words(text), max_merges)

Code and math connection

Equation	Role	Lines	Code
Pair frequency, weighted by word count	Computes it	tokenizer.py:35	`weight = word_freqs[word]`
Pair frequency, weighted by word count	Computes it	tokenizer.py:37	`key = (tokens[i], tokens[i + 1])`
Pair frequency, weighted by word count	Computes it	tokenizer.py:38	`pair_freq[key] = pair_freq.get(key, 0) + weight`
The best pair	Computes it	tokenizer.py:46	`if pair_freq[pair] > best_count:`
The best pair	Computes it	tokenizer.py:48	`best_pair = pair`
The best pair	Tests it	tests/test_tokenizer.py:24	`merges = train_bpe_on_text(CORPUS, max_merges=3)`

Where is it used?

Source

tokenizer.py, lines 24-61, train_bpe

tokenizer.py, lines 76-77, train_bpe_on_text

PDF reference

Section 4.1, Eq. 3, p. 4

Used by

Called by: tokenizer.train_bpe_on_text, tokenizer.build_tokenizer

Tested by: tests.test_tokenizer.test_first_three_merges_match_hand_trace

Tests it: tests/test_tokenizer.py, line 24 (test_first_three_merges_match_hand_trace), for The best pair

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
Pair frequency, weighted by word count	Computes it	`tokenizer.py`	`train_bpe`	35	GitHub: tokenizer.py, line 35
Pair frequency, weighted by word count	Computes it	`tokenizer.py`	`train_bpe`	37	GitHub: tokenizer.py, line 37
Pair frequency, weighted by word count	Computes it	`tokenizer.py`	`train_bpe`	38	GitHub: tokenizer.py, line 38
The best pair	Computes it	`tokenizer.py`	`train_bpe`	46	GitHub: tokenizer.py, line 46
The best pair	Computes it	`tokenizer.py`	`train_bpe`	48	GitHub: tokenizer.py, line 48

Applying a merge

What is it?

Applying a mergeImplemented

Tokenizer

merge_tokens scans a list of tokens from left to right. Wherever the two tokens of the pair sit side by side, it replaces them with one token and moves past both.

Why does the model need it?

Training uses it to update each split after choosing a pair, and encoding uses it to replay the learned merges. Both go through this one function, so they treat a pair the same way.

Mathematics

Replace an adjacent pair with one token #

[\,\dots,\, a^*,\, b^*,\, \dots\,] \;\to\; [\,\dots,\, a^* b^*,\, \dots\,]

What does each symbol mean?

Symbol	Meaning	In code	Shape
$a^, b^$	The two tokens of the pair	`pair[0], pair[1]`	strings
$a^* b^*$	The two strings joined	`merged`	string

Tiny numerical example

The second row shows the left-to-right scan: in three a tokens, the first two merge and the third stays alone.

merge_tokensIllustrative ExampleHand-picked tokens.

Tokens	Pair	Result
`a` `b` `c` `a` `b`	`a` `b`	`ab` `c` `ab`
`a` `a` `a`	`a` `a`	`aa` `a`

Tensor shapes

None. A split is a list of strings.

Actual implementation

tokenizer.pymerge_tokensGitHub

12def merge_tokens(tokens, pair, merged):13    result = []14    i = 015    while i < len(tokens):16        if i < len(tokens) - 1 and tokens[i] == pair[0] and tokens[i + 1] == pair[1]:17            result.append(merged)18            i += 219        else:20            result.append(tokens[i])21            i += 122    return result

Code and math connection

Equation	Role	Lines	Code
Replace an adjacent pair with one token	Computes it	tokenizer.py:16	`if i < len(tokens) - 1 and tokens[i] == pair[0] and tokens[i + 1] == pair[1]:`
Replace an adjacent pair with one token	Computes it	tokenizer.py:17	`result.append(merged)`
Replace an adjacent pair with one token	Calls it	tokenizer.py:57	`word_splits[word] = merge_tokens(word_splits[word], best_pair, merged)`
Replace an adjacent pair with one token	Calls it	tokenizer.py:70	`tokens = merge_tokens(tokens, m["pair"], m["merged"])`
Replace an adjacent pair with one token	Tests it	tests/test_tokenizer.py:20	`assert merge_tokens(["a", "a", "a"], ("a", "a"), "aa") == ["aa", "a"]`

Where is it used?

Source

tokenizer.py, lines 12-22, merge_tokens

PDF reference

Section 4.1, p. 4

Used by

Called by: tokenizer.apply_merges, tokenizer.train_bpe

Tested by: tests.test_tokenizer.test_merge_tokens_overlap_is_left_to_right, tests.test_tokenizer.test_merge_tokens_simple

Calls it: tokenizer.py, line 57 (train_bpe), for Replace an adjacent pair with one token
Calls it: tokenizer.py, line 70 (apply_merges), for Replace an adjacent pair with one token
Tests it: tests/test_tokenizer.py, line 20 (test_merge_tokens_overlap_is_left_to_right), for Replace an adjacent pair with one token

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
Replace an adjacent pair with one token	Computes it	`tokenizer.py`	`merge_tokens`	16	GitHub: tokenizer.py, line 16
Replace an adjacent pair with one token	Computes it	`tokenizer.py`	`merge_tokens`	17	GitHub: tokenizer.py, line 17

The vocabulary

What is it?

The vocabularyImplemented

Tokenizer

build_tokenizer trains the merges and then lists the vocabulary. The unknown token comes first, so it has id 0. Next come the characters of the text in order of first appearance, then each merged token in merge order. A merged string already in the list is not added again.

Because of that last check (tokenizer.py:111), V is at most one, plus the number of distinct characters, plus the number of merges.

Why does the model need it?

An id is a row of the token table and a column of the head, so every token needs a fixed id. Keeping all the characters means any text made of seen characters can be encoded, even when no merge applies.

Mathematics

The order of the vocabulary #

\text{itos} = [\,\texttt{<unk>}\,] + [\,\text{characters, first seen first}\,] + [\,\text{new merged tokens, in merge order}\,]

What does each symbol mean?

Symbol	Meaning	In code	Shape
$\text{itos}$	From an id to its token	`itos`	list of V strings
$\text{stoi}$	From a token to its id	`stoi`	dict of V entries
$V$	The vocabulary size	`vocab_size`	integer

Tiny numerical example

The vocabulary of "low lower lowest"Illustrative ExampleA tiny text chosen by hand, not the corpus. These are not the real model's.

Id	Token	Comes from
0	`<unk>`	The unknown token, always first
1	`l`	Character, by first appearance
2	`o`	Character, by first appearance
3	`w`	Character, by first appearance
4	`␠`	Character, by first appearance
5	`e`	Character, by first appearance
6	`r`	Character, by first appearance
7	`s`	Character, by first appearance
8	`t`	Character, by first appearance
9	`lo`	Merged token, by merge order
10	`low`	Merged token, by merge order
11	`lowe`	Merged token, by merge order
12	`lower`	Merged token, by merge order
13	`lowes`	Merged token, by merge order
14	`lowest`	Merged token, by merge order

Tensor shapes

None here. V from this step sets the size of tok_emb and head_w, as listed on Notation & Shapes.

Actual implementation

tokenizer.pybuild_tokenizerGitHub

99def build_tokenizer(text, max_merges=1000):100    merges = train_bpe_on_text(text, max_merges)101102    itos = [UNK]103    seen = {UNK}104105    for ch in text:106        if ch not in seen:107            seen.add(ch)108            itos.append(ch)109110    for m in merges:111        if m["merged"] not in seen:112            seen.add(m["merged"])113            itos.append(m["merged"])114    115    stoi = {}116    for i in range(len(itos)):117        stoi[itos[i]] = i118119    return Tokenizer(merges, stoi, itos)

Code and math connection

Equation	Role	Lines	Code
The order of the vocabulary	Computes it	tokenizer.py:102	`itos = [UNK]`
The order of the vocabulary	Computes it	tokenizer.py:105	`for ch in text:`
The order of the vocabulary	Computes it	tokenizer.py:110	`for m in merges:`
The order of the vocabulary	Tests it	tests/test_tokenizer.py:35	`assert tok.itos[0] == UNK`

Where is it used?

Source

tokenizer.py, lines 99-119, build_tokenizer

PDF reference

Section 4.1, p. 4

Used by

Called by: train.main

Tested by: tests.test_checkpoint.saved, tests.test_checkpoint.test_non_ascii_characters_survive, tests.test_tokenizer.tok

Tests it: tests/test_tokenizer.py, line 35 (test_unk_is_id_zero), for The order of the vocabulary

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
The order of the vocabulary	Computes it	`tokenizer.py`	`build_tokenizer`	102	GitHub: tokenizer.py, line 102
The order of the vocabulary	Computes it	`tokenizer.py`	`build_tokenizer`	105	GitHub: tokenizer.py, line 105
The order of the vocabulary	Computes it	`tokenizer.py`	`build_tokenizer`	110	GitHub: tokenizer.py, line 110

Encode and decode

What is it?

Encode and decodeImplemented

Tokenizer

encode cuts the text into pre-tokens, replays every merge in order on each one, and looks up each resulting token. A token that is not in the vocabulary gets the id of the unknown token. decode looks up the string of each id and joins them.

The unknown token works like this (tokenizer.py:89):

Only a character that was not in the training text can be missing from the vocabulary. Merged tokens and seen characters are all listed.
Each unseen character becomes its own id 0. zzz gives three ids, and a word keeps its known part: lowz gives the id of low, then 0.
decode turns id 0 into the literal string <unk>, so the original character is lost. decode(encode(x)) returns x only when every character of x was seen in training.
The text <unk> is not special. It is five characters, and each is looked up on its own.
train.main builds the tokenizer from the same text it encodes, so every character is known and id 0 does not occur in the training ids. It can occur in a prompt given to talk.py.

Why does the model need it?

The model works on ids. Encoding turns the corpus and each prompt into ids, and decoding turns the ids that generation produces back into text.

Mathematics

The id of a token #

\text{id}(\tau) = \begin{cases} \text{stoi}[\tau] & \text{if } \tau \in \text{stoi} \\ \text{stoi}[\,\texttt{<unk>}\,] & \text{otherwise} \end{cases}

Decoding joins the strings #

\text{decode}(i_1, \dots, i_n) = \text{itos}[i_1] \,\Vert\, \cdots \,\Vert\, \text{itos}[i_n]

What does each symbol mean?

Symbol	Meaning	In code	Shape
$\tau$	A token after all merges	`token`	string
$i_k$	An id	`i`	integer
$\Vert$	String concatenation	`"".join(tokens)`	operator

Tiny numerical example

Encoding and decoding with the tokenizer of "low lower lowest"Illustrative ExampleA tiny text chosen by hand, not the corpus. These are not the real model's.

Text	Tokens	Ids	Decoded
`lowest low`	`lowest` `␠` `low`	`[14,4,10]`	`lowest low`
`low!`	`low` `!`	`[10,0]`	`low<unk>`
`lowz`	`low` `z`	`[10,0]`	`low<unk>`
`zzz`	`z` `z` `z`	`[0,0,0]`	`<unk><unk><unk>`
`<unk>`	`<` `u` `n` `k` `>`	`[0,0,0,0,0]`	`<unk><unk><unk><unk><unk>`

Tensor shapes

encode returns a Python list. encode_corpus turns it into a one-dimensional array of integers, which is where the Data Pipeline starts.

The ids of the corpus, as a list and then as an arraydata.py:16

tok.encode(text)(N,)
ids(N,)

N is the number of ids. It depends on the corpus.

Actual implementation

tokenizer.pyUNKGitHub

3UNK = "<unk>"

tokenizer.pyapply_mergesGitHub

63def apply_merges(text, merges):64    result = []6566    for pre_token in PRE_TOKEN_RE.findall(text):67        tokens = list(pre_token)6869        for m in merges:70            tokens = merge_tokens(tokens, m["pair"], m["merged"])71        result.extend(tokens)7273    return result

tokenizer.pyencodeGitHub

86    def encode(self, text):87        ids = []88        for token in apply_merges(text, self.merges):89            ids.append(self.stoi.get(token, self.stoi[UNK]))90        return ids

tokenizer.pydecodeGitHub

92    def decode(self, ids):93        tokens = []94        for i in ids:95            tokens.append(self.itos[i])96        return "".join(tokens)

Code and math connection

Equation	Role	Lines	Code
The id of a token	Computes it	tokenizer.py:89	`ids.append(self.stoi.get(token, self.stoi[UNK]))`
The id of a token	Calls it	tokenizer.py:70	`tokens = merge_tokens(tokens, m["pair"], m["merged"])`
The id of a token	Tests it	tests/test_tokenizer.py:45	`assert tok.encode("low!")[-1] == 0`
Decoding joins the strings	Computes it	tokenizer.py:95	`tokens.append(self.itos[i])`
Decoding joins the strings	Computes it	tokenizer.py:96	`return "".join(tokens)`
Decoding joins the strings	Tests it	tests/test_tokenizer.py:50	`assert tok.decode(tok.encode(s)) == s`

Where is it used?

Source

tokenizer.py, lines 86-90, encode

tokenizer.py, lines 92-96, decode

tokenizer.py, lines 63-73, apply_merges

PDF reference

Section 4.1, p. 4

Used by

Called by: data.encode_corpus, talk.main, talk.main, train.main, tokenizer.Tokenizer.encode

Tested by: tests.test_checkpoint.test_loaded_model_gives_identical_logits, tests.test_checkpoint.test_non_ascii_characters_survive, tests.test_checkpoint.test_tokenizer_round_trips, tests.test_tokenizer.test_no_characters_dropped, tests.test_tokenizer.test_round_trip, tests.test_tokenizer.test_unseen_character_becomes_unk, tests.test_checkpoint.test_non_ascii_characters_survive, tests.test_checkpoint.test_tokenizer_round_trips, tests.test_tokenizer.test_round_trip

Calls it: tokenizer.py, line 70 (apply_merges), for The id of a token
Tests it: tests/test_tokenizer.py, line 45 (test_unseen_character_becomes_unk), for The id of a token
Tests it: tests/test_tokenizer.py, line 50 (test_round_trip), for Decoding joins the strings

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
The id of a token	Computes it	`tokenizer.py`	`encode`	89	GitHub: tokenizer.py, line 89
Decoding joins the strings	Computes it	`tokenizer.py`	`decode`	95	GitHub: tokenizer.py, line 95
Decoding joins the strings	Computes it	`tokenizer.py`	`decode`	96	GitHub: tokenizer.py, line 96

Every symbol in tokenizer.py

This table lists every function, class and constant in the file. It is checked against the code at build time.

Symbol	Lines	On this page
`tokenizer.UNK`	tokenizer.py:3	Encode and decode
`tokenizer.PRE_TOKEN_RE`	tokenizer.py:4	Pre-tokens and word counts
`tokenizer.count_words`	tokenizer.py:6-10	Pre-tokens and word counts
`tokenizer.merge_tokens`	tokenizer.py:12-22	Applying a merge
`tokenizer.train_bpe`	tokenizer.py:24-61	Pair frequency and the best pair
`tokenizer.apply_merges`	tokenizer.py:63-73	Encode and decode
`tokenizer.train_bpe_on_text`	tokenizer.py:76-77	Pair frequency and the best pair
`tokenizer.Tokenizer`	tokenizer.py:79-96	Skipped. Holds no logic of its own. Its encode and decode methods are shown separately.
`tokenizer.Tokenizer.__init__`	tokenizer.py:80-84	Skipped. Stores merges, stoi and itos, and sets vocab_size to len(itos). There is nothing to compute.
`tokenizer.Tokenizer.encode`	tokenizer.py:86-90	Encode and decode
`tokenizer.Tokenizer.decode`	tokenizer.py:92-96	Encode and decode
`tokenizer.build_tokenizer`	tokenizer.py:99-119	The vocabulary