Autoregressive Generation

generate turns the one-token step of Sampling into a longer text, by feeding each new token back in as input.

This page has no backward pass. generate calls forward only. It computes no loss and no gradient, and it changes no parameter. Every number here is an Illustrative Example, so no text generated by a trained model is shown.

The loop

What is it?

Autoregressive loopImplemented

Sampling

Starts from a list of ids and repeats one step max_new_tokens times. It runs the model on the last window ids, takes the logits of the last position only, samples one id, and appends it. It returns the whole list, seed included.

Why does the model need it?

The model predicts one next token per position. Text longer than one token comes from feeding each new token back in. The window keeps the input within the rows of pos_emb, because forward asserts that the length is at most context_len.

Mathematics

The last window ids #

x = \mathrm{ids}[-w:]

Logits of the last position only #

Z = \mathrm{forward}(x), \qquad z = Z_{T-1}

Sample and append #

\mathrm{ids} \leftarrow \mathrm{ids} + [\,\mathrm{sample\_next}(z, \tau, k, \rho)\,]

What does each symbol mean?

Symbol	Meaning	In code	Shape
$\mathrm{ids}$	Every id so far. It starts as a copy of seed_ids and only grows	`ids`	python list
$w$	The longest input the model is given	`window`	scalar
$x$	The ids the model sees on this step	`np.array(ids[-window:])`	(T,)
$T$	The length of x, the smaller of len(ids) and window	not named in the code	scalar
$Z$	One row of logits per position in x	`logits`	(T, V)
$z$	The last row of Z	`logits[-1]`	(V,)
$\tau, k, \rho$	The settings passed through to sample_next	`temperature, top_k, top_p`	scalars

Tiny numerical example

Step through the list as it grows. Change the settings to compare a sampled run with a greedy one. The first steps have a window shorter than window, and later steps drop the oldest token from what the model sees.

The sequence growing, one token per stepIllustrative ExampleThe logits come from the hand-picked table below, not from the model. Each row depends only on the last token, while a real model's logits depend on the whole window. The window is 4 here so that it fills quickly. The code passes a larger one. The settings and seed are those in config.py. The draws were recorded by running the real generate.generate with forward replaced by the table.

Settings

Step 1 of 7: Seed

The list starts as a copy of seed_ids. Nothing has been sampled yet.

the
cat

In each sequence: in the window outside the window appended on this step

Last token	the	cat	sat	on	mat
the	`-1`	`2`	`0.5`	`0`	`1.5`
cat	`0`	`-1`	`2`	`1`	`-1`
sat	`0.5`	`-1`	`-1`	`2`	`0`
on	`2.5`	`0`	`-1`	`-1`	`0.5`
mat	`1`	`0`	`1.5`	`0.5`	`-1`

Tensor shapes

One step of the loopgenerate.py:32

x(T,)(16,)
logits(T, V)(16, V)
logits[-1](V,)

T = seq_len = 16. T is at most seq_len: generation uses the last seq_len ids, or fewer for a short prompt. V depends on the corpus. It is not a value in config.py.

T starts at the length of the seed and stops growing at window.

Actual implementation

generate.pygenerateGitHub

28def generate(params, cfg: ModelConfig, seed_ids, max_new_tokens, window, temperature, top_k, top_p, rng):29    ids = list(seed_ids)3031    for _ in range(max_new_tokens):32        logits, _ = forward(params, np.array(ids[-window:]), cfg)33        ids.append(sample_next(logits[-1], temperature, top_k, top_p, rng)) # logits[-1] is used because the last position is the only one that has seen the whole context3435    return ids

Reading it line by line:

The window. ids[-window:] is the last window ids, or all of them if there are fewer (generate.py:32). The list itself keeps every id, and the function returns all of them.
The last position only. forward returns one row of logits for each position, and the loop uses only logits[-1] (generate.py:33). The causal mask in attention (transformer.py:82) means an earlier position cannot read later tokens, so only the last row has read the whole window. That is the reason the comment on the line gives.
No reuse between steps. Each step runs forward on the whole window again. The cache that forward returns is for the backward pass, and here it is discarded by logits, _ = (generate.py:32).
Positions follow the window. embed_forward adds pos_emb rows from 0 up to the input length (transformer.py:53). So the same token gets a different position row as the window slides, and row 0 always goes to the oldest token in the window.

Code and math connection

Equation	Role	Lines	Code
The last window ids	Computes it	generate.py:32	`logits, _ = forward(params, np.array(ids[-window:]), cfg)`
The last window ids	Calls it	transformer.py:156	`assert len(ids) <= cfg.context_len, "sequence longer than the position table"`
The last window ids	Tests it	tests/test_generate.py:75	`assert len(out) == 26`
Logits of the last position only	Computes it	generate.py:32	`logits, _ = forward(params, np.array(ids[-window:]), cfg)`
Logits of the last position only	Computes it	generate.py:33	`ids.append(sample_next(logits[-1], temperature, top_k, top_p, rng)) # logits[-1] is used because the last position is the only one that has seen the whole context`
Sample and append	Computes it	generate.py:33	`ids.append(sample_next(logits[-1], temperature, top_k, top_p, rng)) # logits[-1] is used because the last position is the only one that has seen the whole context`
Sample and append	Computes it	generate.py:31	`for _ in range(max_new_tokens):`
Sample and append	Tests it	tests/test_generate.py:66	`assert len(out) == 13 and out[:3] == [1, 2, 3]`

Where is it used?

Source

generate.py, lines 28-35, generate

PDF reference

Section 4.8, p. 15

Used by

Called by: talk.main, train.main

Tested by: tests.test_generate.test_generate_length_and_range, tests.test_generate.test_generate_window_keeps_long_sequences_legal, tests.test_generate.test_greedy_generation_is_deterministic

Calls it: transformer.py, line 156 (forward), for The last window ids
Tests it: tests/test_generate.py, line 75 (test_generate_window_keeps_long_sequences_legal), for The last window ids
Tests it: tests/test_generate.py, line 66 (test_generate_length_and_range), for Sample and append

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
The last window ids	Computes it	`generate.py`	`generate`	32	GitHub: generate.py, line 32
Logits of the last position only	Computes it	`generate.py`	`generate`	32	GitHub: generate.py, line 32
Logits of the last position only	Computes it	`generate.py`	`generate`	33	GitHub: generate.py, line 33
Sample and append	Computes it	`generate.py`	`generate`	33	GitHub: generate.py, line 33
Sample and append	Computes it	`generate.py`	`generate`	31	GitHub: generate.py, line 31

What train.py and talk.py pass

generate has two callers in the pipeline. Both pass the same window and sampling settings. They differ in the seed and in how many tokens they ask for.

Argument	`train.main`	`talk.main`
`seed_ids`	The first 3 ids of the first validation window	The encoded prompt
`max_new_tokens`	The literal 30	`config.max_new_tokens`, which is `50`
`window`	`config.seq_len`, which is `16`	`config.seq_len`, which is `16`
`temperature`	`0.8`	`0.8`
`top_k`	`0`	`0`
`top_p`	`0.9`	`0.9`
`rng`	The generator also used to shuffle batches	Its own generator, kept across prompts

train.pymainGitHub

72        seed_ids = [int(i) for i in va_x.iloc[0][:3]]

train.pymainGitHub

73        out = generate(params, cfg, seed_ids, 30, config.seq_len,74                       config.temperature, config.top_k, config.top_p, rng)

talk.pymainGitHub

24        out = generate(params, cfg, ids, config.max_new_tokens, config.seq_len,25                       config.temperature, config.top_k, config.top_p, rng)

The window is 16 tokens, while the pos_emb table has 32 rows. forward asserts that its input fits the table (transformer.py:156). Embeddings shows what this means for the rows past the window.

train.main shares one generator between sampling and batch shuffling, so printing a sample changes the next epoch's batch order. Training Loop covers this.

talk.main creates its generator once, before the prompt loop. So a second prompt does not restart the random sequence, while a new run of talk.py does.

Defaults

These are the values in config.py. top_k is 0, so top-k is off by default. Greedy decoding is also off, because temperature is above the guard.

Name	Value	State	Effect	Line
`temperature`	`0.8`	On	The guard does not fire, and the logits are divided by this value.	config.py:33
`top_k`	`0`	Off	0 < top_k is false, so the top-k branch is skipped.	config.py:34
`top_p`	`0.9`	On	top_p < 1.0 is true, so the top-p branch runs.	config.py:35
`max_new_tokens`	`50`	talk.py only	train.py passes its own literal instead.	config.py:36
`seq_len`	`16`	On	Both callers pass it as window. It is also the training window length.	config.py:7
`seed`	`42`	On	Both callers create their generator from it.	config.py:39

What the model is and is not

The model is a text continuation model. Given a list of token ids, it gives probabilities for the next token, and the loop extends the list one token at a time. talk.py describes itself the same way:

talk.pymainGitHub

11    print("Type the start of a text and press Enter. Empty line quits.")

It is not an assistant. The code gives no basis for treating it as one:

No conversation. Each prompt is encoded and generated from alone (talk.py:22), so nothing from an earlier prompt reaches the model. The only thing that carries over is the generator's position.
No message format. The vocabulary has <unk>, characters and merged tokens. It has no tokens that mark a speaker, a question or an answer.
No stopping. The only limit is max_new_tokens, so the loop always appends exactly that many tokens. It can stop in the middle of a sentence.
Output is the continuation only. talk.py prints just the new tokens (talk.py:27). A blank line quits (talk.py:19).

Not in This Model lists what the code does not do.