Sampling

sample_next receives one row of logits and returns one token id. It runs these steps in this order:

If temperature is below 1e-6, return the index of the largest logit and stop.
Compute p, the softmax of the logits divided by temperature.
If 0 < top_k < len(p), apply top-k.
If top_p < 1.0, apply top-p.
Draw one token from p with the generator.

Code listing: generate.sample_next

generate.pysample_nextGitHub

7def sample_next(logits, temperature, top_k, top_p, rng):8    if temperature < 1e-6:9        return int(np.argmax(logits))1011    p = softmax_forward(logits / temperature)1213    if 0 < top_k < len(p):14        cutoff = np.sort(p)[-top_k] # # k-th largest probabilit15        p = np.where(p >= cutoff, p, 0.0) # zero everything below it16        p = p / p.sum() # renormalise so it sums to 1 again1718    if top_p < 1.0:19        order = np.argsort(-p)20        cum = np.cumsum(p[order])21        keep = order[: np.searchsorted(cum, top_p) + 1]22        q = np.zeros_like(p)23        q[keep] = p[keep]24        p = q / q.sum()2526    return int(rng.choice(len(p), p=p))

At the settings in config.py, steps 2, 4 and 5 run. Step 1 does not fire and step 3 is skipped. Autoregressive Generation lists those settings and shows how the loop calls this function.

This page has no backward pass. Sampling runs after the model and computes no loss or gradient, and nothing in generate.py calls a backward function. Every number here is an Illustrative Example, except the panel headed Actual Model Output at the end.

In this page's equations, z is the logits and the temperature is written with the Greek letter tau. T is not used for it, because on this site T is the sequence length.

Greedy guard

What is it?

Greedy guardImplemented

Sampling

The first two lines of sample_next. If temperature is below 1e-6, the function returns the index of the largest logit at once. It does not compute a softmax, and it does not read top_k, top_p or rng.

Why does the model need it?

The next line divides the logits by temperature, so a temperature of 0 cannot reach it. The guard handles that case and gives a way to get the same token every time from the same logits.

Mathematics

Greedy guard #

\tau < 10^{-6} \implies \mathrm{next} = \arg\max_i z_i

What does each symbol mean?

Symbol	Meaning	In code	Shape
$\tau$	The temperature. T is not used for it on this site, because T is the sequence length	`temperature`	scalar
$z$	The logits of one position	`logits`	(V,)
$\mathrm{next}$	The chosen token id	`the value returned by sample_next`	scalar

Tiny numerical example

The guard, on hand-picked logitsIllustrative ExampleHand-picked logits, not model output.

logits	temperature	Returned token
`[0.1, 2, 1]`	`0`	`1`
`[0.1, 2, 1]`	`5e-7`	`1`
`[1, 3, 3, 2]`	`0`	`1`

At exactly 0.000001 the test temperature < 1e-6 is false, so the code computes a distribution: [0, 1, 0]. It is close to greedy, because dividing by so small a number leaves one exponential far above the rest, but it still draws from the generator.

np.argmax returns the first index among equal values, so a tie goes to the lowest index. That is the third row of the table.

Tensor shapes

From the model's logits to the row sample_next reads. The result is one int, with no axesgenerate.py:9

logits(T, V)(16, V)
logits[-1](V,)

T = seq_len = 16. T is at most seq_len: generation uses the last seq_len ids, or fewer for a short prompt. V depends on the corpus. It is not a value in config.py.

Actual implementation

generate.pysample_nextGitHub

8    if temperature < 1e-6:9        return int(np.argmax(logits))

The return happens before rng is touched, so a greedy step draws nothing from the generator. Two tests check this: the same token twenty times, and the same output from two different seeds.

Code and math connection

Equation	Role	Lines	Code
Greedy guard	Computes it	generate.py:8	`if temperature < 1e-6:`
Greedy guard	Computes it	generate.py:9	`return int(np.argmax(logits))`
Greedy guard	Tests it	tests/test_generate.py:29	`assert all(sample_next(np.array([0.1, 2.0, 1.0]), 0.0, 0, 1.0, rng) == 1`
Greedy guard	Tests it	tests/test_generate.py:82	`assert a == b`

Where is it used?

Source

generate.py, lines 7-26, sample_next

PDF reference

Section 4.7, p. 14

Used by

Called by: generate.generate

Tested by: tests.test_generate.probs_used, tests.test_generate.test_temperature_zero_is_greedy, tests.test_generate.test_top_k_one_is_greedy

Tests it: tests/test_generate.py, line 29 (test_temperature_zero_is_greedy), for Greedy guard
Tests it: tests/test_generate.py, line 82 (test_greedy_generation_is_deterministic), for Greedy guard

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
Greedy guard	Computes it	`generate.py`	`sample_next`	8	GitHub: generate.py, line 8
Greedy guard	Computes it	`generate.py`	`sample_next`	9	GitHub: generate.py, line 9

Temperature

What is it?

TemperatureImplemented

Sampling

Divides the logits by temperature, then applies softmax_forward. The result is a probability for every token in the vocabulary.

Why does the model need it?

The model gives logits, and a random choice needs probabilities. Dividing first changes how far apart the logits are. Below 1 the gaps grow and the largest probability rises. Above 1 the gaps shrink and the probabilities move closer together.

Mathematics

Softmax of the logits divided by temperature #

p = \mathrm{softmax}\left(\frac{z}{\tau}\right)

The softmax itself is on Softmax & Cross-Entropy. It subtracts the largest value first, so large logits do not overflow.

What does each symbol mean?

Symbol	Meaning	In code	Shape
$z$	The logits of one position	`logits`	(V,)
$\tau$	The temperature, at least 1e-6 on this line because of the guard above it	`temperature`	scalar
$p$	The probabilities of the next token	`p`	(V,)

Tiny numerical example

One set of logits at three temperaturesIllustrative ExampleHand-picked logits [2, 1, 0], not model output.

temperature	logits / temperature	p
`0.5`	`[4, 2, 0]`	`[0.8668, 0.1173, 0.0159]`
`1`	`[2, 1, 0]`	`[0.6652, 0.2447, 0.09]`
`2`	`[1, 0.5, 0]`	`[0.5065, 0.3072, 0.1863]`

These inputs are the ones in test_temperature_examples. At temperature 1 the logits pass through unchanged.

Tensor shapes

Temperature keeps the shapegenerate.py:11

logits(V,)
p(V,)

V depends on the corpus. It is not a value in config.py.

Actual implementation

generate.pysample_nextGitHub

11    p = softmax_forward(logits / temperature)

ops.pysoftmax_forwardGitHub

22def softmax_forward(x):23    e = np.exp(x - np.max(x, axis=-1, keepdims=True))24    return e / e.sum(axis=-1, keepdims=True)

Code and math connection

Equation	Role	Lines	Code
Softmax of the logits divided by temperature	Computes it	generate.py:11	`p = softmax_forward(logits / temperature)`
Softmax of the logits divided by temperature	Calls it	ops.py:24	`return e / e.sum(axis=-1, keepdims=True)`
Softmax of the logits divided by temperature	Tests it	tests/test_generate.py:22	`assert np.allclose(probs_used(z, 0.5), [0.867, 0.117, 0.016], atol=1e-3)`

Where is it used?

Source

generate.py, lines 7-26, sample_next

PDF reference

Section 4.7, Eq. 60, p. 14

Used by

Called by: generate.generate

Tested by: tests.test_generate.probs_used, tests.test_generate.test_temperature_zero_is_greedy, tests.test_generate.test_top_k_one_is_greedy

Calls it: ops.py, line 24 (softmax_forward), for Softmax of the logits divided by temperature
Tests it: tests/test_generate.py, line 22 (test_temperature_examples), for Softmax of the logits divided by temperature

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
Softmax of the logits divided by temperature	Computes it	`generate.py`	`sample_next`	11	GitHub: generate.py, line 11

Top-k

What is it?

Top-kDerived from implementation

Sampling

An optional step that runs only if 0 < top_k < len(p). It finds the k-th largest probability, sets every probability below it to zero, and renormalises. A token equal to the cut-off stays, so ties can leave more than k tokens.

Why does the model need it?

It stops the choice from landing on unlikely tokens by limiting how many tokens can be chosen. The model does not need it at the committed settings, because top_k is 0 in config.py and the step is skipped.

Mathematics

The cut-off #

c = k\text{-th largest value of } p \text{, counting repeats}

Zero everything below the cut-off #

p'_i = \begin{cases} p_i & p_i \geq c \\ 0 & \text{otherwise} \end{cases}

Renormalise #

p''_i = \frac{p'_i}{\sum_j p'_j}

The branch runs only if 0 < top_k < len(p). So top_k = 0 skips it, a negative value skips it, and so does a value equal to or above the vocabulary size.

How the cut-off is computed. np.sort(p) sorts from smallest to largest, and [-top_k] counts back from the end (generate.py:14). Repeated values each take a place in that list. With top_k = 2 and p holding three equal largest values, the cut-off is that shared value.

What happens with ties. The test is p >= cutoff (generate.py:15), not a count. Every token equal to the cut-off stays, so more than top_k tokens can remain. Nothing breaks the tie.

The PDF gives a worked example of top-k on p. 14 and no equation, so this concept is marked as derived from the implementation.

What does each symbol mean?

Symbol	Meaning	In code	Shape
$p$	The probabilities after temperature	`p`	(V,)
$k$	How many of the largest values to count down to	`top_k`	scalar
$c$	The cut-off	`cutoff`	scalar
$p'$	The probabilities with the lower values set to zero	`the result of np.where`	(V,)
$p''$	The same values, scaled to sum to 1	`p, rebound`	(V,)

Tiny numerical example

Top-k without and with a tieIllustrative ExampleHand-picked logits at temperature 1, not model output.

Quantity	Value
`p`	`[0.5, 0.3, 0.15, 0.05]`
`p`, sorted largest first	`[0.5, 0.3, 0.15, 0.05]`
`cutoff`, for `top_k = 2`	`0.3`
Tokens kept	`2`
`p` after renormalising	`[0.625, 0.375, 0, 0]`

Quantity	Value
`p`	`[0.2855, 0.2855, 0.2855, 0.105, 0.0386]`
`p`, sorted largest first	`[0.2855, 0.2855, 0.2855, 0.105, 0.0386]`
`cutoff`, for `top_k = 2`	`0.2855`
Tokens kept	`3`
`p` after renormalising	`[0.3333, 0.3333, 0.3333, 0, 0]`

In the second table top_k is 2 and three tokens remain, because all three equal the cut-off.

Tensor shapes

Top-k changes values, not the shape. Removed tokens stay as zerosgenerate.py:15

p before(V,)
p after(V,)

V depends on the corpus. It is not a value in config.py.

Actual implementation

generate.pysample_nextGitHub

13    if 0 < top_k < len(p):14        cutoff = np.sort(p)[-top_k] # # k-th largest probabilit15        p = np.where(p >= cutoff, p, 0.0) # zero everything below it16        p = p / p.sum() # renormalise so it sums to 1 again

Code and math connection

Equation	Role	Lines	Code
The cut-off	Computes it	generate.py:13	`if 0 < top_k < len(p):`
The cut-off	Computes it	generate.py:14	`cutoff = np.sort(p)[-top_k] # # k-th largest probabilit`
Zero everything below the cut-off	Computes it	generate.py:15	`p = np.where(p >= cutoff, p, 0.0) # zero everything below it`
Zero everything below the cut-off	Tests it	tests/test_generate.py:41	`assert all(sample_next(z, 1.0, 1, 1.0, rng) == 1 for _ in range(30))`
Renormalise	Computes it	generate.py:16	`p = p / p.sum() # renormalise so it sums to 1 again`
Renormalise	Tests it	tests/test_generate.py:35	`assert np.allclose(probs_used(z, top_k=2), [0.625, 0.375, 0, 0])`

Where is it used?

Source

generate.py, lines 7-26, sample_next

PDF reference

Not stated in the PDF

Used by

Called by: generate.generate

Tested by: tests.test_generate.probs_used, tests.test_generate.test_temperature_zero_is_greedy, tests.test_generate.test_top_k_one_is_greedy

Tests it: tests/test_generate.py, line 41 (test_top_k_one_is_greedy), for Zero everything below the cut-off
Tests it: tests/test_generate.py, line 35 (test_top_k_example), for Renormalise

Status

Derived from implementation

Implemented in

Equation	Role	File	Function	Lines	GitHub
The cut-off	Computes it	`generate.py`	`sample_next`	13	GitHub: generate.py, line 13
The cut-off	Computes it	`generate.py`	`sample_next`	14	GitHub: generate.py, line 14
Zero everything below the cut-off	Computes it	`generate.py`	`sample_next`	15	GitHub: generate.py, line 15
Renormalise	Computes it	`generate.py`	`sample_next`	16	GitHub: generate.py, line 16

Top-p

What is it?

Top-pDerived from implementation

Sampling

An optional step that runs only if top_p < 1.0. It sorts the tokens from most to least likely, adds up their probabilities, keeps the shortest run from the top whose sum reaches top_p, and renormalises. The token that crosses top_p is kept.

Why does the model need it?

It removes the unlikely tail. The number of tokens kept follows the distribution, so a confident distribution keeps few and an unsure one keeps many. It is on in config.py.

Mathematics

Sort #

\pi = \text{token indices sorted by } p \text{, largest first}

Cumulative sum #

C_m = \sum_{j=0}^{m} p_{\pi_j}

The smallest prefix that reaches top_p #

s = \min \{ m : C_m \geq \rho \}, \qquad \text{keep } \pi_0, \dots, \pi_s

Renormalise #

q_i = \frac{p_i}{\sum_{m=0}^{s} p_{\pi_m}} \text{ for kept } i, \qquad q_i = 0 \text{ otherwise}

The prefix rule. np.searchsorted(cum, top_p) returns the first position where cum is at least top_p, counting from 0. The slice then adds 1, so the token at that position is kept (generate.py:21). The kept tokens are always a run from the top of the sorted order, and the sum of what is kept is at least top_p, not at most.

If cum lands exactly on top_p, the prefix ends at that token.
If top_p is 0 or below, position 0 already qualifies, so one token is kept.
If cum never reaches top_p, for example through rounding at a value just below 1, searchsorted returns len(cum), and the slice keeps every token.
If top-k ran first, its removed tokens are in the sort as zeros. They sit at the end of the order and add nothing to cum.

Ties at the cut. The code does not treat ties specially here, unlike top-k. If the cut falls inside a group of equal probabilities, some of them stay and some go. np.argsort(-p) (generate.py:19) uses NumPy's default sort, which does not promise an order for equal values, so the code does not define which ones stay. The number of tokens kept does not depend on that order.

The PDF gives a worked example of top-p on p. 15 and no equation, so this concept is marked as derived from the implementation.

What does each symbol mean?

Symbol	Meaning	In code	Shape
$p$	The probabilities after temperature and top-k	`p`	(V,)
$\rho$	The threshold	`top_p`	scalar
$\pi$	Token indices, largest probability first	`order`	(V,)
$C$	Running sum of the sorted probabilities	`cum`	(V,)
$s$	The first position where C reaches the threshold, counting from 0	`np.searchsorted(cum, top_p)`	scalar
$q$	The kept probabilities, scaled to sum to 1	`q / q.sum()`	(V,)

Tiny numerical example

Top-p: the token that crosses is keptIllustrative ExampleHand-picked logits at temperature 1, not model output.

Sorted position	Token	p	cum	Kept	After renormalising
`0`	`0`	`0.5`	`0.5`	Yes	`0.5263`
`1`	`1`	`0.3`	`0.8`	Yes	`0.3158`
`2`	`2`	`0.15`	`0.95`	Yes	`0.1579`
`3`	`3`	`0.05`	`1`	No	`0`

Sorted position	Token	p	cum	Kept	After renormalising
`0`	`0`	`0.25`	`0.25`	Yes	`0.5`
`1`	`1`	`0.25`	`0.5`	Yes	`0.5`
`2`	`2`	`0.25`	`0.75`	No	`0`
`3`	`3`	`0.25`	`1`	No	`0`

In the first table cum is below top_p at position 1 and above it at position 2, so position 2 is the crossing token and it is kept. These are the inputs of test_top_p_example. In the second table cum equals top_p at position 1, so the prefix stops there.

Tensor shapes

Top-p sorts and sums over all V tokens. Only keep is shorter, and p keeps its shapegenerate.py:21

order(V,)
cum(V,)
p(V,)

V depends on the corpus. It is not a value in config.py.

keep is the one shorter array, with length s + 1. q has the full length, so removed tokens remain as zeros.

Actual implementation

generate.pysample_nextGitHub

18    if top_p < 1.0:19        order = np.argsort(-p)20        cum = np.cumsum(p[order])21        keep = order[: np.searchsorted(cum, top_p) + 1]22        q = np.zeros_like(p)23        q[keep] = p[keep]24        p = q / q.sum()

Code and math connection

Equation	Role	Lines	Code
Sort	Computes it	generate.py:18	`if top_p < 1.0:`
Sort	Computes it	generate.py:19	`order = np.argsort(-p)`
Cumulative sum	Computes it	generate.py:20	`cum = np.cumsum(p[order])`
The smallest prefix that reaches top_p	Computes it	generate.py:21	`keep = order[: np.searchsorted(cum, top_p) + 1]`
The smallest prefix that reaches top_p	Tests it	tests/test_generate.py:47	`assert np.allclose(probs_used(z, top_p=0.9), [0.526, 0.316, 0.158, 0], atol=1e-3)`
The smallest prefix that reaches top_p	Tests it	tests/test_generate.py:54	`assert (probs_used(unsure, top_p=0.9) > 0).sum() == 4`
Renormalise	Computes it	generate.py:23	`q[keep] = p[keep]`
Renormalise	Computes it	generate.py:24	`p = q / q.sum()`

Where is it used?

Source

generate.py, lines 7-26, sample_next

PDF reference

Not stated in the PDF

Used by

Called by: generate.generate

Tested by: tests.test_generate.probs_used, tests.test_generate.test_temperature_zero_is_greedy, tests.test_generate.test_top_k_one_is_greedy

Tests it: tests/test_generate.py, line 47 (test_top_p_example), for The smallest prefix that reaches top_p
Tests it: tests/test_generate.py, line 54 (test_top_p_adapts_to_confidence), for The smallest prefix that reaches top_p

Status

Derived from implementation

Implemented in

Equation	Role	File	Function	Lines	GitHub
Sort	Computes it	`generate.py`	`sample_next`	18	GitHub: generate.py, line 18
Sort	Computes it	`generate.py`	`sample_next`	19	GitHub: generate.py, line 19
Cumulative sum	Computes it	`generate.py`	`sample_next`	20	GitHub: generate.py, line 20
The smallest prefix that reaches top_p	Computes it	`generate.py`	`sample_next`	21	GitHub: generate.py, line 21
Renormalise	Computes it	`generate.py`	`sample_next`	23	GitHub: generate.py, line 23
Renormalise	Computes it	`generate.py`	`sample_next`	24	GitHub: generate.py, line 24

Random choice

What is it?

Random choiceImplemented

Sampling

The last line of sample_next. It draws one token id with the probabilities left after the earlier steps, using the generator it was given. A token with probability zero cannot be drawn.

Why does the model need it?

Generation needs one token to append. Drawing at random lets the same logits give different tokens. The generator is seeded, so a run with the same seed and the same calls repeats the same draws.

Mathematics

One draw from the final probabilities #

P(\mathrm{next} = i) = p_i

What does each symbol mean?

Symbol	Meaning	In code	Shape
$p$	The final probabilities	`p`	(V,)
$\mathrm{next}$	The chosen token id	`the value returned by sample_next`	scalar
$V$	The number of tokens to choose from	`len(p)`	scalar

Tiny numerical example

1000 draws from one distributionIllustrative ExampleHand-picked logits, not model output. The settings are those in config.py. The draws were recorded by calling the real sample_next with np.random.default_rng(42), because that generator is not reproduced in this site's engine. p comes from the engine.

First draws: [1, 0, 1, 1, 0, 2, 1, 1, 0, 0, 0, 2]

Token	p	Times drawn
`0`	`0.6945`	`689`
`1`	`0.199`	`206`
`2`	`0.1065`	`105`
`3`	`0`	`0`
`4`	`0`	`0`

Tokens 3 and 4 have probability 0 and are never drawn. Running the same calls again from the same seed gave the same draws.

Tensor shapes

The array before the draw. The draw itself returns one int, with no axesgenerate.py:26

p after top-k(V,)
p after top-p(V,)

V depends on the corpus. It is not a value in config.py.

The draw returns one number, and int(...) makes it a plain Python integer.

Actual implementation

generate.pysample_nextGitHub

26    return int(rng.choice(len(p), p=p))

sample_next does not create the generator. Its caller passes one in, and both callers create it from 42:

train.pymainGitHub

37    rng = np.random.default_rng(config.seed)

talk.pymainGitHub

9    rng = np.random.default_rng(config.seed)

Code and math connection

Equation	Role	Lines	Code
One draw from the final probabilities	Computes it	generate.py:26	`return int(rng.choice(len(p), p=p))`
One draw from the final probabilities	Calls it	train.py:37	`rng = np.random.default_rng(config.seed)`
One draw from the final probabilities	Calls it	talk.py:9	`rng = np.random.default_rng(config.seed)`
One draw from the final probabilities	Tests it	tests/test_generate.py:10	`self.p = np.asarray(p)`

Where is it used?

Source

generate.py, lines 7-26, sample_next

PDF reference

Section 4.7, p. 15

Used by

Called by: generate.generate

Tested by: tests.test_generate.probs_used, tests.test_generate.test_temperature_zero_is_greedy, tests.test_generate.test_top_k_one_is_greedy

Calls it: train.py, line 37 (main), for One draw from the final probabilities
Calls it: talk.py, line 9 (main), for One draw from the final probabilities
Tests it: tests/test_generate.py, line 10 (choice), for One draw from the final probabilities

Status

Implemented

Implemented in

Equation	Role	File	Function	Lines	GitHub
One draw from the final probabilities	Computes it	`generate.py`	`sample_next`	26	GitHub: generate.py, line 26

Probability lab

Edit the logits, temperature, top_k and top_p, or start from a preset. The first preset uses the settings in config.py. Every result comes from this site's mirror of sample_next. The presets' results are compared with the real function by the golden tests.

The lab stops at the distribution. It does not draw, because NumPy's generator is not reproduced in this site's engine.

logits
	token 0	token 1	token 2	token 3	token 4
logits

temperature

top_k

top_p

p = softmax_forward(logits / 0.8).
top_k = 0: 0 < top_k < len(p) is false, so top-k is skipped.
top_p = 0.9: the sum first reaches it within the first 3 sorted tokens, and the token that crosses it is kept.
rng.choice would then draw from the 3 kept tokens.

Before truncation: softmax of logits / temperatureIllustrative ExampleHand-picked logits, not model output. Accent bars are the kept set. Grey bars are the tokens the later steps remove.

token 00.6471

token 10.1854

token 20.0992

token 30.0531 removed

token 40.0152 removed

Data table

Label	Probability
token 0	0.6471
token 1	0.1854
token 2	0.0992
token 3	0.0531 removed
token 4	0.0152 removed

After truncation: what rng.choice receivesIllustrative ExampleThe kept probabilities, scaled to sum to 1. Both charts run from 0 to 1, so the bars can be compared.

token 00.6945

token 10.1990

token 20.1065

token 30.0000 removed

token 40.0000 removed

Data table

Label	Probability
token 0	0.6945
token 1	0.1990
token 2	0.1065
token 3	0.0000 removed
token 4	0.0000 removed

Token	logits	After temperature	After top-k	Sort position	cum	Final p	Kept
token 0	`2`	`0.6471`	`0.6471`	`0`	`0.6471`	`0.6945`	Yes
token 1	`1`	`0.1854`	`0.1854`	`1`	`0.8324`	`0.199`	Yes
token 2	`0.5`	`0.0992`	`0.0992`	`2`	`0.9317`	`0.1065`	Yes
token 3	`0`	`0.0531`	`0.0531`	`3`	`0.9848`	`0`	No
token 4	`-1`	`0.0152`	`0.0152`	`4`	`1`	`0`	No

Actual Model Output

The distributions of a real model, for one prompt. The lab above does not read these numbers, and these panels do not read the lab.

Next-token probabilities, softmax of the last logitsActual Model OutputCommit 66da45b. Checkpoint model.npz sha256 74de2ec88a63. Seed 42, from config.py, since a checkpoint does not record its own. Read from checkpoints/model, an ignored local file, trained on data/corpus.csv (git-ignored, and not at the pinned commit), with training settings that by inference differ from config.py.The prompt is hand-picked: "Once there was a little ", which encodes to 12 tokens. The probabilities are what the model returned for this one prompt. The 10 most likely tokens are listed, and the other 565 tokens share one bar.

p0.0700

b0.0575

s0.0492

f0.0451

boy0.0357

c0.0342

g0.0245

d0.0222

th0.0153

little0.0130

all other tokens0.6333

Data table

Label	Probability
p	0.0700
b	0.0575
s	0.0492
f	0.0451
boy	0.0357
c	0.0342
g	0.0245
d	0.0222
th	0.0153
little	0.0130
all other tokens	0.6333

What sample_next draws from, at the config.py settingsActual Model OutputCommit 66da45b. Checkpoint model.npz sha256 74de2ec88a63. Seed 42, from config.py, since a checkpoint does not record its own. Read from checkpoints/model, an ignored local file, trained on data/corpus.csv (git-ignored, and not at the pinned commit), with training settings that by inference differ from config.py.The same prompt, with temperature 0.8, top_k 0 and top_p 0.9, read from the real sample_next before it draws. 84 tokens keep a non-zero probability.

p0.1213

b0.0949

s0.0781

f0.0699

boy0.0523

c0.0496

g0.0327

d0.0288

th0.0182

little0.0148

all other tokens0.4395

Data table

Label	Probability
p	0.1213
b	0.0949
s	0.0781
f	0.0699
boy	0.0523
c	0.0496
g	0.0327
d	0.0288
th	0.0182
little	0.0148
all other tokens	0.4395

Where these numbers came from

Repository commit	`66da45bea72176fa5d2f8dba2ea1c7e597e30dd3`
`model.npz` sha256	`74de2ec88a63ceb131a27e03cc63070c6d8c4174cc78bc189195e9356e84550e`
`model.json` sha256	`f93002949162068d573aab9b38990b12aec7f32b468ec9eaaef986075df7d3b4`
`model.tok.json` sha256	`bf874ab4e4de2b61cb936f38635a7c589c5c0d29879947a5b45d13be2456b17d`
Corpus `data/corpus.csv` sha256	`c1bec80f274294e6ce989fcc63fd7022a206f415e7d3df85e82c410e56dc4f4f`
Tokenizer rebuilt from this corpus equals the checkpoint's	Yes
Model shape, from the checkpoint	`{"context_len":32,"emb_dim":32,"ff_dim":128,"ln_eps":0.00001,"num_heads":2,"num_layers":2,"vocab_size":575}`
`config.py`, when this was exported	`{"adam_eps":1e-8,"batch_size":16,"beta1":0.9,"beta2":0.999,"context_len":32,"corpus_column":"text","corpus_path":"data/corpus.csv","emb_dim":32,"epochs":20,"ff_dim":128,"grad_clip":1,"head_dim":16,"ln_eps":0.00001,"lr_max":0.001,"lr_min":0.0001,"max_new_tokens":50,"max_rows":2000,"num_heads":2,"num_layers":2,"num_merges":500,"seed":42,"seq_len":16,"stride":1,"temperature":0.8,"top_k":0,"top_p":0.9,"val_fraction":0.1,"warmup_steps":50,"weight_decay":0.01}`
Seed	`42`
Environment	`Python 3.14.4, NumPy 2.5.2, pandas 3.1.0`