Central-difference gradient of every element of x
- Reached from
- Neither train.main nor talk.main
- Used by, in the pipeline
- Nothing
- Used by, in tests
tests.test_gradcheck.test_numerical_grad_on_quadratic- Mathematics
- Central difference
gradcheck.py
Finite-difference gradient checker, used only by tests
Where it fits
0 of 3 functions, classes and methods here are reachable from train.main, and 0 from talk.main. Only tests use this file.
Imports and used by
- Imports from this repository
- None
- Other imports
numpy- Imported by
tests/test_gradcheck.py- Used by, in the pipeline
- No file
- Used by, in tests
tests/test_gradcheck.py
Mathematics used
| Concept | What this file does | Lines | Status |
|---|---|---|---|
| Central difference | Computes it, Writes it in place | 12, 31 | Implemented |
| Relative error | Computes it, Calls it | 17, 31 | Implemented |
| Checking every parameter | Computes it | 31, 32 | Implemented |
Functions and constants
Relative difference between two scalars
- Reached from
- Neither train.main nor talk.main
- Used by, in the pipeline
gradcheck.check_params- Used by, in tests
- Nothing
- Mathematics
- Relative error
def check_params(loss_fn, params, grads, rng, samples_per_param = 5, h = 1e-5)Lines 19-33GitHub: gradcheck.check_paramsWorst relative error per parameter at randomly sampled indices
- Reached from
- Neither train.main nor talk.main
- Used by, in the pipeline
- Nothing
- Used by, in tests
tests.test_gradcheck.test_every_tensor_passes- Calls
gradcheck.relative_error- Mathematics
- Checking every parameter, Central difference, Relative error
Source
import numpy as npdef numerical_grad(f, x, h = 1e-5): g = np.zeros_like(x) for idx in np.ndindex(*x.shape): old = x[idx] x[idx] = old + h fp = f(x) x[idx] = old - h fm = f(x) x[idx] = old g[idx] = (fp - fm) / (2 * h) return gdef relative_error(a, b, eps = 1e-12): return abs(a - b) / max(abs(a), abs(b), eps)def check_params(loss_fn, params, grads, rng, samples_per_param = 5, h = 1e-5): worst: dict[str, float] = {} for name, W in params.items(): w = 0.0 for _ in range(samples_per_param): idx = tuple(int(rng.integers(s)) for s in W.shape) old = W[idx] W[idx] = old + h fp = loss_fn(params) W[idx] = old - h fm = loss_fn(params) W[idx] = old w = max(w, relative_error(grads[name][idx], (fp - fm) / (2 * h))) worst[name] = w return worst