You ran numpy.testing.assert_allclose and got a wall of text opening with Not equal to tolerance rtol=1e-07, atol=0. The odd part: the same two arrays passed np.allclose a moment earlier. Your maths did not change between the two calls, so the disagreement is not about correctness. It is about which rule, and which default tolerances, each function applies.
This page decodes that error line by line, states the exact comparison both functions use, and shows the one default that causes almost every "passes with allclose but fails with assert_allclose" report. Then it covers how to choose a tolerance from your data rather than guessing.
A default assert_allclose failure in recent NumPy looks like this:
Not equal to tolerance rtol=1e-07, atol=0 Mismatched elements: 1 / 3 (33.3%) Max absolute difference among violations: 1.e-06 Max relative difference among violations: 3.33e-07 ACTUAL: array([1. , 2. , 3.000001]) DESIRED: array([1., 2., 3.])
Read it as five separate facts:
|a - b| across the failing elements only.|a - b| / |b| across those same elements.On NumPy before 1.25 the wording is terser: Max absolute difference: and Max relative difference: with no "among violations", and the arrays are labelled x: and y:. Same meaning. The "among violations" phrasing simply tells you the maximum was taken over the failing elements, not the whole array.
The single most useful diagnostic reading: if max absolute difference is tiny (say 1e-9) while max relative difference is enormous (1.0, or inf), you are looking at a near-zero element, and atol is your problem, not rtol.
Elementwise test: |a - b| <= atol + rtol * |b|
Here b is the reference — the second argument to allclose, or the desired argument to assert_allclose.
np.allclose, np.isclose and numpy.testing.assert_allclose all apply exactly this inequality. np.isclose returns the per-element boolean array; np.allclose is np.isclose(...).all(); assert_allclose raises when any element fails. Nothing else differs in the maths.
Two consequences worth internalising. First, the test is asymmetric: rtol scales |b|, so swapping the arguments can change the outcome when the two values differ in magnitude. Keep your known-good reference as the second argument consistently. Second, the right-hand side has two additive terms — an absolute floor (atol) and a relative allowance (rtol * |b|) — and whichever is larger dominates. That is the whole game.
The two functions ship with different defaults, and this is the crux of the entire "why does it disagree" question.
| Function | rtol | atol | equal_nan |
|---|---|---|---|
np.allclose / np.isclose | 1e-5 | 1e-8 | False |
numpy.testing.assert_allclose | 1e-7 | 0 | True |
assert_allclose is stricter on both knobs, but the one that bites is atol=0. Substitute it into the rule for an element whose reference is at or near zero:
|a - b| <= 0 + 1e-7 * |b|
When b is exactly 0, the right-hand side is 0, so the only way to pass is bit-exact equality. When b is a tiny 1e-9, the allowance is 1e-16 — smaller than the noise in almost any real computation. So any value that should be zero but carries a little rounding error fails. Meanwhile np.allclose passes the same element easily, because its atol=1e-8 gives an absolute floor that 1e-9 of noise sits comfortably under:
a = np.array([1e-9, 1.0, 2.0]) b = np.array([0.0, 1.0, 2.0]) np.allclose(a, b) # True np.testing.assert_allclose(a, b) # AssertionError
The failure reports Max absolute difference among violations: 1.e-09 and Max relative difference among violations: inf (division by the zero reference). If you see that signature, do not tighten your code — set a sensible atol.
Do not squint at the truncated arrays in the traceback. Rebuild the boolean mask with the same tolerances and index into it:
import numpy as np
mask = np.isclose(a, b, rtol=1e-7, atol=0)
bad = np.where(~mask)[0]
for i in bad:
d = abs(a[i] - b[i])
r = d / abs(b[i]) if b[i] else np.inf
print(i, a[i], b[i], d, r)
Now you can see whether the offenders cluster near zero (an atol problem) or are spread across large-magnitude values (an rtol problem). If you would rather skip the boilerplate, paste both arrays into Perpendis array diff and it reports the first differing index, the max absolute and relative differences, and the ULP distance directly.
The two terms cover opposite ends of the magnitude range:
atol when your failures are small numbers that ought to be zero.|b|. It governs large-magnitude values, where a fixed absolute epsilon would be far too strict. Raise rtol when your failures are big numbers off in their last few significant digits.Most real comparisons need both: an atol sized to the near-zero noise floor and an rtol sized to the relative error of the well-scaled values. Setting only one and leaving the other at a default is how people end up here.
A ULP (unit in the last place) is the gap between adjacent representable floats at a given magnitude. It is the natural unit for "how close is close", because a fixed relative error corresponds to a roughly fixed number of ULPs regardless of scale — which is exactly the point Bruce Dawson makes in his floating-point comparison writing: absolute epsilons break at both large and small magnitudes, ULP-based thinking does not.
For a value near 1.0, one ULP equals the machine epsilon:
| dtype | significand bits | 1 ULP at 1.0 (eps) | a few ULP (relative) |
|---|---|---|---|
| float32 | 24 | 1.19e-7 | ~1e-6 |
| float64 | 53 | 2.22e-16 | ~1e-15 |
This lets you sanity-check a tolerance against the type. assert_allclose's default rtol=1e-7 is about 0.8 ULP for float32 — genuinely tight, near the limit of what single precision can promise. For float64 the same 1e-7 is roughly 450 million ULP, a very loose relative allowance. So for double-precision work the default rtol almost never trips on its own; the failures come from atol=0 at near-zero values, as above.
What is normal? For a well-conditioned float64 computation, a few ULP — a relative difference of a few times 1e-16, so an rtol around 1e-13 to 1e-15 — is the honest target. Long reductions loosen this: summing N terms in naive order accumulates error that can grow with N, so a large dot product can legitimately differ by 1e-9 or more relative. Matrix multiplications, convolutions and other long chains are reductions in disguise. For float32, treat a few times 1e-6 as the well-conditioned baseline.
torch.testing.assert_close uses the same inequality, but its defaults are chosen per input dtype — a more considered design than a single fixed rtol, and notably its atol defaults are non-zero:
| dtype | rtol | atol |
|---|---|---|
| float16 | 1e-3 | 1e-5 |
| bfloat16 | 1.6e-2 | 1e-5 |
| float32 | 1.3e-6 | 1e-5 |
| float64 | 1e-7 | 1e-7 |
assert_close also checks dtype, shape and device by default, so a float32-versus-float64 mismatch fails before tolerances are even evaluated. Pass check_dtype=False to compare across dtypes. torch.allclose mirrors NumPy's looser rtol=1e-5, atol=1e-8 instead.
Guessing tolerances is how flaky tests are born. Measure the actual disagreement once, then set thresholds a small margin above it:
abs_diff = np.abs(a - b)
denom = np.abs(b)
rel_diff = np.where(denom > 0,
abs_diff / denom, 0.0)
print("max abs:", abs_diff.max())
print("max rel:", rel_diff.max())
Set atol just above the observed max absolute difference (this covers the near-zero elements), and rtol just above the observed max relative difference (this covers the large ones). Add roughly one order of magnitude of headroom for platform variation, and no more — a tolerance loose enough to pass anything catches nothing.
The maths is deterministic; the last few ULPs are not portable. Common causes when the numbers only diverge in CI:
The fix is almost never to chase bit-exactness. It is to set an rtol/atol pair grounded in your data's real numerical error, so the test passes wherever the answer is correct and fails only when it is genuinely wrong.
Diff two arrays without writing a script — Perpendis array diff. Paste two float arrays (brackets, commas and a tensor(...) wrapper are all accepted) or load two .npy files. It reports whether they are bit-identical, the first differing index, the max absolute and p95 absolute difference, the max relative difference, the A and B values at that index, the ULP distance, and the raw float32/float64 bit patterns of both values in a collapsible panel, plus a divergence-by-class breakdown. An optional tolerance field restates the measured max against your atol. Everything runs in WebAssembly in your browser — nothing is uploaded, no account.
Both apply the rule |a - b| <= atol + rtol * |b|, but with different defaults. np.allclose uses rtol=1e-5, atol=1e-8 and equal_nan=False. numpy.testing.assert_allclose uses rtol=1e-7, atol=0 and equal_nan=True. The atol=0 is the decisive difference: it makes assert_allclose fail on values at or near zero, which is why arrays routinely pass allclose yet fail assert_allclose.
atol is an absolute floor that governs values near zero, where the relative term rtol * |b| shrinks to nothing. rtol is a relative allowance that governs large-magnitude values. If your failing elements are small numbers that should be zero, raise atol. If they are large numbers off in their last significant digits, raise rtol. Most real comparisons need both set deliberately rather than one left at a default.
For a well-conditioned float64 computation, a few ULP is normal, which is a relative difference of a few times 1e-16, so an rtol around 1e-13 to 1e-15. For float32, a few ULP is roughly 1e-6 relative. Long reductions such as large sums, dot products and matmuls accumulate more error and can legitimately differ by 1e-9 or more relative. As a scale check, one ULP near 1.0 is 2.22e-16 for float64 and 1.19e-7 for float32.
The last few ULPs are not portable. A different BLAS backend, CPU features like FMA and wider SIMD, a different thread count that reorders parallel reductions, a library version bump, or a different architecture (x86 versus ARM) all change the final bits of a computation. The fix is to set rtol and atol from your data's observed numerical error rather than chasing bit-exact equality across machines.