You ran torch.onnx.export, loaded the graph in ONNX Runtime, compared the outputs against PyTorch, and np.testing.assert_allclose failed. Before you assume the export is broken: two correct implementations of the same network will almost never produce bit-identical outputs. The question is not whether they differ, but where, by how much, and whether the shape of the difference looks like rounding or like a fault.
This page is about telling those two apart. We look at why the numbers legitimately diverge, how to compare tensors so the answer is evidence rather than a guess, what tolerances are a reasonable starting point for fp32 and fp16, and how to bisect a genuine op-level bug by layer.
Before any talk of tolerance, check the two things no tolerance can rescue: shape and dtype. A transposed axis, an NHWC-versus-NCHW layout, a dynamic axis you forgot to declare, or one side returning fp16 while the other returns fp32 all present as a "mismatch" that a looser atol only hides. torch.testing.assert_close checks dtype and shape by default and names which one failed, so run it before you reach for numbers.
Non-numerical causes to eliminate first
model.eval() so dropout is off and BatchNorm uses running stats.randn calls.dynamic_axes.The root cause is that floating-point addition is not associative. (a + b) + c and a + (b + c) can round to different results. Any reduction — the sum inside a matmul, a softmax denominator, the mean in a norm layer — has no single canonical order. PyTorch's kernel (cuDNN, oneDNN, or its own) and ONNX Runtime's or TensorRT's kernel tile and accumulate that sum differently, so the last bit or two legitimately disagree. Nothing is wrong. On top of that root cause, several specific triggers widen the gap:
torch.backends.cuda.matmul.allow_tf32 and torch.backends.cudnn.allow_tf32.a*b + c with a single rounding instead of two, and conv+bias+activation fusion reorders work. The fused result is usually more accurate, and different from the unfused reference.eps is 1e-5. A reimplementation or converter that places it differently — sqrt(var + eps) versus sqrt(var) + eps — or picks another default will make low-variance channels diverge visibly.Do not compare two totals or a mean. A mean hides one catastrophic element among thousands of good ones, and inflates when a thousand elements are each off by a bit. Localise instead. Upcast to float64 before subtracting so the difference itself is exact, then find the worst element, its absolute and relative diff, and the ULP gap.
import numpy as np
import onnxruntime as ort
import torch
model.eval()
x = torch.randn(1, 3, 224, 224)
with torch.no_grad():
ref = model(x).cpu().numpy()
sess = ort.InferenceSession(
"model.onnx",
providers=["CPUExecutionProvider"],
)
got = sess.run(None, {"input": x.numpy()})[0]
# 1. dtype and shape before anything else
assert ref.shape == got.shape, (ref.shape, got.shape)
assert ref.dtype == got.dtype, (ref.dtype, got.dtype)
# 2. upcast so the diff is not itself rounded
a = ref.astype(np.float64)
b = got.astype(np.float64)
diff = np.abs(a - b)
# 3. the worst element, not an average
i = np.unravel_index(diff.argmax(), diff.shape)
rel = diff / np.maximum(np.abs(a), 1e-12)
print("max abs", diff.max(), "at", i)
print("max rel", rel.max())
print("A", a[i], "B", b[i])
The ULP distance — the count of representable floats between two values — is the measurement that tells rounding from corruption. Zero means bit-identical; one or two means the values are a step or two apart, which is pure rounding; a large ULP gap on a handful of elements is still fine if their absolute and relative diffs are small.
def ulp_diff(a, b):
ia = np.float32(a).view(np.int32).astype(np.int64)
ib = np.float32(b).view(np.int32).astype(np.int64)
lo = -(1 << 31)
ka = np.where(ia < 0, lo - ia, ia)
kb = np.where(ib < 0, lo - ib, ib)
return np.abs(ka - kb)
# pass the original fp32 values, not the upcast
print(ulp_diff(ref[i], got[i]))
If you would rather not hand-roll the ULP step, paste both arrays — a tensor(...) wrapper is fine — or load two .npy files into Perpendis diff and read the first differing index, both bit patterns, and the ULP distance straight off.
Test at batch=1 and at a realistic batch. At batch=1 the runtime may pick a tiling that happens to match PyTorch; at batch=128 it selects a different algorithm, accumulates a longer sum, and shows more rounding. A comparison that passes at batch=1 and fails at batch=128 is usually still rounding — but it is exactly the regime where a real broadcasting or reshape bug also hides, so find the first divergent index before you conclude either way.
These are starting points, not guarantees. The right tolerance depends on network depth, output dynamic range, and hardware. Both np.testing.assert_allclose and torch.testing.assert_close test |actual - reference| <= atol + rtol * |reference|, so rtol scales with the reference magnitude and atol is the floor near zero — you almost always want both. One trap worth naming: numpy.testing.assert_allclose defaults to rtol=1e-7 and atol=0. With atol=0, any element whose reference sits near zero is held to a pure relative bar it can almost never meet, so the assertion fails on legitimate rounding until you set a real atol. (np.allclose is a separate function with different defaults: rtol=1e-5, atol=1e-8.)
| Scenario | atol (start) | rtol (start) | Notes |
|---|---|---|---|
| fp32, CPU vs ONNX Runtime CPU | 1e-5 | 1e-3 | Tightest case; many models pass at 1e-5 to 1e-4. |
| fp32 with TF32 (Ampere+ GPU) | 1e-3 | 1e-2 | Disable TF32 to get a clean fp32-vs-fp32 read. |
| fp16 | 1e-2 | 1e-2 | Abs diffs of 1e-3 to 1e-2 are common; lean on rtol. |
| bf16 | 1e-2 | 2e-2 | Only 7 mantissa bits, so wider still. |
| High-dynamic-range output | near 0 | 1e-3 to 1e-2 | Logits and unnormalised scores need relative, not absolute. |
For reference, torch.testing.assert_close's own dtype-dependent defaults are atol=1e-5, rtol=1.3e-6 for float32 and atol=1e-5, rtol=1e-3 for float16. A high-dynamic-range output is where absolute tolerance breaks: a single atol is either too strict on the large values or too loose on the small ones, so switch to relative and keep atol only as a near-zero floor.
Once you have the worst element localised, the pattern of the error is diagnostic.
| Looks like rounding (expected) | Looks like a bug (investigate) |
|---|---|
| A handful of elements off by 1 to 2 ULP. | A whole contiguous region wrong — a channel, a row, an attention head — while the rest matches. |
| Diffs that scale with each element's magnitude (roughly constant relative error). | A NaN or Inf on one side only. |
| Error that grows smoothly with depth and batch size. | A diff that jumps at one specific op and stays large downstream. |
| Max relative diff near the dtype's unit roundoff times network depth. | Error that does not shrink when you force fp32 everywhere; a constant offset or a sign flip. |
When it looks like a bug, bisect by layer. Register PyTorch forward hooks to capture intermediates, or mark the ONNX graph's internal tensors as outputs, and compare each. NVIDIA Polygraphy automates this: it runs the same model through ONNX Runtime and TensorRT and compares every marked layer.
polygraphy run model.onnx \ --trt --onnxrt \ --atol 1e-3 --rtol 1e-3 \ --onnx-outputs mark all \ --trt-outputs mark all
The first layer whose output exceeds tolerance is where to look. From there it is usually a single op: an attribute silently dropped on export, an opset version that changed a default, a norm epsilon in the wrong place, or an unsupported operator that the converter approximated.
Find where two arrays first diverge. Perpendis diff takes your two arrays — paste them from a REPL with brackets and commas or a tensor(...) wrapper, or load two .npy files — and reports the first differing index, both bit patterns at that index, and the ULP distance between them. Equality is bit equality, not ==, so +0 and -0 are treated as differing, two NaNs with matching bits are not, and infinity against a finite value is never reported as a distance. It works on float32 and float64, runs in the tab with nothing uploaded and no account, and it will not guess which side is correct — that judgement is what this page is about. Open Perpendis diff.
For an fp32 export compared on CPU, start at atol=1e-5 and rtol=1e-3, then tighten if it passes; many models are fine at atol=1e-4. On an Ampere or later GPU with TF32 enabled, or with fused kernels in play, loosen to about atol=1e-3, rtol=1e-2, or disable TF32 to get a clean fp32-versus-fp32 read. Treat these as starting points: the correct value depends on depth and dynamic range, so localise the worst element rather than trusting one global threshold.
Kernel selection and reduction order depend on tensor shape. At batch=1 the runtime may pick a tiling that happens to match PyTorch; at batch=128 it selects a different algorithm and accumulates a longer sum, so more rounding surfaces. If the per-element relative error stays near the dtype's unit roundoff and no single region is wrong, that growth is expected. It is also the regime where a genuine reshape or broadcasting bug hides, so find the first divergent index before deciding.
For fp32 it is loose: torch.testing.assert_close defaults to rtol=1.3e-6 for float32, so 1e-3 will pass differences you might want to see. For fp16 it is about right, since per-op relative error is already around 5e-4. Loose is not wrong in itself; it just has to match the precision you are actually running. Pair rtol with a small atol so near-zero values are not held to an impossible relative bar.
fp16 resolves about three significant decimal digits, so per-op relative error near 1e-3 is normal and accumulates with depth. Output absolute diffs of 1e-3 to 1e-2 against an fp32 reference are common and usually fine. Judge by the task metric, not the raw tensor: if top-1 class, detection boxes, or your downstream loss are unchanged, the model is behaving. Watch for NaN or Inf, or a whole region wrong — those are bugs, not precision.