X-GIS
Blog

Tag

precision

13 posts

precision

5 min read

20.7 km and 0.7 m: a difference forgives the bias its terms share

I spec'd a drift guard for the globe's camera-relative ECEF offset: force the ellipsoid constant to a sphere, assert the result shifts >10 km. The implementer measured it: 0.7 m. Turning the WHOLE computation spherical barely moves a relative offset, because both endpoints carry the same 21 km bias and a subtraction cancels it. The regression that actually bites is the mixed frame — ellipsoid tile, spherical camera — and only a guard that breaks the symmetry the same way sees it.

precision

6 min read

The precision fix that opened a seam

Making the line outline's projection precise, but not its polygon fill, turned a shared invisible jitter into a ~3px fill≠outline seam at deep zoom. The fix isn't more f64 — it's the same camera-relative reframe, which the linear term needs no emulation for.

precision

3 min read

Reconstructing the absolute coordinate was the bug

Line strokes shook at deep zoom in non-Mercator projections: the shader rebuilt an absolute longitude in f32 degrees, then a later stage subtracted the camera again — catastrophic cancellation. The fix subtracts first; the proof needs no GPU.

precision

8 min read

The error term you compute in integers

Fast-math deleted our emulated-double multiply on Apple GPUs, and every float-side guard — including hardware FMA — died with it. So we rebuilt twoSum and twoProd in u32 bit arithmetic: same pair format, same compositions, exact by construction. The device that zeroed every float variant returned the golden answer.

precision

6 min read

Extended precision on the GPU: a field guide

The GPU has no double. There are four ways to get one anyway — double-float error-free transforms, integer emulation, reference-point rendering, and perturbation — and they are not interchangeable. A decision guide, from a survey of Thall, QD/CAMPARY, Cesium, deck.gl, and fractal deep-zoom engines.

precision

6 min read

The multiply you cannot guard

I shipped four in-shader fixes for the df64 multiply collapsing to zero on an Apple GPU. Every one failed on-device. Then the research settled it: Metal's default fast-math deletes the error term by construction, no opaque-guard trick reliably survives, and the durable answer is to restructure the math so the high-precision multiply never happens — which is why deck.gl deprecated emulated fp64.

precision

9 min read

The low word a load throws away

The same df64 value—1e8+0.5—minus 1e8. Computed on the GPU it reads 0.5; loaded from a uniform it reads 0.0, with the fast-math guard bound in both. To a reassociating compiler a loaded low word and a computed one are not the same number, and the fix is to launder one into the other.

floating-point

7 min read

The split constant was never the bug

GPU double-single splits floats with 8193; the textbook says 4097. I was sure 8193 was cargo-culted from a wider type — until two million float32 products came back error-free for both. The split constant that breaks is one too small; the real bug was never the split.

shader-dsl

9 min read

An opaque guard defeats the compiler, not the hardware

After a texel-fetched guard fixed the fast-math collapse, isolated df64 ops passed on the reporter's phone but composed ones still broke. There are two failure classes, not one — and the second lives in the ALU, where no value you thread into the shader can reach it. df64 is per-vendor, and our CI's SwiftShader is structurally blind to half of it.

shader-dsl

5 min read

Testing emulated doubles

The verification stack behind shader-dsl's fp64: a correctly-rounding f32 machine built from Math.fround, a metamorphic oracle gate, byte goldens, and discriminative real-GPU known answers.