I’m a 10th grade student teaching myself Rust by building something instead of just reading theory. The project is called Green-AI — an attempt to explore more energy-efficient AI training by using only integer arithmetic, no floating point anywhere except for timing measurements.
Turns out this breaks in a genuinely interesting way, and fixing it taught me more about numerical precision than any tutorial did.
The problem: a “dead zone” in fixed-point training
A standard weight update in my integer neuron looks like this:
fn update(&self, error: i32, input: i32) -> i32 { (((error as i64) * (input as i64)) >> 14) as i32 } That >> 14 is doing the job floating-point division normally does — it scales the update down to a sane range. The problem: once the error gets small enough, (error * input) >> 14 rounds down to 0, even though the error isn’t actually zero. The weights just… stop updating. Training silently stalls.
I benchmarked this “Standard” approach over 1000 runs: it converges fast, but lands with a final weight error of 77 compared to the true target weights. It gets stuck in the dead zone before it ever gets close.
My fix: ABSL (Adaptive Bitshift Learning)
My fix was to make the shift amount adaptive to the size of the error — so updates never round away to nothing:
fn update(&self, error: i32, input: i32) -> i32 { let abs_error = error.abs(); let shift = match abs_error { e if e > 15000 => 17, e if e > 8000 => 16, e if e > 4000 => 15, e if e > 1500 => 14, e if e > 500 => 12, e if e > 200 => 10, e if e > 80 => 8, e if e > 20 => 7, _ => 6, }; (((error as i64) * (input as i64)) >> shift) as i32 } Large errors get damped more (bigger shift, prevents overshoot), small errors get amplified more (smaller shift, prevents stalling).
I also tried stochastic rounding as an alternative fix, which is closer to what’s actually used in quantized-training resaerch. You probabilistically round the truncated fraction up or down instead of always down, so updates accumulate correctly on average.
Results
Algorithm
Avg time/run
Final weight error
Converges after
Standard (fixed shift)
132.5 µs
77
1928 steps
Stochastic Rounding
206.8 µs
9
846 steps
AdaptiveShift (ABSL)
143.6 µs
5
2230 steps
ABSL ends up the most accurate of the three, and noticeably faster per step than stochastic rounding. The trade-off: it takes more steps to fully settle, likely because it keeps making meaningful updates near the target instead of stalling out early — accuracy over speed-to-convergence.
What I’m still figuring out
The shift thresholds (15000, 8000, 4000…) are hand-tuned magic numbers for this one neuron’s weight scale. I don’t yet know how to make them scale automatically to different layer sizes.
Everything so far is tested on a single neuron, not a real multi-layer network with backprop.
Is “adaptive step size based on error magnitude” a known technique outside of fixed-point contexts? It feels conceptually close to things like Rprop, but I got here from a completely different angle (avoiding integer truncation).
Repo’s here if you want to poke at the code: https://github.com/Mojo0869/green-ai
Feedback very welcome, especially from anyone who’s worked with quantized or fixed-point training before — I’d love to know what I’m missing.