{"id":3931,"date":"2026-07-28T03:42:07","date_gmt":"2026-07-28T03:42:07","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/07\/28\/i-taught-myself-integer-only-neural-net-training-in-rust-and-ran-into-a-real-quantization-problem\/"},"modified":"2026-07-28T03:42:07","modified_gmt":"2026-07-28T03:42:07","slug":"i-taught-myself-integer-only-neural-net-training-in-rust-and-ran-into-a-real-quantization-problem","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/07\/28\/i-taught-myself-integer-only-neural-net-training-in-rust-and-ran-into-a-real-quantization-problem\/","title":{"rendered":"I taught myself integer-only neural net training in Rust \u2014 and ran into a real quantization problem"},"content":{"rendered":"
\n
\n
\n

I’m a 10th grade student teaching myself Rust by building something instead of just reading theory. The project is called Green-AI \u2014 an attempt to explore more energy-efficient AI training by using only integer arithmetic, no floating point anywhere except for timing measurements.
Turns out this breaks in a genuinely interesting way, and fixing it taught me more about numerical precision than any tutorial did.<\/p>\n

The problem: a “dead zone” in fixed-point training<\/strong><\/p>\n

A standard weight update in my integer neuron looks like this: <\/p>\n

\n
fn<\/span> update<\/span>(<\/span>&<\/span>self<\/span>,<\/span> error<\/span>:<\/span> i32<\/span>,<\/span> input<\/span>:<\/span> i32<\/span>)<\/span> -><\/span> i32<\/span> {<\/span> (((<\/span>error<\/span> as<\/span> i64<\/span>)<\/span> *<\/span> (<\/span>input<\/span> as<\/span> i64<\/span>))<\/span> >><\/span> 14<\/span>)<\/span> as<\/span> i32<\/span> }<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
That >> 14 is doing the job floating-point division normally does \u2014 it scales the update down to a sane range. The problem: once the error gets small enough, (error * input) >> 14 rounds down to 0, even though the error isn’t actually zero. The weights just… stop updating. Training silently stalls.
 I benchmarked this “Standard” approach over 1000 runs: it converges fast, but lands with a final weight error of 77 compared to the true target weights. It gets stuck in the dead zone before it ever gets close.<\/p>\n
My fix: ABSL (Adaptive Bitshift Learning)<\/strong><\/p>\n
My fix was to make the shift amount adaptive to the size of the error \u2014 so updates never round away to nothing: <\/p>\n
\n
fn<\/span> update<\/span>(<\/span>&<\/span>self<\/span>,<\/span> error<\/span>:<\/span> i32<\/span>,<\/span> input<\/span>:<\/span> i32<\/span>)<\/span> -><\/span> i32<\/span> {<\/span> let<\/span> abs_error<\/span> =<\/span> error<\/span>.abs<\/span>();<\/span> let<\/span> shift<\/span> =<\/span> match<\/span> abs_error<\/span> {<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 15000<\/span> =><\/span> 17<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 8000<\/span> =><\/span> 16<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 4000<\/span> =><\/span> 15<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 1500<\/span> =><\/span> 14<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 500<\/span> =><\/span> 12<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 200<\/span> =><\/span> 10<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 80<\/span> =><\/span> 8<\/span>,<\/span> e<\/span> if<\/span> e<\/span> ><\/span> 20<\/span> =><\/span> 7<\/span>,<\/span> _<\/span> =><\/span> 6<\/span>,<\/span> };<\/span> (((<\/span>error<\/span> as<\/span> i64<\/span>)<\/span> *<\/span> (<\/span>input<\/span> as<\/span> i64<\/span>))<\/span> >><\/span> shift<\/span>)<\/span> as<\/span> i32<\/span> }<\/span> <\/code><\/pre>\n
\n<\/p><\/div>\n<\/p><\/div>\n
Large errors get damped more (bigger shift, prevents overshoot), small errors get amplified more (smaller shift, prevents stalling).
 I also tried stochastic rounding as an alternative fix, which is closer to what’s actually used in quantized-training resaerch. You probabilistically round the truncated fraction up or down instead of always down, so updates accumulate correctly on average.<\/p>\n
Results<\/strong>
 Algorithm 
 Avg time\/run 
 Final weight error
 Converges after<\/p>\n
Standard (fixed shift)
 132.5 \u00b5s
 77
 1928 steps<\/p>\n
Stochastic Rounding
 206.8 \u00b5s
 9
 846 steps<\/p>\n
AdaptiveShift (ABSL)
 143.6 \u00b5s
 5
 2230 steps<\/p>\n
ABSL ends up the most accurate of the three, and noticeably faster per step than stochastic rounding. The trade-off: it takes more steps to fully settle, likely because it keeps making meaningful updates near the target instead of stalling out early \u2014 accuracy over speed-to-convergence.<\/p>\n
What I’m still figuring out<\/strong>
 The shift thresholds (15000, 8000, 4000…) are hand-tuned magic numbers for this one neuron’s weight scale. I don’t yet know how to make them scale automatically to different layer sizes.
 Everything so far is tested on a single neuron, not a real multi-layer network with backprop.
 Is “adaptive step size based on error magnitude” a known technique outside of fixed-point contexts? It feels conceptually close to things like Rprop, but I got here from a completely different angle (avoiding integer truncation).<\/p>\n
Repo’s here if you want to poke at the code: https:\/\/github.com\/Mojo0869\/green-ai<\/a><\/p>\n
Feedback very welcome, especially from anyone who’s worked with quantized or fixed-point training before \u2014 I’d love to know what I’m missing.<\/p>\n<\/p><\/div>\n<\/div>\n<\/div>\n<\/div>\n
Fuente: Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
I’m a 10th grade student teaching myself Rust by building something instead of just reading theory. The project is called Green-AI \u2014 an attempt to explore more energy-efficient AI training by using only integer arithmetic, no floating point anywhere except for timing measurements. Turns out this breaks in a genuinely interesting way, and fixing it […]<\/p>\n","protected":false},"author":1,"featured_media":3930,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[41],"tags":[],"class_list":["post-3931","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/3931","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=3931"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/3931\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/3930"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=3931"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=3931"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=3931"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}