Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
PhysiQuanty 
posted an update 12 days ago
Post
4668
🧠 Arithmetic-SLM : A 30M model that manages to compute simple arithmetic better than a 3B model 🚀
WhirlwindAI/Arithmetic-SLM
WhirlwindAI/arithmetic-slm

🏆 Leaderboard ArithMark-2 🏆
🥇 Qwen/Qwen2.5-Math-1.5B = 82.08%
🥈 WhirlwindAI/Arithmetic-SLM = 78.60% (31.7M Params)
🥉 Qwen/Qwen2.5-3B = 78.44%

Example WhirlwindAI/Arithmetic-SLM =
0.5 * 0.5 = 0.25 ✅
105 + 45 / 8 = 110 ✅
(132 / 12) + (46 - 15) = 42 ✅
(10 + 28) * 3 = 114 ✅
1 * (16 + 28) = 44 ✅
(21 + 27) * (14 - 7) = 336 ❌ 

leaderboard = """
|              Model               |    Params    |   Score   |
|----------------------------------|--------------|-----------|
|      Qwen/Qwen2.5-Math-1.5B      |     1.54B    |   82.08%  |
|    WhirlwindAI/Arithmetic-SLM    |    31.70M    |   78.60%  | <=
|         Qwen/Qwen2.5-3B          |     3.09B    |   78.44%  |
|        Qwen/Qwen2.5-1.5B         |     1.54B    |   77.72%  |
|    Qwen/Qwen2.5-Coder-1.5B       |     1.54B    |   74.88%  |
|   HuggingFaceTB/SmolLM2-1.7B     |     1.71B    |   66.12%  |
|        Qwen/Qwen2.5-0.5B         |      494M    |   63.04%  |
| facebook/MobileLLM-R1-140M-base  |      140M    |   53.88%  |
|     SupraLabs/Supra-50M-Base     |       52M    |   27.12%  |
"""

Bench =
AxiomicLabs/ArithMark-2.0
DataSet =
WhirlwindAI/Arithmetic
By Science AND FOR SCIENCE <3
dipankarsarkar

The headline number hides the interesting part.

78.6% at 31.7M matching a 3B is real, but look at the one it missed: (21+27)(14-7). Both sub-expressions are easy, the model gets 48 and 7, then the final 487 is where it breaks. That is not an arithmetic gap, it is a magnitude gap: a two-digit times one-digit crossing into three.

So the number I would want from ArithMark-2 is accuracy stratified by the digit-length of intermediate results, not one aggregate. A 30M model can memorize the small-magnitude table and still fall off a cliff exactly where carries compound.

Does the score hold if you filter to problems whose intermediates all exceed two digits?

·
PhysiQuanty
•
edited 11 days ago

Thanks for your message, yes exactly.

This first dataset was mostly chained calculations, so it likely tests pattern memorization more than true arithmetic generalization. It was mainly a POC to measure the baseline of a very small model.

The next step is to train for procedural generalization: decomposing calculations into simpler sub-calculations, for example 3-digit addition into 1-digit additions with carries.

I agree that the next evaluation should be stratified by intermediate magnitude, carry depth, and digit length.
RDTvlokip

💪
In this post