most labs won't bother teaching a small model to write and think in real romanian. no invented words, no english leaking in mid-sentence, diacritics intact.
we did.
surogate 3.5 is out. 2B and 4B, apache 2.0. invented word-forms cut from 4 per 1k down to ~1.5, and it reasons in the language you prompt it in, every time.
I recently released FlameF0X/TinyMoE-100m-2x8-retrained, a small Mixture of Experts language model trained on the Smollm-Corpus. Built on top of the Mixtral architecture, itβs fully compatible with π€ Transformers right out of the box!
The model can produce somewhat coherent text on its own, and for some reason, it generates even more coherent responses when given a ChatLM template.
Iβm excited to see what you all come up with, and feel free to fine-tune it if youβd like. In the meantime, Iβll be working on developing the chat-trained version.