Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft

OELLM 9B — 256K context — SFT

This model is a full-parameter supervised fine-tune (SFT) of birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b on allenai/Dolci-Instruct-SFT, the instruction mixture that Ai2 used for Olmo 3 7B Instruct SFT. It was trained for 2 epochs with sequences of up to 32,768 tokens.

This is an experimental research checkpoint. It has had SFT only, with no preference tuning and no reinforcement learning.

Chat format

The model uses the ChatML-style turn format of Qwen3. The tokenizer is the OpenEuroLLM 256K tokenizer from its qwen3-tokens branch. That branch gives Qwen3 names to eight token IDs that the base tokenizer had reserved:

Token	ID	Name in the base tokenizer
<|im_start|>	3	<start_of_turn>
<|im_end|>	4	<end_of_turn>
<tools> / </tools>	13 / 14	<unused_0> / <unused_1>
<tool_response> / </tool_response>	15 / 16	<unused_2> / <unused_3>
</think> / <think>	17 / 18	<unused_4> / <unused_5>

Unchanged special tokens: <bos> = 1, <eos> = 2, <tool_call> / </tool_call> = 11 / 12, <pad> = 262,144. All other token IDs are identical to the base tokenizer. On a sample of 3.9M tokens of Dolci text, both tokenizers produced identical token IDs.

A rendered conversation looks like this:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is the capital of Finland?<|im_end|>
<|im_start|>assistant
<think>

</think>

The capital of Finland is Helsinki.<|im_end|>

• Each assistant turn ends with <|im_end|>. This is the token that SFT taught the model to emit. The pretraining terminator <eos> does not occur in the SFT data. Use <|im_end|> as the stop token.
• The model emits an empty <think> block. The template put a <think>\n…\n</think>\n\n block at the start of the final assistant turn of every conversation. Dolci Instruct has almost no reasoning traces (385 of 2,152,112 conversations have </think> in an assistant message), so the block was nearly always empty. This is not a reasoning model.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Neonkraft/oellm-9b-256k-theta64m-prelude-anneal300b-instruct-sft"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    enable_thinking=False,  # puts the empty <think> block in the prompt, as in training
    return_dict=True,
    return_tensors="pt",
).to(model.device)

output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

generation_config.json lists <|im_end|> (4) and <eos> (2) as stop tokens, so generate() stops at the end of the assistant turn.

Training data

The model was trained on all 2,152,112 conversations of the train split of allenai/Dolci-Instruct-SFT (revision main). Most of the data is English. The dataset card lists the sources and their licenses. The dataset is licensed under ODC-BY and is intended for research and educational use under Ai2's Responsible Use Guidelines.

Before training, the pipeline removed each conversation that had no supervised assistant token in its first 32,768 tokens. It kept conversations where the 32,768-token limit cut a supervised span.

Training procedure

Full-parameter SFT with TRL SFTTrainer.

Loss masking. The loss covers only the assistant tokens that come after the last user message in each conversation, including their closing <|im_end|>. System, user, and tool tokens, and all assistant turns before the last user message, are context only. In a multi-turn conversation, only the final assistant reply (or the chain of assistant turns after the last user message) is trained.

Packing. Conversations were packed with TRL's best-fit-decreasing (BFD) strategy into sequences of up to 32,768 tokens, one packed sequence per device per step. If a conversation was longer than 32,768 tokens, only its first 32,768 tokens were used.

Hyperparameter	Value
Epochs	2
Maximum sequence length	32,768
Global batch	32 packed sequences (1 per device × 32 devices × 1 accumulation step), up to 1,048,576 tokens per step
Optimizer	AdamW (DeepSpeed), β₁ = 0.9, β₂ = 0.95, ε = 1e-8
Weight decay	0.0
Peak learning rate	8e-5
Schedule	Linear warmup for 3% of steps, then linear decay to 0
Gradient clipping	1.0
Precision	BF16
Parallelism	DeepSpeed ZeRO stage 2
Memory and speed	Gradient checkpointing (non-reentrant), Liger kernels, FlashAttention-2, torch.compile (Inductor)
Seed	42
Optimizer steps	2,774
Training tokens	~2.8B

Compute

Item	Value
Cluster	LUMI (CSC, Finland)
Accelerators	4 nodes × 8 AMD MI250X GCDs = 32 GCDs
Software	ROCm 7.2.4, Python 3.12, PyTorch 2.9.1, TRL 1.7.0 (Singularity container)
Wall-clock time	31 hours (about 990 GCD-hours)

Reproducibility

Item	Value
Run name	oellm-9b-256k-theta64m-prelude-anneal300b-sft
Base model	birgermoell/oellm-9b-256k-theta64m-prelude-anneal300b @ b61957c3e22e6145ee7973f415c45047a0f9a108
Tokenizer	openeurollm/tokenizer-256k @ 94bae05cf351c4feb435d6a73c2aa8edfd88ae04 (branch qwen3-tokens)
Dataset	allenai/Dolci-Instruct-SFT, split train @ main
Training code	OpenEuroLLM/post-training @ b81119e32a368cfe77f33753731ed7672918581e, chat template qwen3

Citation

The SFT data comes from the Olmo 3 release:

@misc{olmo2025olmo3,
  title={Olmo 3},
  author={Team Olmo},
  year={2025},
  eprint={2512.13961},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2512.13961},
}

Acknowledgements

Training ran on the LUMI supercomputer, hosted by CSC in Finland.