A 7.7-Point Seed Swing Exposes Low-Resource Fine-Tuning Benchmarks as Noise

The assumption baked into most low-resource language fine-tuning work is that accuracy benchmarks capture whether training worked. They don't. Changing only the random seed moves the score by 7.7 points, more than every data choice and recipe effect measured across three frontier mixture-of-experts models. The benchmark isn't signal. It's variance.

The real changes happen in dimensions accuracy cannot reach. Take language fidelity: base models never reason in Greek. Zero of 1,000 reasoning traces, even when the question is written in Greek, contain Greek-language reasoning. The model answers correctly while thinking in a language its user cannot read, audit, or correct. That's not a minor UX issue. It's a structural transparency failure hiding behind a clean accuracy score.

Supervised fine-tuning fixes that specific failure completely. After SFT, every released checkpoint reasons in the language of the question on approximately 98% of items. One model family does it at three times fewer tokens. Judged grammaticality improves across all four models. General ability stays within a few points of the base. Nothing is forgotten; fluency is gained. To make these changes measurable without being fooled by length-correlated metrics, six behavioral dimensions are proposed, each gated to reject any metric that correlates with output length. Six instrument failures are reported, each caught by a control. The methodology is as much the contribution as the results.

SFT cannot fix everything it breaks, though. A quarter of answers skip the requested format. Answers leak into the reasoning channel. An explicit instruction to "think in English" is followed less than half the time. These are not edge cases. They are systematic defects that SFT introduces and cannot self-correct.

Reinforcement learning with verifiable rewards, pre-registered before training, addresses all three. Format fallback drops from 24% to 2.5%. Reasoning-channel leakage drops from 3.5% to 0.0%. Both results hold against a flat random-reward control, confirming the RL signal is doing real work. Instruction-following on language choice improves by 9.1 percentage points. The Greek reasoning habit survives an accuracy-only gradient untouched, meaning RL is doing something SFT's accuracy gradient cannot. For teams fine-tuning models for low-resource language deployment, the takeaway is direct: if your evaluation stops at accuracy, you are not evaluating the thing that actually changed.

We're thinking: The 7.7-point seed swing is the number that should end careers of benchmark-only evaluation in low-resource NLP. We read this as an indictment of an entire publication genre: papers reporting 2-4 point accuracy gains from recipe changes in low-resource settings are, with high probability, reporting noise. The behavioral measurement framework here, six dimensions with explicit length-correlation controls and pre-registered RL, is more methodologically honest than most published fine-tuning work at any scale. The specific implication for practitioners is uncomfortable: if your team has shipped a low-resource fine-tuning result justified by accuracy numbers, you may not know whether your training recipe did anything at all.

Key takeaways:

• SFT reliably instills target-language reasoning (0% to ~98% fidelity) and improves grammaticality without forgetting general ability, but introduces format and leakage defects it cannot self-correct; RL with verifiable rewards closes those gaps where accuracy-only gradients fail entirely.
• Seed variance alone produces a 7.7-point accuracy swing across three frontier MoE models (3.6-4.0B active parameters), exceeding every measured training effect; the six behavioral dimensions with length-correlation controls and pre-registered RL are the replicable instruments, not the Greek-specific results.
• Teams fine-tuning models for low-resource language deployment should treat accuracy benchmarks as a sanity check only, adopt behavioral dimensions gating out length-correlated metrics, and use pre-registered RL with verifiable rewards to fix format compliance and reasoning-channel leakage that SFT cannot resolve.

Source: Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot See