Small Language Models Gain Ground Through Specialized Training
Researchers have developed targeted fine-tuning approaches to improve small language models’ performance on specialized tasks, according to multiple papers published on arxiv.org.
In work focused on the constraint modeling language MiniZinc, researchers found that large language models “struggle with rare, domain-specific languages” despite excelling at mainstream programming languages, according to arxiv.org. The study investigated whether small language models (0.6B to 20B parameters) could be taught to generate syntactically correct MiniZinc models from natural language descriptions.
According to the research, “syntax errors dominate failures” when working with this domain-specific language, with “out-of-the-box execution accuracy of small language models such as Qwen3, LLaMa, Gemma, and GPT-OSS” being “near-zero.” The researchers proposed a “cross-model error bootstrapping approach” that collects syntax errors from multiple LLM runs to create an error correction training dataset. This approach achieved “up to 98% execution accuracy” with self-reflection and ensembling, though “solution accuracy still remains at 35%,” indicating that “while syntax is learnable, constraint reasoning remains a challenge,” according to arxiv.org.
Separately, researchers developed MiniCache, a framework that “transforms Program-of-Thought (PoT) programs into parameterized cache objects” to reduce inference costs, according to arxiv.org. The system achieved “up to 3.1x lower latency and 2.8x higher throughput under parallel serving” in experiments.