Researchers have introduced MetroLLM-Bench, a 955-case benchmark designed to evaluate language models as the policy layer of transit kiosks, according to a paper published on arxiv.org. The benchmark covers six real metro systems ranging from 37 to 414 stations and tests eleven categories including routing, fare calculation, disruptions, accessibility, and adversarial input.
According to the paper, the benchmark uses fourteen deterministic scoring components for Tier 1 evaluation and eight semantic-quality components for Tier 2, six of which employ a language-model judge. The dataset is split 75/25, with 717 cases for training and 238 for held-out evaluation.
In testing twenty-six models from six vendors, the researchers found that a 4B Qwen 3.5 student model trained through parameter-efficient fine-tuning (PEFT) achieved 91.3 on Tier 1, exceeding both GPT-5.6 tiers (90.6 and 90.0) with only a 2.6 GB footprint. According to the findings, larger 9B and 27B students provided no additional Tier 1 improvement over the 4B student at this training scale.
The paper reports that PEFT gains over base models decreased from +7.03 points at 2B to -0.91 at 27B. A deterministic rule-based baseline reached 84.6 on Tier 1, with language models showing advantages in “policy adaptation, compound scenarios, accessibility, and temporal reasoning,” according to arxiv.org. The benchmark, harness, and fine-tuned students are publicly available on GitHub.