According to arxiv.org, researchers have introduced MetroLLM-Bench, a 955-case benchmark designed to evaluate language models as the policy layer for transit kiosks. The benchmark covers six real metro systems ranging from 37 to 414 stations and tests models across eleven categories including routing, fare calculation, disruptions, accessibility, and adversarial input.
The benchmark uses fourteen deterministic scoring components in Tier 1 and eight semantic-quality components in Tier 2, six of which employ a language-model judge. According to the researchers, the benchmark uses a stratified 75/25 split with 717 cases for training and 238 for evaluation.
Twenty-six models from six vendors were evaluated, with twenty-three ranked. According to the study, a 4B Qwen 3.5 student trained through parameter-efficient fine-tuning (PEFT) achieved 91.3 on Tier 1, exceeding both GPT-5.6 tiers (90.6 and 90.0) and matching GPT-5.4 at maximum reasoning effort (91.4), with a 2.6 GB Q4_K_M footprint. Larger 9B and 27B students provided no further Tier 1 improvement over the 4B student. A deterministic rule-based baseline reached 84.6 on Tier 1, with the remaining advantage concentrated in policy adaptation, compound scenarios, accessibility, and temporal reasoning.
The benchmark, harness, reproduction guide, and fine-tuned students have been released on GitHub, according to arxiv.org.