Three new research papers published on arXiv address critical challenges in managing and evaluating large language model (LLM)-based systems.
According to arxiv.org, researchers have developed AgentLocate, a framework designed to diagnose failures in LLM-based multi-agent systems. The paper, set to appear at COLM 2026, addresses the difficulty of identifying “which agent is responsible and at what point the trajectory first becomes irreversibly misdirected” when distributed AI systems fail. AgentLocate combines an LLM-based judging mechanism with multi-perspective verification by independent evaluators, using a confidence-aware aggregation strategy. The framework then adapts through lightweight fine-tuning to improve attribution quality. Testing on two benchmarks showed AgentLocate “consistently outperforms existing failure localization methods” in identifying both responsible agents and failure steps while remaining efficient in token usage and running time.
In related research, arxiv.org reports on a paradigm shift from “LLM-as-a-Judge” to “Representation-as-a-Judge.” The work proposes that smaller language models can serve as efficient evaluators by leveraging internal representations rather than surface generation, based on what researchers call the “Semantic Capacity Asymmetry Hypothesis”—the principle that “evaluation requires significantly less semantic capacity than generation.” The researchers instantiated this approach through INSPECTOR, a probing-based framework that examines internal model structures instead of relying on prompted outputs.