New Benchmarks Expose Limitations in AI Vision and Reasoning Capabilities

New Benchmarks Expose Limitations in AI Vision and Reasoning Capabilities

Researchers have introduced ActiveVision, a benchmark designed to test whether multimodal large language models (MLLMs) can perform active visual observation—the continuous redirection of attention based on intermediate hypotheses, according to a paper published on arxiv.org. The benchmark comprises 17 tasks across 3 categories that require repeated visual perception rather than static description.

According to the research, frontier MLLMs performed poorly on ActiveVision. GPT-5.5 at its highest reasoning-effort tier solved only 10.6% of items and scored zero on 11 of the 17 tasks. Claude Fable 5 solved just 3.5% of items, despite topping most reasoning and coding leaderboards. In contrast, three human participants averaged 96.1% accuracy. The paper states that “much of the gap persists even when models write and run their own vision code,” indicating that current MLLMs lack robust active visual observation.

In separate research on arxiv.org examining reasoning across the pretraining-to-post-training pipeline, researchers used chess as a controlled testbed. They found that post-reinforcement learning (RL) performance “is well-predicted from the pretraining loss,” and that the slope of RL reward curves “improves approximately linearly with the pretraining tokens.” The study also revealed that RL amplifies correct moves on easy puzzles but surfaces nearly absent correct moves on hard puzzles.

Additionally, arxiv.org published research on Loopie, a looped Transformer model with two Mixture-of-Experts variants (20B parameters with 2B active, and 6B parameters with 0.6B active). According to the paper, Loopie achieved gold-medal performance at the 2025 IMO and IPhO without tools.