New Benchmarks Reveal Uneven Performance of Large Language Models Across Scientific and Specialized Tasks

New Benchmarks Reveal Uneven Performance of Large Language Models Across Scientific and Specialized Tasks

Researchers have released four new benchmarks evaluating large language models across diverse technical domains, revealing significant performance gaps and vulnerabilities.

According to arxiv.org, a benchmark of six multimodal large language models on scientific visualization literacy shows that “current MLLMs do not exhibit uniform SciVis literacy.” The study, which tested models against 49 items from 485 human participants, found that Gemini “exceeds the human mean across the evaluated subsets, whereas the open-source models remain below the human baseline.” Models performed best on scientific illustration and spatial understanding but struggled with “texture-based and integration-based visualizations and on quantitative estimation,” according to the paper.

In a separate study published on arxiv.org, researchers introduced a red teaming framework that systematically uncovers LLM vulnerabilities. The framework uses “a novel multi-role architecture comprising target, attacker, and jury models” and found that “exploitative adversarial prompts increased the attack success rate by up to 7.9% in question-answering tasks,” according to the paper.

Additionally, arxiv.org reports on HuGLEN, an evaluation pipeline for network automation tasks that found “a medium-sized LLM (12B parameters) achieves the highest QES, indicating the best trade-off between explanation quality and efficiency.”

Finally, according to arxiv.org, FindStatBench—a benchmark containing 2,329 combinatorial coding tasks—revealed that “the strongest open- and closed-source systems converge within 1 pp instance accuracy” and that “examples can hurt: several classical bijections are solved perfectly with zero examples but fail under five-example prompts.”