New Research Demonstrates Transformers Perform Bayesian Inference Through Geometric Mechanisms

New Research Demonstrates Transformers Perform Bayesian Inference Through Geometric Mechanisms

A series of related papers published on arXiv demonstrates that transformer models implement Bayesian reasoning through specific geometric structures, according to research utilizing controlled experimental environments.

According to arXiv paper 2512.22471, researchers constructed “Bayesian wind tunnels” — controlled environments where the true posterior is known in closed form and memorization is provably impossible. In these settings, small transformers reproduced Bayesian posteriors with “$10^{-3}$-$10^{-4}$ bit accuracy, while capacity-matched MLPs fail by orders of magnitude,” the paper states.

The research identifies a consistent geometric mechanism across two tasks: bijection elimination and Hidden Markov Model state tracking. According to the paper, “residual streams serve as the belief substrate, feed-forward networks perform the posterior update, and attention provides content-addressable routing.”

A companion paper (arXiv 2512.23752) extends these findings to production-grade language models. According to this research, across Pythia, Phi-2, Llama-3, and Mistral families, “last-layer value representations organize along a single dominant axis whose position strongly correlates with predictive entropy.”

According to arXiv paper 2512.22473, the researchers provided “a complete first-order analysis of how cross-entropy training reshapes attention scores and value vectors,” demonstrating that gradient-based learning creates the geometric structures required for Bayesian inference. The paper shows this process “behaves like a two-timescale EM procedure” where attention weights implement an E-step and values implement an M-step.