Several new research papers published on arXiv examine approaches to understanding and explaining large language model (LLM) behavior, addressing challenges in transparency and trust.
According to one arxiv.org paper published April 21, 2026, researchers conducted a comparative study of three explainability techniques—Integrated Gradients, Attention Rollout, and SHAP—applied to a fine-tuned DistilBERT model for sentiment classification. The 14-page study found that “gradient-based attribution provides more stable and intuitive explanations, while attention-based methods are computationally efficient but less aligned with prediction-relevant features,” according to the paper’s abstract. The researchers emphasized that these methods serve as “diagnostic tools rather than definitive explanations.”
A separate arxiv.org paper, accepted to ACL 2026, takes a different approach by reviewing “intrinsic interpretability” methods that build transparency directly into model architectures. According to this paper, the researchers categorized existing approaches into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction.
Another arxiv.org study explores using knowledge graphs combined with LLMs to improve explainability of machine learning results in manufacturing environments. According to the paper, researchers stored domain-specific data alongside ML results in a knowledge graph, then used an LLM to generate user-friendly explanations by extracting relevant information.
All papers note that LLM opacity remains a significant challenge for deployment in real-world systems.