According to arxiv.org, a new study investigates hallucinations in domain-adapted Large Language Models, specifically examining the fine-tuning of the Llama-2 model with the Lamini dataset. The research found that while the model shows proficiency in tasks similar to its training data, “its capability to accurately reason about and recall new domain-specific information remains limited, leading to instances of hallucination,” according to the paper. The model demonstrated “a tendency to provide correct answers with extra information, suggesting an inclination toward over-generation.”
In a separate development, arxiv.org reports on a novel framework for LLM optimization that jointly addresses structural pruning and mixed-precision quantization. According to the research, at ultra-low precisions (1-3 bits), the proposed quantization method reduces WikiText perplexity by up to 21% compared to state-of-the-art weight-activation quantization baselines. Against weight-only quantization methods, it achieves up to 59% and 85% lower perplexity on WikiText and C4, respectively.
Additionally, arxiv.org introduces RTL-BenchLS, a large-scale benchmark for hardware design containing over 10,000 formally verified Verilog designs. According to the paper, even the best model reaches only 23% on natural-language round-trip reasoning, 28% on masked-content reasoning, and 12% on repository-issue fixing, demonstrating substantial room for improvement in LLM-based hardware design methods.