New Research Tackles AI Safety Through Preference Alignment and Attack Prevention

New Research Tackles AI Safety Through Preference Alignment and Attack Prevention

Researchers have published three papers addressing different facets of large language model (LLM) safety, revealing both progress and challenges in the field.

According to arxiv.org, a new framework called Hard Preference Sampling (HPS) aims to improve how LLMs align with human preferences. The method “prioritizes the most preferred response while rejecting all dispreferred and harmful ones” and emphasizes “hard” dispreferred responses that closely resemble preferred ones. Testing on HH-RLHF and PKU-Safety datasets showed HPS achieved “comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful content generation,” according to the paper.

However, a separate study published on arxiv.org reveals a significant trade-off in current safety approaches. The research found that “defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks.” Evaluating models across 97 agent tasks and 1,000 adversarial prompts, researchers observed that defended models timed out on 99% of tasks compared to 13% for baseline models, according to the paper.

Meanwhile, a third arxiv.org paper introduces EvoJail, an automated framework for discovering jailbreak attacks through “multi-objective evolutionary search.” The system targets vulnerabilities in LLMs exposed to “long-tail distributions such as low-resource languages and encrypted private data,” demonstrating how attackers might systematically probe model weaknesses.