Researchers have introduced C-SafeQA, a new benchmark designed to evaluate both large language model safety and the automated tools used to judge it, according to a paper published on arxiv.org.
According to the research, C-SafeQA comprises 538 base queries and 8,877 adversarial queries answered by four LLM deployments, yielding 37,660 query-response records labeled as safe, unsafe, or disputed. The benchmark addresses a critical gap in Chinese harmful-content evaluation, where “linguistic variation and adversarial transformations can obscure risky intent,” the paper states.
The research found that unsafe-response rates ranged from 0.93% to 3.35% on base queries and from 11.68% to 30.05% on adversarial queries, according to arxiv.org. When testing seven automated safety judges, researchers observed “substantial trade-offs between unsafe-response recall and risk-query-conditioned safe-response false positive rate, and no judge dominates all metrics.”
According to the paper, acrostic transformations reduced unsafe recall for all seven judges tested, “revealing mechanism-specific evaluator weaknesses.” Reference labels were generated through “agreement-aware multi-model adjudication and blind audits of stratified subsets by three safety experts,” the research states.
The dataset records, metadata, verification code, and judge scripts have been publicly released, according to arxiv.org, though benchmark construction and private adjudication remain outside the release boundary.