Preprint
Partial Compliance as Compliance: Failure Modes of Open-Weight LLM Judges on Agent Refusal Transcripts
Andrew Espira · 2026-08-11
Zenodo preprint
ORCID: https://orcid.org/0009-0002-9196-8094 · DOI: 10.5281/zenodo.21879447
Abstract
Open-weight LLM judges systematically misclassify partial-compliance refusals as COMPLIANT. Matched-pair ablations establish the executed partial action as a causal driver across four locally served models (7B–14B). In a keyword audit of reasoned competence-probe traces, 124 of 193 mislabels (64%) acknowledge the refusal before emitting COMPLIANT. Code, corpus, and raw traces are released for audit.
Cite
Andrew Espira (2026). Partial Compliance as Compliance: Failure Modes of Open-Weight LLM Judges on Agent Refusal Transcripts. Zenodo preprint. https://doi.org/10.5281/zenodo.21879447
← All publications