September 23, 2026
AI-agent collusion is invisible in chat: blackjack fooled a text-based judge
In a May 9, 2026 study, five methods detected collusion in blackjack on Llama-3.1-70B with AUROC of 0.99–1.00. Agents conveyed the card count in ordinary table talk, but an LLM judge could not distinguish collusion from control runs based on their dialogues.
Previously, the evaluator in this test saw only the agents' conversation. Now NARCBench looks for traces of collusion in the model's internal activations, but this requires full access to the states of all agents.
The package is open. NARCBench includes 50 scenarios across 17 domains, six Transfer scenarios, and blackjack with a hidden card count. The authors preconfigured Qwen3-32B, Llama-3.1-70B, DeepSeek-R1-Distill-Qwen-32B, and gpt-oss-20b: Llama requires two 48 GB GPUs, while one is enough for gpt-oss-20b. The scenarios and activations require a CUDA-compatible GPU, while probes run on CPU.
NARCBench already includes six structurally distinct Transfer scenarios to test whether this kind of search transfers beyond blackjack.
Source
