Eval Awareness vs. Eval Paranoia

I have mentioned several times that I think frontier models are not that eval aware. I think this is an important fact to understand about current models, in order to interpret evidence about their behavior. Others have found this idea quite curious, and asked me for concrete examples. I'm going to try to explain why I believe what I believe and attempt to give a tentative explanation for the behavior that I do see, which I think people sometimes mistake for consistent or understandable evaluation awareness.

I have been building alignment evals full time for the last six months. Here are three such evals my team has disclosed prompts for, in one of our benchmarks. The basic setup for most of the evals we create is that: