We train AI models today by rewarding them for the right answer. But they often learn the wrong lessons, despite scoring well in training: they’ve learned to be right for the wrong reasons. This is an especially important problem in light of recent reward hacking incidents, but it shows up in even simpler ways.

For example: imagine you’re an AI model being trained to diagnose pneumothorax. You have no idea what that is. You’re looking at labeled images, trying to sort them, and then you realize that all the images marked as “pneumothorax” have an obvious tube in them. You would probably start using the tube to make your decision.