We wanted to test whether LLMs and decision models, specifically Jev and Luna, could solve a hard problem that we encounter regularly (grading identity resolution results). We found that while both models performed okay out of the box (88.8–97.3% accuracy), implementing a hybrid model that combines Jev + Opus achieves frontier accuracy at > 4x lower costs.

Our other findings:

  • Luna performs better than Jev out of the box, but the gap disappears on harder cases.
  • Jev's confidence is better calibrated. Most of its mistakes come with low confidence its right. Most of Luna's mistakes come with high confidence. This enables Jev to perform better than Luna in a hybrid model harness.
  • Frontier models are still more accurate than decision models
  • Decision models are up to 90× cheaper and more than 10× faster out of the box. They cost $0.5 - $0.10 per 1,000 identities and answer in about 0.15 seconds, but trail Opus accuracy by about 6 percentage points on harder benchmarks.