Benchmarking: An Introduction

“When a measure becomes a target, it ceases to be a good measure” – Goodhart’s law Current AI research, especially the frontier LLM research, is dominated by benchmarks. It is the first thing we look at when a new model comes out, it is the headline of each release, and they dominate the discourse when […]

Guide · AI · Benchmarks · LLMs · Research

Frequently Asked Questions (And Answers) About AI Evals

Hamel Husain and Shreya Shankar’s sharp FAQ on AI/LLM product evals: start with error analysis on real traces, build targeted evaluators, validate LLM judges with TPR/TNR, and avoid generic off-the-shelf metrics.

Guide · AI · LLMs · Engineering · Machine Learning

Tuning a Server for Benchmarking

How to tune a Linux server so benchmarks are repeatable: isolating noise from CPU frequency scaling, interrupts, and background services so small performance wins are actually visible.

Guide · Performance · Benchmarks · Linux · Infrastructure