![\"\"](\"https://lethain.com/static/blog/2019/aep-small-lq.jpg\")
small-changePublished on October 17, 2026.\nmanagement (222)
\nImprint’s average engineer is merging 4x more PRs than 2 years ago. Yay.\nAggregated behind the average is something even more interesting:\nwe have two teams that are merging 10x their PRs over that period.\nOne engineer is merging 25x the PRs per week of our average engineer two years ago,\nand a number of others are closing in on that number.\nThat’s pretty neat, and the slope of the line suggests our average in a year is going to look much closer to\ntoday’s p90 than today’s p50.\nEven more importantly, we aren’t just shipping more, our quality has improved:\nincidents are down 42% half-over-half, and critical incidents are down 78% half-over-half.\nI think we’ve always known the quality/velocity tradeoff wasn’t a particularly useful\nconstruct, but that’s undeniable today.
\nAs we’ve walked this path, we–like everyone else\nprogressing down it–have run into the problem of pull request review. The p50 pull request review took about 33 minutes to complete\n2 years ago, but now takes us about 77 minutes. The slope here correlates tightly with the number of\npull requests getting merged. Tightly enough that addressing it became one of my priorities for the week.
\nWe solved it! Just kidding, we’ve absolutely not solved it, but I think our current\napproach is an interesting one, which we call the small-change.
The work discussed here was shaped and done by many folks at Imprint including Dan, Joy, Daniel, Lenny, Taylor, Ted, and Annie, among many others.
\nThere are four reasons we review pull requests:
\nKnowledge sharing between author and reviewer – software engineering is generally learned via
apprenticeship, and reviews are a primary mechanism of experienced engineers teaching others what\n good software looks like. We believe the current volume means that this knowledge sharing\n will have to move towards operating on clusters of pull requests, rather than individual pull requests
Preventing errors – review has always been an expensive time to catch errors,
and we believe that effective CICD along with harness verification techniques like poteto’s\n pstack are more effective mechanisms\n for preventing errors than review
Preventing malicious behavior – my experience is that even careful code review will generally
miss competent malicious behavior, although fortunately most malicious behavior is incompetent.\n We believe both sorts of behavior are best identified using a combination of fixed rules (modifying\n these specific objects/classes/files requires review), anti-abuse mechanisms that we’ve refined in\n our LLM-based member support experiences, auditing, and audit log inspection
Compliance obligations in our documented controls – these can be changed, but today they require that
an independent human reviews changes in a subset of our repositories
In the modern development paradigm, we think there is a better solution for each of these problems.\nBut, we also want to make these changes very cautiously.
\nsmall-changeWe believe we move fast by finding narrow, contained ways to validate new practices,\nso to iterate our way into an effective solution, we’ve developed the concept of a\nsmall-change. Here’s how it works:
When a pull request is marked ready, CICD determines whether it is eligible to be a small-change.
This is represented as a passing Github check. There is also a label for visibility,\n but it is the check itself, managed by the build and not human modifiable, that determines eligibility.\n (The label is used for initial filtering, but the check is still independently verified.)
small-change if it meets all of these criteria:ai-code-review automated review botThe author has “experience” in the service or application, defined as 10+ prior merged pull requests
into that service or application’s directories.\n For changes working across multiple services or applications, the author must have experience in all of them to qualify
Impacts fewer than N distinct services or applications. N is 1 on our frontend monorepo,
and N is 3 in our backend monorepo, where typical changes will modify both a gRPC service and a common HTTP gateway
small-changeEach repository has a .small_change file which allows per-path customization while we continue to tune the behaviors.
For example, a service dealing with money movement can be excluded
Our Slack automation tags every reference to a pull request with reacji labeling the state of that pull request,
for example if it has failing tests, if it has been approved but tests are still running, or if it is ready to merge.\n We added a small-change reacji that indicates which of those pull requests meet the small-change criteria
review-small-change which takes a list of repositories and reviews all open, non-draft, small-changes,
including providing approval if they meet the review’s criteria (all checks pass including small-change check,\n and not touching on sensitive topics such as payments and security).
Importantly, this skill has only two options: approve or ignore. It never comments on the PR to avoid\n the proliferation of “meat proxy review”\n which has become a bottleneck-pretending-to-be-a-solution on some teams
review-pr-cluster which reviews batches of pull requests to identify common themes that came up, and suggest opportunities
for discussion and adding verifications
We set the social expectation that any error caused by a small-change is a failure in our system,
and has to be solved at the system level: improvements to ai-code-review, to our verification mechanisms,\n to our CICD, to our harness, etc.
This took about two days of infrequent attention to implement and rollout,\nalthough I imagine we will be tuning the knobs here for a long time. Maybe forever.
\nI mentioned this in 4.1 above, but meat proxy review has become\nan interesting bottleneck for many teams. It’s a particularly difficult bottleneck because it initially looks like a solution,\nbut it makes things worse. The pattern is roughly:
Pull requests get a large number of comments, some of which make sense, and some of which don’t.
It’s unclear which is which without study, and you don’t know which comments reflect the reviewer’s\n point of view. So you end up having to do substantial work to disprove comments that the reviewer may\n not even agree with
Those teams then start using harnesses to respond to the harness-generated comments,
making the entire review process an illusion that doesn’t facilitate learning,\n and most importantly could be done without the involvement of the other team members
In every case that meat proxy review is performed, the team would get better outcomes by\ninvesting into the centralized review mechanism, the harness’ loop for addressing the centralized review’s feedback,\nand simply stamping approval.
\nfleet: centrally orchestrated harnessI’ll briefly mention our other approach to reducing code review\nvolume, which anchors on our centrally-orchestrated harness, fleet.\n(Which is similar to Stripe’s Minions\nalthough not following the one-shot approach in particular).\nWe rely on fleet\nto implement and verify software in a highly controlled harness running in the cloud, with independent\nevaluation mechanisms to identify the correct number of independent reviewers for\nany given change. That number can be as low as one (e.g. fleet writes, one human reviews)\nfor changes evaluated as safe.
I’ll be honest, it’s too early to give a useful result here.\nFrom early results, about 10% of changes qualify as a small-change.\nThose are now eligible for very fast approval, and generally get it unless\nthey trip a flag, generally due to touching a security or money adjacent topic.
While 10% is a small percentage, the core concept here is working,\nand ultimately will succeed or fail based on our ability to expand\nthat percentage over the next few months in tandem with expanding\nour verification mechanisms to be increasingly comprehensive.\nThis is ultimately going to fail or succeed depending on our ability\nto improve our verifications, which does require more difficult sorts of techniques\nto accomplish in low scale-complexity, high domain-complexity software\nthan in the high scale-complexity, low domain-complexity products where many techniques\nlike incremental rollouts are exceptionally effective. (Don’t get me wrong,\nthey still work, they just provide less coverage when most users have different configuration and behavior.)
\nFolks have not been uniformly excited about this,\nit’s a substantial change from how we’ve learned to be good engineers,\nand it’s uncomfortable to reevaluate whether a long-held best practice has become merely dogma.\nThe most energizing part of the current cycle is that we get to reevaluate these\npractices we grew up on every week; the most painful part of the current cycle is that\nwe’re likely to fall behind if we ever stop reevaluating them.
\nHi folks. I'm Will Larson.
\nIf you're looking to reach out to me, here are ways I help.
\nBooks
\nI wrote\nAn Elegant Puzzle,\nStaff Engineer,\nThe Engineering Executive's Primer, and\nCrafting Engineering Strategy.
\n
![\"\"](\"https://lethain.com/static/blog/staffeng/StaffEngBookMed.jpg\")
![\"\"](\"https://lethain.com/static/blog/2023/primer-cover-small.jpg\")
![\"\"](\"https://lethain.com/static/ces_cover_small.png\")
Newsletter
\nIf you'd like to get updates, subscribe for weekly emails, or follow my RSS feed.
\nPopular
\nRecent
\nsmall-changeRelated
\n","headings":[{"level":1,"text":"Speeding up code review with small-change","id":"speeding-up-code-review-with-small-change"},{"level":2,"text":"Why review pull requests?","id":"why-review-pull-requests"},{"level":2,"text":"Small steps with small-change","id":"small-steps-with-small-change"},{"level":2,"text":"Meat proxy review","id":"meat-proxy-review"},{"level":2,"text":"fleet: centrally orchestrated harness","id":"fleet-centrally-orchestrated-harness"},{"level":2,"text":"Results / next steps / reception","id":"results-next-steps-reception"}]}}