Phil Eaton explains why The Consensus added an 'AI tells' detector to its curated technical-writing feed, so it can keep freely promoting thoughtfully human-written articles while letting readers filter likely AI-written ones.

Has AI tells

A big goal of The Consensus is to promote good technical writing in software infrastructure. We feature more than just our own writing. We give you an interesting article every day in the at a glance section. We feature many more articles in our free weekly newsletter. And we filter for good technical articles in our free and public third-party feed.

But Cynthia Dunlop's survey on tech reader preferences has shown that people overwhelming dislike AI-written articles, preferring even "bad" human writing to "good" AI writing. And occasionally I'd feature something that turned out to read very strongly of AI-generated writing and internet commenters would call me out on it. Sometimes these egregiously AI-written articles were written by companies who spoke very passionately about temperate and responsible use of AI in communication. (That was disappointing).

Honestly, I can't stand reading AI-written prose either. It infuriated me to click on a random cool-sounding article in our feed just for the actual writing to grate. A waste of time.

There are high-quality commercial detectors like Pangram. The Consensus pays for my subscription to it. But running Pangram against every article that enters The Consensus feed is financially impossible. So for the past month I've been working on an AI tells detector that's integrated into the site.

It is not a machine learning model, just a bunch of static rules built up by comparing many different aspects of pre-2023 writing to post-2023 writing, accounting for how writing has drifted each year since then.

The detector does not say that the article was written by AI. It only exists to say the article might remind you of AI writing. This is important because the AI tells detector is cheap and not always correct. It has occasionally labeled some of my own articles as quite certainly having AI tells.

With the AI tells detector in place, we label every article in the feed in the 99th percentile of having AI tells. Articles labelled like this are not eligible to show up in the daily At a Glance section of the homepage and on our social media. They are not eligible for the sparkle icon on the feed page that indicates it seems like an interesting article. And they are not eligible to be featured in the weekly newsletter.

Can this be gamed? Teaching the AI to write less like stock Claude and stock ChatGPT? Yes, absolutely. And honestly that would be great. Because half of the irritation is that all these AI-written article sound the same! If they're all being written with each individial's, each company's, own rules and preferences and character that'd probably be nice?

It is the prerogative of every company to decide how they communicate in public. Using AI to write to doesn't make a person or company bad. And for now, we're not completely filtering out articles that have AI tells (though you can filter them out by clicking on the sparkle filter or setting aitells<90 in the feed search). But at least in terms of what we choose to freely promote, we'd like those articles to be thoughtfully written by a human.