I posted to HN, and then my website was flooded with AI BOT traffic

So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).

Digging further, I looked at aggregate NGINX logs to find abusers, and, you're never going to believe this, but it turns out that nearly half of our traffic was coming from one specific user. I won't name names, but I looked them up, and they describe themselves as a service that listens for changes to a website, and then summarizes them into an AI slop news letter. And by 'listens to changes' I mean query our website many many times per-minute. Blocked.

Several of the others were your standard known crawlers, and kindly identified themselves as such. Claude bot, OpenAI bot, google's search indexer, and several others. I even picked up a few vulnerability scanners along the way. At least, somebody is trying to access -/wp-content/phpinfo.php (my website is not a wordpress site). I added robots.txt to discourage most of the AI scanners, but allow google-bot and other search-related traffic, and asked claude to query all the vulnerability paths and mimic the scanners to make sure I wasn't leaking anything. I also added reasonable rate limits, etc., etc.

The most important lesson though, is that it's time to make my website faster. I want to be the website that can day handle xxx QPS of real traffic, and it should scale better than it did, even under heavy load. Inspired by this encounter, I've made quite a few improvements to the underlying infrastructure over the past few weeks, and these should result in pretty substantial performance gains for all users.

P.S. if you'd like to check it out, the website is fully free, and available here

The TLDR is that it lets you define your own customizations, in addition to things like the points totals from hacker news. You can use it to filter out topics, like AI or LLMs, but the topics are completely arbitrary and you can program your own customizations.

2. Using LLMs to Make the Emacs Web Browser Great Again

When I tell people about the emacs web browser, their first reaction is 'Wait emacs has a web browser?!', followed by, 'yeah… I guess that's kind of a web browser'.

The emacs web browser is designed to render webpages in a terminal-based, textonly environment and, well, it goes about exactly the way you'd expect. It works well on HTML-only, small-content websites, but not so good for anything else.

The problem is that emacs is primarily a text-only editor, but the modern internet is very much not plain-text based. Simple HTML, designed in the simple style of the 1990s has a clear translation into plain text, but modern UI design, with it's visual styling, heavy javascript, and just, overall chaos, not so much.

The emacs web browser would be great, if only we could find a way to go back in time…

Ok, so the plan is simple. We take modern webpages, rewrite their source code into old-school simply-styled HTML, and then pipe it through to the emacs web browser. We just need to find a way to do this easily and at scale.

So um, you didn't forget about LLMs did you? It's nice to be using them in a way that will hopefully reduce the amount of spam content in the world.

Basically, the plan is we write a program that:

• Opens a webpage in firefox
• Loads the page and runs all the javascript
• Asks an LLM to generate a re-usable program that transforms the current-page's contents into simplified text
• Inject the generated javascript into the firefox browser
• Grab the new page source and forward it to emacs web browser.

Yeah, and so, I've been using it, and it runs very well. Like realistically I can use this on many webpages where eww (emacs web wowser) did not run previously.

Below are some examples of before and after:

The docs for Clojure's mapcat

Figure 1: (before) Using eww-readable

Figure 2: (after) Using simpleweb-simplify-page

It takes a moment to generate the initial modifier program, but once it's generated, it runs pretty fast, and during day-to-day browsing, it mostly doesn't need any extra LLM calls. I have been using it myself and enjoying the experience.

The source code lives here: https://github.com/sstraust/simpleweb

and is open-source, and free to install, etc., etc.

2. If technology got better, why did everything get worse?

I think we all know the answer to this: bad incentives.

Technology makes systems more productive, but it does not, in fact, make you in particular more productive. The tools that enable computers to be more efficient also enable people to be more efficient at exploiting your time, attention, and wellbeing. I'm talking neural nets babyyy.

I first noticed the problem in 2018. Or rather, I noticed that *I* had a problem, and that problem was spendinng 3 hours a day watching starcraft 2 replays and wondering where the time went, and how I managed to stay up til 2am again.

What's interesting to me is that I didn't have any of these problems until the late 2010s. I had a computer with an internet connection since at least the mid 2000s, and a smartphone with twitter and reddit and the like not long after. It's possible that I've gotten less resiliant over the years, but I have a sneaking suspiscion that it's the computers who changed.

I can't quite put my finger on it, but I have this sensation that every time I interact with a system or website, I'm playing this kind of adversarial game, where I'm trying to just go about my day, and maybe read a little article about computers, and they're trying to figure out a way to keep me on their site for just a little bit longer.

I also have this feeling that as software engineering has moved from a discipline of hobbyists and enthusiasts to leaders who increasingly developed through finance or business-minded tracks, we've found ourselves with these major tools that we used to love become both extremely bland and extremely profitable.

So what can we do it make it stop?

1. touch grass. really when the world is like this, i think the best solution is to just leave. it's not easy, especially when the planning is often online, but if you can set it up, I promise that a day in the park playing chess with your friends is better than any amount of time you can spend on the internet.
2. take to the streets. i think that protesting is effective, and more people should do it. if you're willing to take action, it models to others that they can take action.
3. stop coming back to the same machines. I think we all have this gamblers fallacy in us, where if it didn't pay out the first time, but it pays out sometimes, we try the machine again, but it doesn't really work like that. if you're losing money the lesson is to stop putting coins in the machine.

editor's note: I've been working on a free customizable news reader as a way to take out some of my frustrations with the modern internet. if you don't use this one, I recommend following individual blogs and websites, because it's so much better to just get off these big platforms entirely.

editors note 2: i haven't yet posted this latest article to my rss xml, because i'm not ready to send notifications for it yet. will add in the coming weeks.

3. Deft version 0.2.0

Now that deft 0.2.0 is released, I figure it's a good time to talk about what I built, and why I built it.

The point of Clojure's everything-is-a-map toolkit is that getters and setters are a waste of time. Clojure was built, in large part, in response to the crappy 2008 Java workflow, where you had to write getters, setters, and builders, and you took on all this bloat that nobody enjoyed.

The problem is that if you want to represent your data using the built-in defprotocol and defrecord, and you want to define functions that depend on the presence of a certain field, then the only way to encode that information into the protocol is to add getter and setter metohds to it. icky.

You can also use a combination of protocol + malli schema to represent this, but then when you use it, you need to specify 'this function takes as input something that both implements this protocol and implements this Malli schema'. Also icky.

What we want is something we can use in the style of defprotocol and defrecord, but gives us all the getters and setters and map-like stuff by default. The key insight is that if a field exists in a record, we get all the other good stuff, so we just need to add support for protocols with required keys.

Deft v0.2.0 adds support for namespaced keys in type and protocol definitions. i.e., you can now do:

(defp Positioned :required-keys [::pos - ::Point])

and

(deft Circle [::location/position radius]) (>Circle ::location/position [1 2] :radius 12)

this is useful because you can now define methods that depend on those fields being present i.e.:

(defp Gremlin :extends [location/Positioned]) (defmethod game-object/get-hitbox ::Gremlin [self] (js/Matter.Bodies.rectangle (get-in self [::location/pos ::location/x]) (get-in self [::location/pos ::location/y]) 15 15))

Check it out!

4. How much LLMs is too much LLMs?

I finished reading On the Edge last week, and it was bad. Not that I don't respect Nate Silver. I respect him a lot. So when his seminal text on gambling and risk taking was a rambling mess, I was suprised and disappointed. Nevertheless, a completionist at heart, I trudged on, to the very end, where I was greeted with this message:

"ChatGPT was a significant help in writing this book, serving as a creative muse when coming up with things like chapter subheadings, metaphors, and analogies, and for refining my understanding of technical topics that are likely to be well represented in its corpus."

Technically he wrote the book by hand, and technically he typed the letters onto his keyboard, but you know, did he really? Did he put his whole thought and mind and being into every word on the page? Maybe. I'm reminded of the absolute classic Simple Made Easy, a video where so much of the core lecture is built on defining precise language, and saying exactly what you mean. When you offload that work, you don't just skip word choices, you skip that thinking, and you miss out on that organization and mental clarity.

In reading Nate's book, there's a real and noticeable difference between the language that feels baked into the organizationa