Setting a challenge to spend the whole of September on only one efficient open model felt like a great idea at the time. Turns out not so much in practice. 2B tokens later, here’s how it went.
\nHere’s the tokens distribution according to AgentsView, one of our Agentic engineering recommendations to keep tabs on AI usage:
\nZooming in on the models split specifically:
\nThe goal was to spend the whole month on GLM 5.3 Flash pictured in teal. Here’s what went well:
\nThe second half of the month didn’t go so well, with 1B tokens going to other models.
\nWe’re pretty transparent that our experimental Wagtail MCP server is a vibe-coded prototype. Vibe coding isn’t quite what we normally aspire to, but for a prototype it’s spot on. Unfortunately there are still consequences to it. I chose the 'wrong' model for the prototype, and we spent 450M tokens / $150 / 5kWh of energy use almost overnight. The MCP server itself works well and we now have a great demo of the capabilities, so it’s not for nothing:
\nNonetheless, it’s a good reminder to be careful with model selection and with agentic patterns. We could have achieved similar results for most likely 5x less cost with not that much more effort. Lessons learned! We need to budget for this, and be more careful. Could have seen it coming, but now we know.
\nAnother unexpected hurdle was infrastructure availability issues. We’ve written extensively about comparing inference providers. Our choices work really most of the times, but it turns out they’re very popular, and do not have the same capacity as the big labs who hoard all the GPUs. We noted degradation with the performance of GLM 5.3 Flash in particular, most likely because of it being so high up the Pareto frontier of relevant models for our work.
\nThis meant having to switch to other similar models (DeepSeek V4.1 Flash, Qwen 3.8 Flash). Which is very simple to do, but nonetheless unexpected!
\nLast but not least, beyond using one model for day-to-day engineering, it felt essential to keep experimenting with a wide range of models, keeping up with what providers are releasing. This is particularly essential as we start to benchmark models’ performance on Wagtail tasks, where we need data across a wide range of models. Sneak peek of our benchmark:
\nIt’s much easier to guide people towards leaner options with this kind of concrete data. And for us to make those options even more viable with agent skills, or our new CLI prototype, which is intended to work well with agents.
\nSo technically this challenge was a failure. Only 50% usage on the target model, 1B out of 2B tokens. About 35 kWh of energy use instead of 10. But we did learn a lot, which is crucial for the current moment. Reflecting on this for October, here’s what will make it work:
\nFor day-to-day developer work, it’s totally viable to focus on one or two flash-tier cheap models. A viable target is probably that the majority of AI inference work should be done with such efficient models, measured in cost or energy use rather than meaningless tokens. That’s the goal for October! You should try it too, you’ll learn a lot in the process.
\nAnd come say hi at Wagtail Space 2026 in November to hear how that all pans out!
","headings":[{"level":1,"text":"One month coding with GLM 5.3 Flash","id":"one-month-coding-with-glm-5-3-flash"},{"level":2,"text":"Where tokens went this month","id":"where-tokens-went-this-month"},{"level":2,"text":"Unexpected hurdles","id":"unexpected-hurdles"},{"level":3,"text":"The cost of vibe coding","id":"the-cost-of-vibe-coding"},{"level":3,"text":"Infrastructure woes","id":"infrastructure-woes"},{"level":3,"text":"The cost of experimentation and R&D","id":"the-cost-of-experimentation-and-r-d"},{"level":2,"text":"Takeways and what to do next","id":"takeways-and-what-to-do-next"}]}}