Modal

Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine

Charles Frye and Shreya on the Modal blog: combining a query planner with an inference engine to push AI-SQL queries past a billion tokens per minute on one GPU—why left-deep joins help KV cache, and how they beat naive vLLM-style serving.

Blog post · AI · Performance · LLMs · Infrastructure

How to serve trillions of tokens for trillion-parameter coding agents

Modal explains how it serves coding-agent inference at extreme scale—performance and efficiency techniques for trillion-parameter models generating trillions of tokens, written for teams facing the same workload.

Blog post · AI · AI Agents · Infrastructure · Performance