Publisher
Serverless cloud for AI and data workloads
Modal Clusters are generally available
Modal announces general availability of Modal Clusters: multi-node GPU clusters with RDMA networking, gang scheduled from its shared capacity pool and billed by the second, enabled with a single @modal.clustered decorator for distributed training, post-training and large-scale inference.
6 min · 1,277 words
Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine
Charles Frye and Shreya on the Modal blog: combining a query planner with an inference engine to push AI-SQL queries past a billion tokens per minute on one GPU—why left-deep joins help KV cache, and how they beat naive vLLM-style serving.
18 min · 4,181 words
How to serve trillions of tokens for trillion-parameter coding agents
Modal explains how it serves coding-agent inference at extreme scale—performance and efficiency techniques for trillion-parameter models generating trillions of tokens, written for teams facing the same workload.
30 min · 6,972 words