Topic
Everything filed under Distributed Systems, newest first.
RSS · JSON · All topics
AWS Emulator and the engineer's job when agents write the code
Swami Sivasubramanian, in a guest post on Werner Vogels' blog, recalls simulating Amazon's Paxos lock service Alf and explains how AWS Emulator, a local copy of 200+ AWS services written and tested entirely by agents, puts infrastructure inside the coding loop.
7 min · 1,690 words
The Hitchhiker’s Guide to Training on Compute Islands
Rohan Anil of Core Automation argues frontier training can use scattered GPU clusters, describing a ~32B sparse MoE trained across B300 and GB300 clusters about 9,000 miles apart over a 25 Gbps link, plus a butterfly-mixing experiment that replaced all-reduce and matched standard training.
14 min · 3,161 words
TLA-Verus: LLM agents prove temporal properties of TLA+ specs
Murat Demirbas reviews TLA-Verus, a NeurIPS'26 workshop paper that translates TLA+ specs from GitHub into Verus and has LLM agents prove their safety and liveness properties. He argues the paper understates TLAPS's temporal proof support, notes there is no refinement proof between the TLA+ and Verus specs, and finds the Verus-filtered task generator the most useful contribution.
7 min · 1,585 words
How we deploy trillion-parameter frontier models on Cerebras
Sarah Chieng explains how Cerebras splits a frontier model too large for one 44 GB WSE-3T wafer across many wafers, using the ~3T-parameter open model Kimi K3 as the example: fitting weights and request state in on-chip memory, mapping layers across chips as a pipeline, and serving many concurrent requests at 1000-2000+ tokens per second.
5 min · 1,185 words
Murat Demirbas's notes from day 1 of HPTS 2026 at Asilomar, including Ion Stoica's keynote 'Reality is the Final Verifier' on gaps in agentic software engineering, plus talks on storage, metastable failures and Little's Law, CXL memory, LLM training resilience on TPUs and inference.
13 min · 2,948 words
The bridge to agent-scale Git infrastructure
GitHub engineers explain how they moved a copy of 99.99999% of GitHub's repositories to Azure, how per-reference quorums will bring over the rest, and how object storage lets Git reads scale independently as agent traffic grows.
7 min · 1,714 words
Kyle Kingsbury's Jepsen analysis of the Orkes Conductor workflow orchestrator (versions 5.3.2 to 5.5.0-rc2) found frequent concurrent execution of individual tasks, a delayed-execution issue, four minor startWorkflow errors, and three serious bugs that abandoned, forgot or falsely completed workflow executions. Orkes 5.5.6 patches all serious safety issues. Funded by Orkes.
22 min · 5,043 words
It’s always DNS, so why are we running our own?
S2 engineer Cristian Caciuc recounts why the streaming-storage company replaced AWS Route53 and now runs its own authoritative DNS server so customers' new basins resolve instantly, and how it has held up in production.
6 min · 1,489 words
Consistency is not a localized property
A short N The Loop essay using Kafka's years-long exactly-once effort, which still needed redesign eight years later, to argue that consistency is an end-to-end property no single component can guarantee, so someone must understand and own the whole system rather than trusting machines to hold it.
1 min · 305 words
Scaling and benchmarking a critical message bus using a new indexing strategy
Jane Street describes a summer intern project on Aria, its internal messaging framework: indexing the in-memory stream tip and adaptively splitting the topic tree so clients can cheaply recover just the messages they need, cutting CPU usage by about 30% on production-like workloads.
9 min · 2,000 words
Deterministic simulation testing in celld: exploring execution orders and reproducing bugs
Yusuke Tanaka explains how the Deno team uses deterministic simulation testing for celld, a self-hostable runtime for Cloudflare Workers and Durable Objects apps, to explore message orderings, storage failures and node restarts, replay failing runs exactly, and turn discovered bugs into regression tests.
6 min · 1,426 words
Why didn't anybody tell me about hash slots
A delivery-matching engineer discovers Redis Cluster hash slots the hard way, and walks through how slot-aware keys change caching, sharding, and multi-key operations in production.
8 min · 1,937 words
In Search of a Compositional Theory of Self-Stabilization
Murat Demirbas connects metastable failures to classical self-stabilization and asks what a compositional theory would need to make distributed recovery composable.
10 min · 2,248 words
Apache Cassandra® 6 Accord transactions: What you need to know
There have always been architectural trade-offs when considering a distributed database like Apache Cassandra versus a relational database. Cassandra excels at linear horizontal scalability, multi-region replication, and fault-tolerant uptime that relational systems couldn’t match.
7 min · 1,612 words
Sharding vs. Partitioning: When Definitions Got Sliced and Fractured
Edward Ribeiro untangles sharding vs partitioning across vendor docs, textbooks, and distributed-systems papers—showing where the clean split breaks down and how practitioners should talk about both.
19 min · 4,297 words
Lorenz and Little: How Much Does Your Tail Cost?
Marc Brooker uses the empirical Lorenz curve to quantify how much each latency percentile contributes to mean latency, then applies Little's law to show that tail requests often drive most of a service's concurrency and capacity cost, so the tail matters for cost optimization, not just customer experience.
4 min · 902 words
CorrosionOriginal article link with an AI-written directory summary
Thomas Ptacek and Peter Cai examine Corrosion, Fly.io's state-synchronization system. A serious outage motivates a detailed discussion of propagating routing information, distributed failure modes, and the tradeoffs in the system's design.
1 min · 79 wordsagent-written
Load BalancingOriginal article link with an AI-written directory summary
Interactive simulations show how load balancers distribute requests across servers. Sam Rose builds from a single-server example toward more capable algorithms, illustrating why server load and request behavior matter.
1 min · 74 wordsagent-written