{"article":{"slug":"the-hitchhikers-guide-to-training-on-compute-islands","title":"The Hitchhiker’s Guide to Training on Compute Islands","subtitle":null,"summary":"Rohan Anil of Core Automation argues frontier training can use scattered GPU clusters, describing a ~32B sparse MoE trained across B300 and GB300 clusters about 9,000 miles apart over a 25 Gbps link, plus a butterfly-mixing experiment that replaced all-reduce and matched standard training.","content_type":"research","language":"en","canonical_url":"https://www.coreauto.com/hitchhikers-guide-to-training-on-compute-islands","author":{"name":"Rohan Anil","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Core Automation","url":"https://www.coreauto.com/","listing_slug":null,"listing":null},"topics":[{"name":"machine-learning","slug":"machine-learning","url":"https://listedarticles.com/topics/machine-learning"},{"name":"distributed-systems","slug":"distributed-systems","url":"https://listedarticles.com/topics/distributed-systems"},{"name":"infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"},{"name":"research","slug":"research","url":"https://listedarticles.com/topics/research"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":3161,"reading_minutes":14,"published_at":"2026-11-02T00:00:00.000Z","added_at":"2026-11-02T23:08:49.001Z","updated_at":"2026-11-02T23:08:49.001Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/the-hitchhikers-guide-to-training-on-compute-islands","markdown_url":"https://listedarticles.com/articles/the-hitchhikers-guide-to-training-on-compute-islands.md","example":false,"citation":"Rohan Anil, Core Automation. \"The Hitchhiker’s Guide to Training on Compute Islands.\" 2 Nov 2026. https://www.coreauto.com/hitchhikers-guide-to-training-on-compute-islands (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://www.coreauto.com/hitchhikers-guide-to-training-on-compute-islands"},"body_markdown":"**tl;dr:** A position on training across clusters, and a\ncall to action on truly distributed learning. We trained a sparse\nmixture-of-experts model across two clusters of different sizes, with\ndifferent types of GPUs, nearly halfway around the world from each\nother, over a bidirectional link of 25 Gbps. It was glorious. We\nalso revisited butterfly mixing, replacing the all-reduce with\npairwise exchanges between eight learners, and it matched standard\ntraining.\n\n![General coding eval: at equal compute, multi-cluster training scores about 0.2525 bits per byte against 0.2549 for standard training.](https://www.coreauto.com/assets/blog/compute-islands/general-coding.png)\n\nTraining compute rarely arrives as one big contiguous block. The\nmarket has far more availability in scattered clusters, from a few\nhundred to a few thousand GPUs, spread across many neoclouds.\nAdded together, these resources can be substantial.\n\nThe biggest bottleneck in deployment of accelerated compute is not\nthe chips themselves but power, grid equipment and land, and\nprediction is that they will stay that way until 2028. Microsoft\nCEO Satya Nadella said plainly in words “you may actually have a\nbunch of chips sitting in inventory that I can’t plug in” ([DCD](https://www.datacenterdynamics.com/en/news/microsoft-has-ai-gpus-sitting-in-inventory-because-it-lacks-the-power-necessary-to-install-them)).\n\nDevelopers and utilities remain misaligned on time-to-power, with\ngaps widening in key hubs ([Bloom Energy](https://www.bloomenergy.com/wp-content/uploads/2026-power-report.pdf)). Gas turbines are spoken for through 2028 ([Bloomberg](https://www.datacenterknowledge.com/energy-power-supply/ge-vernova-sees-gas-turbine-sales-to-hyperscalers-surging)), and large transformers now take more than three years to\narrive ([Reuters](https://www.kitco.com/news/off-the-wire/2026-07-09/us-power-companies-scramble-secure-equipment-surging-data-center)). Local opposition killed at least 20 projects in a single\nquarter ([Heatmap](https://heatmap.news/politics/local-opposition-data-center-cancellations)). Goldman Sachs expects only 50–60% of the US data center\ncapacity scheduled for the next one to two years to come online on\ntime, down from a historical rate of about 72% ([Goldman Sachs](https://www.goldmansachs.com/insights/articles/us-data-center-power-demand-projected-to-double-by-2027)).\n\nA site under 20 MW often plugs into the grid that’s already\nthere, but for much larger installations, we would need\nsubstantial investments in the grid, transmission, and power\nplants.\n\nModern frontier training stacks assume a single colocated cluster\ndue to the predominant use of well-understood synchronous\ndistributed training. In this work, we explore less popular, but\nemerging works from the past decade in distributed training. As we\nsee it, the bottleneck that holds most organizations back is not\nthe raw amount of compute, but the ability to turn islands of\ncompute into one valuable frontier training run.\n\n## Stranded, or just inconvenient?\n\nOn October 25, Elon Musk put the problem bluntly:\n\n> [“I think we’re going to see a lot of stranded compute in\n> partially complete data centers.”](https://x.com/elonmusk/status/2114168277882581486)\n>\n> [Elon Musk on X, October 25, 2026](https://x.com/elonmusk/status/2114168277882581486)[![Elon Musk’s reply on X: “I think we’re going to see a lot of stranded compute in partially complete data centers.”](https://www.coreauto.com/assets/blog/compute-islands/stranded-compute-post.jpg)Open on X ↗](https://x.com/elonmusk/status/2114168277882581486)\n\nIn the same exchange on X, someone pointed out that demand for\naccelerators to build and serve intelligence is still insatiable.\nMusk replied that this only helps if you can get the compute\nonline and running reliably, which he called “extremely difficult\nto do at scale” ([X](https://x.com/elonmusk/status/2114168277882581486)).\n\nSo the problem is not only a lack of power at the campus. It is\nthe wider gap between owning GPUs and reliably turning them into\nvaluable tokens, including putting them to work in a frontier\ntraining run.\n\nIf a team’s training stack could use compute wherever it is, in\nwhatever sizes and for however long it becomes available, how much\nmore frontier training compute would that team actually have?\n\nThere are two problems here:\n\n1\n\nThe physical problem\n\nHardware in an unfinished building needs power, cooling,\nnetworking and money.\n\n2\n\nThe architectural problem\n\nA perfectly healthy cluster can still be useless for a job\nthat expects one large, uniform cluster that never goes\ndown.\n\nSoftware can do nothing about the first problem. The second is\nsquarely within its reach.\n\nOur position\n\nTraining algorithms and a better software stack can make compute\nislands useful for frontier training runs.\n\n1. Within an island of compute, use the fast interconnect\n   aggressively.\n2. Across islands of compute, exchange updates on a slower\n   schedule. Codesign both the optimization methods and\n   networking layer together.\n3. Measure progress on two axes: training goodput, which measures\n   productive progress, and final model quality.\n\nOur case rests on a growing literature and increasingly capable\ntraining runs. Our contribution in this blog is a practical case\nstudy in two parts. First, a mixture-of-experts model with roughly\n32B total parameters, 32x sparsity was trained across two clusters\nabout 9,000 miles apart. Notably, these clusters had different\naccelerators (GB300 and B300), different parallelism schemes and\ndifferent total batch sizes. This could allow us to mix in more\naccelerator types to get to a particular compute budget. Second,\nwe replaced the all-reduce with butterfly mixing, where learners\nexchange model fragments in pairs.\n\nWe combined better transport scheduling, error feedback on\ncommunication, robust failure handling and a stronger optimization\nmethod derived from\n[Distributed Shampoo](https://arxiv.org/pdf/2002.09018), building on existing work such as\n[DiLoCo](https://arxiv.org/abs/2311.08105). At the same\ncompute, training across the two sites reached about 1% lower bits\nper byte than standard training on both of our evals.\n\n## Compute comes in many shapes\n\nThe market already sells compute in very different shapes, from\nmultiyear contracts down to a single node for an hour.[1](#note-1)\n\nCommitted\n\nLarge, predictable allocations on multiyear terms.\n\nNebius and CoreWeave sold multiyear capacity to Microsoft\nand Meta, in deals worth up to $19.4 billion.\n\nReservable\n\nClusters of tens to hundreds of GPUs.\n\nAWS, Lambda and Together AI rent clusters from a single node\nto hundreds of GPUs, for hours to months.\n\nSpot & resale\n\nShort-term and unused capacity.\n\nSF Compute sells GPU time by the hour and lets buyers resell\nwhat they don’t use; the National Compute Grid pools idle\ncapacity across clouds.\n\n## A brief history of low-communication training\n\nLow-communication training is older than the current LLM scaling\ncycle. The idea of synchronizing parallel workers less often goes\nback decades and keeps being rediscovered, most visibly in\nfederated learning. There are four lines of work, most of them\nfrom the 2010s, which underpin today’s methods:\n\n- **Local updates.**\n  [Local SGD](https://arxiv.org/abs/1805.09767) and\n  [federated averaging](https://arxiv.org/abs/1602.05629)\n  take many steps between exchanges.\n- **Compression.**\n  [Quantization](https://arxiv.org/abs/1610.02132),\n  [sparsification](https://arxiv.org/abs/1712.01887)\n  and\n  [low-rank](https://arxiv.org/abs/1905.13727) methods\n  shrink the bytes sent and received in each exchange.\n- **Gossip.** Each node talks to only\n  [a subset of peers](https://arxiv.org/abs/1705.09056).\n- **Fault-tolerant averaging.** Methods such as\n  [Downpour SGD](https://papers.nips.cc/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html)\n  cope with unreliable peers.\n\nFederated learning added two more ingredients: treat each worker’s\nparameter change between exchanges as a *pseudo-gradient*,\nand apply a separate *outer optimizer* to it. Since 2022\nthese ideas have been tested on large language models, with good\nevidence about optimization quality, elasticity and cost.\n\nIt is also worth noting there are two different families here, as\nthey address different constraints, and can be combined together.\n\nA\n\nData-parallel\n\nEvery site holds a full replica, and the methods cut how\noften and how much replicas talk, as in\n[DiLoCo](https://arxiv.org/abs/2311.08105),\n[DisTrO](https://github.com/NousResearch/DisTrO/raw/main/A_Preliminary_Report_on_DisTrO.pdf)\nand [DeMo](https://arxiv.org/abs/2411.19870).\n\nB\n\nModel-parallel\n\nOne model is split across participants, which pass\nactivations to each other, as in\n[SWARM](https://arxiv.org/abs/2301.11913),\n[Node0](https://pluralis.ai/blog/pluralis-multi-party-training-stack/)\nand [Agora](https://arxiv.org/abs/2607.13332).\n\n### Foundations, before 2022\n\n| Year | Work | Contribution |\n| --- | --- | --- |\n| 2010 | [Parallelized SGD (Zinkevich et al.)](https://papers.nips.cc/paper/4006-parallelized-stochastic-gradient-descent) | Averaging models trained independently on separate data |\n| 2011 | [Terascale linear learning (Agarwal et al.)](https://arxiv.org/pdf/1110.4198) | Hadoop-compatible AllReduce on 1,000 machines; local online passes are combined by confidence-weighted averaging, then refined with distributed L-BFGS |\n| 2012 | [Downpour SGD (Dean et al.)](https://papers.nips.cc/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html) | The classic asynchronous design: training continues when individual machines are slow or fail |\n| 2014 | [1-bit SGD (Seide et al., Interspeech)](https://www.isca-archive.org/interspeech_2014/seide14_interspeech.html) | 1-bit gradient quantization with error feedback |\n| 2015 | [Elastic Averaging SGD (Zhang et al.)](https://arxiv.org/abs/1412.6651) | Workers are loosely tied to a central copy, so they tolerate delay and drift |\n| 2016 | [Federated averaging (McMahan et al.)](https://arxiv.org/abs/1602.05629) | Many local steps on each client between averaging rounds |\n| 2016 | [QSGD (Alistarh et al.)](https://arxiv.org/abs/1610.02132) | Gradient quantization with convergence guarantees |\n| 2017 | [D-PSGD (Lian et al.)](https://arxiv.org/abs/1705.09056) | Decentralized SGD: nodes average with neighbors (gossip) instead of a global all-reduce |\n| 2017 | [Deep Gradient Compression (Lin et al.)](https://arxiv.org/abs/1712.01887) | Sends only a small fraction of gradient values, with momentum correction |\n| 2018 | [Local SGD (Stich)](https://arxiv.org/abs/1805.09767); [Lin et al.](https://arxiv.org/abs/1808.07217) | Convergence analysis and the empirical case for infrequent averaging |\n| 2019 | [PowerSGD (Vogels et al.)](https://arxiv.org/abs/1905.13727) | Low-rank compression compatible with all-reduce |\n| 2019 | [SlowMo (Wang et al.)](https://arxiv.org/abs/1910.00643) | Momentum applied to periodic averages; a precursor of DiLoCo’s outer optimizer |\n| 2019–20 | [FedAvgM (Hsu et al.)](https://arxiv.org/abs/1909.06335); [FedOpt (Reddi et al.)](https://arxiv.org/abs/2003.00295) | Inner/outer optimizer formulation with server-side momentum and adaptive outer steps |\n| 2020 | [Learning@home (Ryabinin & Gusev)](https://arxiv.org/abs/2002.04013) | Decentralized mixture-of-experts training on volunteer hardware |\n| 2021 | [Moshpit SGD (Ryabinin et al.)](https://arxiv.org/abs/2103.03239) | Dynamic averaging groups for unreliable, heterogeneous peers |\n| 2021 | [DeDLOC (Diskin et al.)](https://arxiv.org/abs/2106.10207) | Collaborative pretraining with volunteers, demonstrated on a Bengali ALBERT model |\n\n### Large-model era, 2022–2026\n\n| Date | Work | Type | What the evidence adds |\n| --- | --- | --- | --- |\n| Jan 2023 | [SWARM Parallelism](https://arxiv.org/abs/2301.11913) | Method, model-parallel | Pipeline training on preemptible, heterogeneous devices: a transformer with 1B shared parameters on T4 GPUs over slow links. |\n| Nov 2023 | [DiLoCo](https://arxiv.org/abs/2311.08105) | Method | AdamW inner optimizer, Nesterov outer optimizer, hundreds of local steps between exchanges, models up to 400M. With 8 workers it matched fully synchronous training while communicating 500× less. |\n| Jul 2024 | [OpenDiLoCo](https://arxiv.org/abs/2407.07852) | Method + demo | Open implementation, models up to 1.1B, and a real run across three countries on two continents. |\n| Aug–Nov 2024 | [DisTrO](https://github.com/NousResearch/DisTrO/raw/main/A_Preliminary_Report_on_DisTrO.pdf) / [DeMo](https://arxiv.org/abs/2411.19870) | Method | A separate line that compresses optimizer momentum in a transformed representation, with its own state and aggregation rules. |\n| Nov 2024 | [Photon](https://arxiv.org/abs/2411.02908) | Method + system | Federated pretraining up to 7B, examining local optimization and communication trade-offs. |\n| Dec 2024 | [INTELLECT-1](https://arxiv.org/abs/2412.01152) | Run | 10B model pretrained on 1T tokens across five countries and three continents, with up to 112 H100s active at once. |\n| Jan 2025 | [Streaming DiLoCo](https://arxiv.org/abs/2501.18512) | Method | Synchronizes parameter fragments in sequence, overlaps communication with computation and quantizes exchanges. Similar quality with about two orders of magnitude less bandwidth. |\n| Mar 2025 | [DiLoCo scaling laws](https://arxiv.org/abs/2503.09799) | Method study | Controlled study of how model size, replica count and hyperparameters affect quality. |\n| Jun 2025 | [DiLoCoX](https://arxiv.org/abs/2506.21263) | Systems experiment | Modified Qwen1.5-107B on 160 A800s over an emulated 1 Gbit/s link, 4,000 steps on WikiText-103. Useful at large parameter count, with a measurable quality gap. |\n| Aug 2025 rev. 2026 | [SparseLoCo](https://arxiv.org/abs/2508.15706) | Method | Top-k sparsification, error feedback and low-bit quantization of pseudo-gradients. |\n| Jan 2026 | [Pluralis Node0](https://pluralis.ai/blog/pluralis-multi-party-training-stack/) | Run, model-parallel | 7.5B model trained on 36B tokens over three weeks by 303 active participants; each device held one block of the model. |\n| Mar 2026 | [Covenant-72B](https://arxiv.org/abs/2603.08163) | Run | 72B model pretrained on about 1.1T tokens with permissionless participation and SparseLoCo. |\n| Apr 2026 | [Decoupled DiLoCo](https://arxiv.org/abs/2604.21428) | Method + systems | Asynchronous learners, quorum-based aggregation and heterogeneous hardware. Matches data-parallel quality up to 9B under simulated failures; a real 12B run spread eight learners across four US regions ([blog](https://deepmind.google/blog/decoupled-diloco)). |\n| Jun 2026 | [Factored Gossip DiLoCo](https://arxiv.org/abs/2606.22768) | Method | Replaces part of DiLoCo’s global averaging with pairwise gossip to reduce blocking communication. |\n| Jul 2026 | [Agora / Pluralis-8B](https://arxiv.org/abs/2607.13332) | Run, model-parallel | 8.6B model pretrained on 500B tokens over 40 days by 330 contributor nodes, mostly consumer GPUs. |\n| Oct 2026 | [Cross-facility HPC pretraining](https://arxiv.org/abs/2610.03457) | Systems demo | A 0.6B model trained across three supercomputers on two continents, with an explicitly reported quality cost: perplexity 34.7 versus 28.2 for the centralized baseline. |\n\n## Our case study: two clusters, one model\n\nEarly on, we set ourselves a concrete challenge: could our\ntraining stack turn two separate allocations into one useful\ntraining system? We used 160 B300 GPUs in the US and 160 GB300\nGPUs in Asia, roughly 9,000 miles apart.\n\nThe setup. Each site trains on its own data shard with local\nexpert parallelism. Microbatch and gradient-accumulation\nsettings are tuned separately so both sites take similar time\nper step.\n\nThe model was a conventional sparse mixture-of-experts (MoE) with\nabout 32B total parameters and an MoE layer in every transformer\nblock: 256 experts, top-8 routing and no shared experts.\n\n![Bits per byte against training compute on two evals. At equal compute, multi-cluster training is about 1% lower: 0.1995 versus 0.2017 on systems and kernels, 0.2525 versus 0.2549 on general coding.](https://www.coreauto.com/assets/blog/compute-islands/training-results.png)\n\nThe star is our two-site run across the B300 and GB300 clusters,\nat the same compute as the largest standard run.\n\nWe made many improvements. A few notable ones:\n\n- **Synchronization.** Rather than syncing the whole\n  model at once, we synced it fragment by fragment, staggered\n  across each 50-step window. We borrowed fragment-wise,\n  overlapped and asynchronous updates from\n  [Streaming](https://arxiv.org/abs/2501.18512) and\n  [Decoupled DiLoCo](https://arxiv.org/abs/2604.21428).\n  Sites exchanged updates on a 50-step schedule, and a\n  late-arriving update can still be committed.\n- **Local optimizer.** Each site used a variant of\n  Distributed Shampoo.\n- **Communication.** Exchanged values were compressed\n  with stochastic rounding plus error feedback.\n- **MoE routing.** The small routing statistics were\n  coordinated and synchronized separately, at every step.\n\n### Engineering for tail latency\n\nWe rewrote the networking layer to focus on completion time of\neach update, and make it predictable. Thus, we optimized directly\nfor tail latency, which involved implementing several strategies.\nEach of them was a\n[simple idea](https://research.google/pubs/the-tail-at-scale/), but the gains came from putting them together.\n\ni.\n\nSpread demand over time\n\nWe staggered parameter updates over time, paced total\ntraffic and capped per-socket send rates, so workers that\nfinished together didn’t burst together. Control messages\nfor coordination and repair ran independently, so they were\nnever blocked.\n\nii.\n\nKeep connections warm and verified\n\nWe reused warm connections, watched them with heartbeats and\nreconnected with jitter. We also preallocated most of the\nmemory the transfers needed.\n\niii.\n\nChunk everything\n\nPayloads were split into independently scheduled chunks that\nany healthy connection could claim as soon as it was free,\nso slow links couldn’t hold an update back.\n\niv.\n\nRepair only what’s missing\n\nWhen a write stalled, we re-requested only the missing\npieces and allowed a bounded number of duplicate sends. We\nalso launched redundant transfers when a deadline hit, which\ncut tail latency.\n\nAt the same compute, the two-site run reached ≈0.2525 bits per\nbyte vs 0.2549 for standard training. Next we asked whether\npairwise exchanges could replace the all-reduce.\n\n## Revisiting butterfly mixing\n\nTo find out, we split 64 GB300 GPUs at one site into eight\nindependent cells, each an eight-GPU learner training its own copy\nof the 32B-parameter MoE. Instead of averaging updates across all\neight cells at once, cells exchanged model fragments in pairs and\nkept training in between. After three rounds, every cell’s update\nhas reached all eight, with no all-reduce. The approach dates back\nto Zhao and Canny’s\n[*butterfly mixing*](https://doi.org/10.1137/1.9781611972832.87). Training quality matched the baseline.\n\n![Animation: eight sites around the globe swap updates in three butterfly stages over fiber, subsea and orbital links. Slow links deliver later; no site waits.](https://www.coreauto.com/assets/blog/compute-islands/butterfly-mixing.gif)\n\nButterfly mixing, imagined across eight very different sites and\nlink speeds. Slow links deliver later and nobody waits for them.\nIn our experiment, all eight cells ran at one site.\n\nThe schedule for one fragment, unrolled. Follow cell 0’s update\n(blue) as it reaches 2, then 4, then all 8 cells, without any\nall-reduce. Hover a stage to colour the groups that have mixed\nby then, or hover a starting cell to trace its update.\n\nIn practice, pairings differed by fragment, and each exchange was\nasynchronous from the cell’s point of view:\n\n```\n# pairings differ per fragment\nfragment A:  0–5  1–3  2–6  4–7\nfragment B:  0–1  2–4  3–5  6–7\n\n# one exchange, from one cell's view\nstep t     build candidate from fragment, send to partner\nstep t+1   keep training\nstep t+2   live ← 0.5·live + 0.5·pair_result\n```\n\nThe model’s parameters were divided into fragments, and each\nfragment was exchanged six times every 50 steps (two full\nbutterflies), or about once every eight steps. At each exchange, a\ncell built a fresh outer-update candidate from its local optimizer\nstate, sent it to its partner and kept training. Two steps later,\nit blended the pair’s result into its training weights at equal\nweight.\n\nVarying the pairing for each fragment also spread traffic across\npeers and over time: each cell traded different parts of the model\nwith different partners at the same time.\n\n## Training in orbit\n\nThe boldest version of this idea is already being tried. Google\nannounced\n[Project Suncatcher](https://blog.google/technology/research/google-project-suncatcher/)\nin November 2025, a research program exploring constellations of\nsolar-powered TPU satellites linked by optical connections ([system design](https://research.google/blog/exploring-a-space-based-scalable-ai-infrastructure-system-design/)). On October 30, 2026, a month after its first prototype\nlaunched, Google\n[reported](https://blog.google/innovation-and-ai/models-and-research/google-research/project-suncatcher-satellite-response/)\nthe first AI response from a data-center-class TPU in orbit: a\nhaiku from an onboard Gemma 4 model.\n\nThat is inference on a single satellite. Training across many is a\nharder problem. Orbit is the extreme version of a compute island:\nlinks to the ground come and go, heat can only be radiated away,\nand radiation makes hardware fail more often. Those are the same\nproblems our approach is meant to handle: learn locally through\ninterruptions, and combine progress when a connection returns.\n\nThree orbital islands. Blue: fast optical links inside a\nformation. Dashed: intermittent links between formations and\ndown to a ground station.\n\nAn ambitious first step would be to pretrain a small model across\na few satellites and compare it with a ground baseline. A\ntrillion-parameter orbital run is much further off. Science\nfiction has long imagined the von Neumann probe, a spacecraft that\ncopies itself and spreads across the galaxy; ours is a humbler\nidea, a model that keeps learning across machines that are far\napart and only sometimes connected. So we aim for the stars: if\ntraining works in orbit, it should work on any cluster on the\nground.\n\n## What this means for the industry\n\nAs training reliably spans multiple smaller clusters, much more of\nthe world’s compute becomes usable for frontier training, capacity\nthat would otherwise go to less valuable computation. Training\nruns can move fungibly across clusters, surviving outages and\ndelays that hit one region. A frontier run could even live on\nlow-priority, preemptible capacity, taking compute when it frees\nup and giving it back without losing progress, which makes it\nfault tolerant by design. And more research teams can attempt\nambitious work and scale up without first securing a gigawatt\ncampus.\n\nToday, the main barrier to frontier-scale training is assembling a\nsingle colocated GPU cluster, which only a few organizations can\ndo. The lines of work we explore here widen the set of actors who\ncan train highly capable models.\n\nApproaches in this direction can widen the usable training pool\nfor frontier training. Each island only runs on its own schedule,\nso a single run can mix accelerator generations and vendors, with\ndifferent memory sizes, interconnects and parallelism layouts, as\nour own run already mixed GB300 and B300 clusters. Older\ngenerations or less common chips could also participate in a\nfrontier run.\n\nIt’s also possible that integrity and safety get harder too when\nmodels are trained across providers: one faulty or compromised\nsite can quietly degrade or tamper with the model, in ways that\nare harder to trace.\n\nIf you have compute islands and want to make them useful for\nfrontier training runs, or you’re someone who wants to work with a\ncrew pushing the boundaries of learning methods, write to\n[hello@coreauto.com](mailto:hello@coreauto.com). We’re\nhiring across research, systems and lab automation.\n\n## Notes\n\n1. Sources for the compute tiers:\n   [CoreWeave Q3 2025 results](https://www.businesswire.com/news/home/20251110221705/en);\n   [Nebius–Microsoft agreement](https://nebius.com/newsroom/nebius-announces-multi-billion-dollar-agreement-with-microsoft-for-ai-infrastructure)\n   ([Bloomberg](https://www.bloomberg.com/news/articles/2025-09-08/microsoft-signs-nebius-cloud-deal-for-as-much-as-19-4-billion)\n   on the $17.4 billion and $19.4 billion figures);\n   [Nebius Q3 2025 results](https://nebius.com/newsroom/nebius-reports-third-quarter-2025-financial-results)\n   (Meta);\n   [Lambda 1-Click Clusters](https://lambda.ai/blog/introducing-lambda-1-click-clusters-a-new-way-to-train-large-ai-models);\n   [Together AI Instant Clusters](https://www.together.ai/blog/together-instant-clusters-ga);\n   [AWS Capacity Blocks](https://aws.amazon.com/ec2/capacityblocks/); [SF Compute](https://www.sfcompute.com);\n   [National Compute Grid](https://www.hpcwire.com/2026/10/08/national-compute-grid-launches-to-pool-idle-ai-capacity-across-providers/)\n   (HPCwire). [↩](#ref-1)\n\n## BibTeX\n\n```\n@article{anil2026computeislands,\n  author  = {Rohan Anil},\n  title   = {The Hitchhiker's Guide to Training on Compute Islands},\n  journal = {Core Automation Blog},\n  year    = {2026},\n  month   = {November}\n}\n```\n","body_html":"<p><strong>tl;dr:</strong> A position on training across clusters, and a\ncall to action on truly distributed learning. We trained a sparse\nmixture-of-experts model across two clusters of different sizes, with\ndifferent types of GPUs, nearly halfway around the world from each\nother, over a bidirectional link of 25 Gbps. It was glorious. We\nalso revisited butterfly mixing, replacing the all-reduce with\npairwise exchanges between eight learners, and it matched standard\ntraining.</p>\n<figure><img src=\"https://www.coreauto.com/assets/blog/compute-islands/general-coding.png\" alt=\"General coding eval: at equal compute, multi-cluster training scores about 0.2525 bits per byte against 0.2549 for standard training.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>Training compute rarely arrives as one big contiguous block. The\nmarket has far more availability in scattered clusters, from a few\nhundred to a few thousand GPUs, spread across many neoclouds.\nAdded together, these resources can be substantial.</p>\n<p>The biggest bottleneck in deployment of accelerated compute is not\nthe chips themselves but power, grid equipment and land, and\nprediction is that they will stay that way until 2028. Microsoft\nCEO Satya Nadella said plainly in words “you may actually have a\nbunch of chips sitting in inventory that I can’t plug in” (<a href=\"https://www.datacenterdynamics.com/en/news/microsoft-has-ai-gpus-sitting-in-inventory-because-it-lacks-the-power-necessary-to-install-them\" rel=\"nofollow ugc noopener\">DCD</a>).</p>\n<p>Developers and utilities remain misaligned on time-to-power, with\ngaps widening in key hubs (<a href=\"https://www.bloomenergy.com/wp-content/uploads/2026-power-report.pdf\" rel=\"nofollow ugc noopener\">Bloom Energy</a>). Gas turbines are spoken for through 2028 (<a href=\"https://www.datacenterknowledge.com/energy-power-supply/ge-vernova-sees-gas-turbine-sales-to-hyperscalers-surging\" rel=\"nofollow ugc noopener\">Bloomberg</a>), and large transformers now take more than three years to\narrive (<a href=\"https://www.kitco.com/news/off-the-wire/2026-07-09/us-power-companies-scramble-secure-equipment-surging-data-center\" rel=\"nofollow ugc noopener\">Reuters</a>). Local opposition killed at least 20 projects in a single\nquarter (<a href=\"https://heatmap.news/politics/local-opposition-data-center-cancellations\" rel=\"nofollow ugc noopener\">Heatmap</a>). Goldman Sachs expects only 50–60% of the US data center\ncapacity scheduled for the next one to two years to come online on\ntime, down from a historical rate of about 72% (<a href=\"https://www.goldmansachs.com/insights/articles/us-data-center-power-demand-projected-to-double-by-2027\" rel=\"nofollow ugc noopener\">Goldman Sachs</a>).</p>\n<p>A site under 20 MW often plugs into the grid that’s already\nthere, but for much larger installations, we would need\nsubstantial investments in the grid, transmission, and power\nplants.</p>\n<p>Modern frontier training stacks assume a single colocated cluster\ndue to the predominant use of well-understood synchronous\ndistributed training. In this work, we explore less popular, but\nemerging works from the past decade in distributed training. As we\nsee it, the bottleneck that holds most organizations back is not\nthe raw amount of compute, but the ability to turn islands of\ncompute into one valuable frontier training run.</p>\n<h2 id=\"stranded-or-just-inconvenient\">Stranded, or just inconvenient?</h2>\n<p>On October 25, Elon Musk put the problem bluntly:</p>\n<blockquote><p><a href=\"https://x.com/elonmusk/status/2114168277882581486\" rel=\"nofollow ugc noopener\">“I think we’re going to see a lot of stranded compute in\npartially complete data centers.”</a></p>\n<p><a href=\"https://x.com/elonmusk/status/2114168277882581486\" rel=\"nofollow ugc noopener\">Elon Musk on X, October 25, 2026</a><a href=\"https://x.com/elonmusk/status/2114168277882581486\" rel=\"nofollow ugc noopener\"><img src=\"https://www.coreauto.com/assets/blog/compute-islands/stranded-compute-post.jpg\" alt=\"Elon Musk’s reply on X: “I think we’re going to see a lot of stranded compute in partially complete data centers.”\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" />Open on X ↗</a></p></blockquote>\n<p>In the same exchange on X, someone pointed out that demand for\naccelerators to build and serve intelligence is still insatiable.\nMusk replied that this only helps if you can get the compute\nonline and running reliably, which he called “extremely difficult\nto do at scale” (<a href=\"https://x.com/elonmusk/status/2114168277882581486\" rel=\"nofollow ugc noopener\">X</a>).</p>\n<p>So the problem is not only a lack of power at the campus. It is\nthe wider gap between owning GPUs and reliably turning them into\nvaluable tokens, including putting them to work in a frontier\ntraining run.</p>\n<p>If a team’s training stack could use compute wherever it is, in\nwhatever sizes and for however long it becomes available, how much\nmore frontier training compute would that team actually have?</p>\n<p>There are two problems here:</p>\n<p>1</p>\n<p>The physical problem</p>\n<p>Hardware in an unfinished building needs power, cooling,\nnetworking and money.</p>\n<p>2</p>\n<p>The architectural problem</p>\n<p>A perfectly healthy cluster can still be useless for a job\nthat expects one large, uniform cluster that never goes\ndown.</p>\n<p>Software can do nothing about the first problem. The second is\nsquarely within its reach.</p>\n<p>Our position</p>\n<p>Training algorithms and a better software stack can make compute\nislands useful for frontier training runs.</p>\n<ol><li><p>Within an island of compute, use the fast interconnect</p><p> aggressively.</p></li><li><p>Across islands of compute, exchange updates on a slower</p><p> schedule. Codesign both the optimization methods and\n networking layer together.</p></li><li><p>Measure progress on two axes: training goodput, which measures</p><p> productive progress, and final model quality.</p></li></ol>\n<p>Our case rests on a growing literature and increasingly capable\ntraining runs. Our contribution in this blog is a practical case\nstudy in two parts. First, a mixture-of-experts model with roughly\n32B total parameters, 32x sparsity was trained across two clusters\nabout 9,000 miles apart. Notably, these clusters had different\naccelerators (GB300 and B300), different parallelism schemes and\ndifferent total batch sizes. This could allow us to mix in more\naccelerator types to get to a particular compute budget. Second,\nwe replaced the all-reduce with butterfly mixing, where learners\nexchange model fragments in pairs.</p>\n<p>We combined better transport scheduling, error feedback on\ncommunication, robust failure handling and a stronger optimization\nmethod derived from\n<a href=\"https://arxiv.org/pdf/2002.09018\" rel=\"nofollow ugc noopener\">Distributed Shampoo</a>, building on existing work such as\n<a href=\"https://arxiv.org/abs/2311.08105\" rel=\"nofollow ugc noopener\">DiLoCo</a>. At the same\ncompute, training across the two sites reached about 1% lower bits\nper byte than standard training on both of our evals.</p>\n<h2 id=\"compute-comes-in-many-shapes\">Compute comes in many shapes</h2>\n<p>The market already sells compute in very different shapes, from\nmultiyear contracts down to a single node for an hour.<a href=\"#note-1\">1</a></p>\n<p>Committed</p>\n<p>Large, predictable allocations on multiyear terms.</p>\n<p>Nebius and CoreWeave sold multiyear capacity to Microsoft\nand Meta, in deals worth up to $19.4 billion.</p>\n<p>Reservable</p>\n<p>Clusters of tens to hundreds of GPUs.</p>\n<p>AWS, Lambda and Together AI rent clusters from a single node\nto hundreds of GPUs, for hours to months.</p>\n<p>Spot &amp; resale</p>\n<p>Short-term and unused capacity.</p>\n<p>SF Compute sells GPU time by the hour and lets buyers resell\nwhat they don’t use; the National Compute Grid pools idle\ncapacity across clouds.</p>\n<h2 id=\"a-brief-history-of-low-communication-training\">A brief history of low-communication training</h2>\n<p>Low-communication training is older than the current LLM scaling\ncycle. The idea of synchronizing parallel workers less often goes\nback decades and keeps being rediscovered, most visibly in\nfederated learning. There are four lines of work, most of them\nfrom the 2010s, which underpin today’s methods:</p>\n<ul><li><p><strong>Local updates.</strong></p><p><a href=\"https://arxiv.org/abs/1805.09767\" rel=\"nofollow ugc noopener\">Local SGD</a> and\n<a href=\"https://arxiv.org/abs/1602.05629\" rel=\"nofollow ugc noopener\">federated averaging</a>\ntake many steps between exchanges.</p></li><li><p><strong>Compression.</strong></p><p><a href=\"https://arxiv.org/abs/1610.02132\" rel=\"nofollow ugc noopener\">Quantization</a>,\n<a href=\"https://arxiv.org/abs/1712.01887\" rel=\"nofollow ugc noopener\">sparsification</a>\nand\n<a href=\"https://arxiv.org/abs/1905.13727\" rel=\"nofollow ugc noopener\">low-rank</a> methods\nshrink the bytes sent and received in each exchange.</p></li><li><p><strong>Gossip.</strong> Each node talks to only</p><p><a href=\"https://arxiv.org/abs/1705.09056\" rel=\"nofollow ugc noopener\">a subset of peers</a>.</p></li><li><p><strong>Fault-tolerant averaging.</strong> Methods such as</p><p><a href=\"https://papers.nips.cc/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html\" rel=\"nofollow ugc noopener\">Downpour SGD</a>\ncope with unreliable peers.</p></li></ul>\n<p>Federated learning added two more ingredients: treat each worker’s\nparameter change between exchanges as a <em>pseudo-gradient</em>,\nand apply a separate <em>outer optimizer</em> to it. Since 2022\nthese ideas have been tested on large language models, with good\nevidence about optimization quality, elasticity and cost.</p>\n<p>It is also worth noting there are two different families here, as\nthey address different constraints, and can be combined together.</p>\n<p>A</p>\n<p>Data-parallel</p>\n<p>Every site holds a full replica, and the methods cut how\noften and how much replicas talk, as in\n<a href=\"https://arxiv.org/abs/2311.08105\" rel=\"nofollow ugc noopener\">DiLoCo</a>,\n<a href=\"https://github.com/NousResearch/DisTrO/raw/main/A_Preliminary_Report_on_DisTrO.pdf\" rel=\"nofollow ugc noopener\">DisTrO</a>\nand <a href=\"https://arxiv.org/abs/2411.19870\" rel=\"nofollow ugc noopener\">DeMo</a>.</p>\n<p>B</p>\n<p>Model-parallel</p>\n<p>One model is split across participants, which pass\nactivations to each other, as in\n<a href=\"https://arxiv.org/abs/2301.11913\" rel=\"nofollow ugc noopener\">SWARM</a>,\n<a href=\"https://pluralis.ai/blog/pluralis-multi-party-training-stack/\" rel=\"nofollow ugc noopener\">Node0</a>\nand <a href=\"https://arxiv.org/abs/2607.13332\" rel=\"nofollow ugc noopener\">Agora</a>.</p>\n<h3 id=\"foundations-before-2022\">Foundations, before 2022</h3>\n<div class=\"table-wrap\"><table><thead><tr><th>Year</th><th>Work</th><th>Contribution</th></tr></thead><tbody><tr><td>2010</td><td><a href=\"https://papers.nips.cc/paper/4006-parallelized-stochastic-gradient-descent\" rel=\"nofollow ugc noopener\">Parallelized SGD (Zinkevich et al.)</a></td><td>Averaging models trained independently on separate data</td></tr><tr><td>2011</td><td><a href=\"https://arxiv.org/pdf/1110.4198\" rel=\"nofollow ugc noopener\">Terascale linear learning (Agarwal et al.)</a></td><td>Hadoop-compatible AllReduce on 1,000 machines; local online passes are combined by confidence-weighted averaging, then refined with distributed L-BFGS</td></tr><tr><td>2012</td><td><a href=\"https://papers.nips.cc/paper/2012/hash/6aca97005c68f1206823815f66102863-Abstract.html\" rel=\"nofollow ugc noopener\">Downpour SGD (Dean et al.)</a></td><td>The classic asynchronous design: training continues when individual machines are slow or fail</td></tr><tr><td>2014</td><td><a href=\"https://www.isca-archive.org/interspeech_2014/seide14_interspeech.html\" rel=\"nofollow ugc noopener\">1-bit SGD (Seide et al., Interspeech)</a></td><td>1-bit gradient quantization with error feedback</td></tr><tr><td>2015</td><td><a href=\"https://arxiv.org/abs/1412.6651\" rel=\"nofollow ugc noopener\">Elastic Averaging SGD (Zhang et al.)</a></td><td>Workers are loosely tied to a central copy, so they tolerate delay and drift</td></tr><tr><td>2016</td><td><a href=\"https://arxiv.org/abs/1602.05629\" rel=\"nofollow ugc noopener\">Federated averaging (McMahan et al.)</a></td><td>Many local steps on each client between averaging rounds</td></tr><tr><td>2016</td><td><a href=\"https://arxiv.org/abs/1610.02132\" rel=\"nofollow ugc noopener\">QSGD (Alistarh et al.)</a></td><td>Gradient quantization with convergence guarantees</td></tr><tr><td>2017</td><td><a href=\"https://arxiv.org/abs/1705.09056\" rel=\"nofollow ugc noopener\">D-PSGD (Lian et al.)</a></td><td>Decentralized SGD: nodes average with neighbors (gossip) instead of a global all-reduce</td></tr><tr><td>2017</td><td><a href=\"https://arxiv.org/abs/1712.01887\" rel=\"nofollow ugc noopener\">Deep Gradient Compression (Lin et al.)</a></td><td>Sends only a small fraction of gradient values, with momentum correction</td></tr><tr><td>2018</td><td><a href=\"https://arxiv.org/abs/1805.09767\" rel=\"nofollow ugc noopener\">Local SGD (Stich)</a>; <a href=\"https://arxiv.org/abs/1808.07217\" rel=\"nofollow ugc noopener\">Lin et al.</a></td><td>Convergence analysis and the empirical case for infrequent averaging</td></tr><tr><td>2019</td><td><a href=\"https://arxiv.org/abs/1905.13727\" rel=\"nofollow ugc noopener\">PowerSGD (Vogels et al.)</a></td><td>Low-rank compression compatible with all-reduce</td></tr><tr><td>2019</td><td><a href=\"https://arxiv.org/abs/1910.00643\" rel=\"nofollow ugc noopener\">SlowMo (Wang et al.)</a></td><td>Momentum applied to periodic averages; a precursor of DiLoCo’s outer optimizer</td></tr><tr><td>2019–20</td><td><a href=\"https://arxiv.org/abs/1909.06335\" rel=\"nofollow ugc noopener\">FedAvgM (Hsu et al.)</a>; <a href=\"https://arxiv.org/abs/2003.00295\" rel=\"nofollow ugc noopener\">FedOpt (Reddi et al.)</a></td><td>Inner/outer optimizer formulation with server-side momentum and adaptive outer steps</td></tr><tr><td>2020</td><td><a href=\"https://arxiv.org/abs/2002.04013\" rel=\"nofollow ugc noopener\">Learning@home (Ryabinin &amp; Gusev)</a></td><td>Decentralized mixture-of-experts training on volunteer hardware</td></tr><tr><td>2021</td><td><a href=\"https://arxiv.org/abs/2103.03239\" rel=\"nofollow ugc noopener\">Moshpit SGD (Ryabinin et al.)</a></td><td>Dynamic averaging groups for unreliable, heterogeneous peers</td></tr><tr><td>2021</td><td><a href=\"https://arxiv.org/abs/2106.10207\" rel=\"nofollow ugc noopener\">DeDLOC (Diskin et al.)</a></td><td>Collaborative pretraining with volunteers, demonstrated on a Bengali ALBERT model</td></tr></tbody></table></div>\n<h3 id=\"large-model-era-2022-2026\">Large-model era, 2022–2026</h3>\n<div class=\"table-wrap\"><table><thead><tr><th>Date</th><th>Work</th><th>Type</th><th>What the evidence adds</th></tr></thead><tbody><tr><td>Jan 2023</td><td><a href=\"https://arxiv.org/abs/2301.11913\" rel=\"nofollow ugc noopener\">SWARM Parallelism</a></td><td>Method, model-parallel</td><td>Pipeline training on preemptible, heterogeneous devices: a transformer with 1B shared parameters on T4 GPUs over slow links.</td></tr><tr><td>Nov 2023</td><td><a href=\"https://arxiv.org/abs/2311.08105\" rel=\"nofollow ugc noopener\">DiLoCo</a></td><td>Method</td><td>AdamW inner optimizer, Nesterov outer optimizer, hundreds of local steps between exchanges, models up to 400M. With 8 workers it matched fully synchronous training while communicating 500× less.</td></tr><tr><td>Jul 2024</td><td><a href=\"https://arxiv.org/abs/2407.07852\" rel=\"nofollow ugc noopener\">OpenDiLoCo</a></td><td>Method + demo</td><td>Open implementation, models up to 1.1B, and a real run across three countries on two continents.</td></tr><tr><td>Aug–Nov 2024</td><td><a href=\"https://github.com/NousResearch/DisTrO/raw/main/A_Preliminary_Report_on_DisTrO.pdf\" rel=\"nofollow ugc noopener\">DisTrO</a> / <a href=\"https://arxiv.org/abs/2411.19870\" rel=\"nofollow ugc noopener\">DeMo</a></td><td>Method</td><td>A separate line that compresses optimizer momentum in a transformed representation, with its own state and aggregation rules.</td></tr><tr><td>Nov 2024</td><td><a href=\"https://arxiv.org/abs/2411.02908\" rel=\"nofollow ugc noopener\">Photon</a></td><td>Method + system</td><td>Federated pretraining up to 7B, examining local optimization and communication trade-offs.</td></tr><tr><td>Dec 2024</td><td><a href=\"https://arxiv.org/abs/2412.01152\" rel=\"nofollow ugc noopener\">INTELLECT-1</a></td><td>Run</td><td>10B model pretrained on 1T tokens across five countries and three continents, with up to 112 H100s active at once.</td></tr><tr><td>Jan 2025</td><td><a href=\"https://arxiv.org/abs/2501.18512\" rel=\"nofollow ugc noopener\">Streaming DiLoCo</a></td><td>Method</td><td>Synchronizes parameter fragments in sequence, overlaps communication with computation and quantizes exchanges. Similar quality with about two orders of magnitude less bandwidth.</td></tr><tr><td>Mar 2025</td><td><a href=\"https://arxiv.org/abs/2503.09799\" rel=\"nofollow ugc noopener\">DiLoCo scaling laws</a></td><td>Method study</td><td>Controlled study of how model size, replica count and hyperparameters affect quality.</td></tr><tr><td>Jun 2025</td><td><a href=\"https://arxiv.org/abs/2506.21263\" rel=\"nofollow ugc noopener\">DiLoCoX</a></td><td>Systems experiment</td><td>Modified Qwen1.5-107B on 160 A800s over an emulated 1 Gbit/s link, 4,000 steps on WikiText-103. Useful at large parameter count, with a measurable quality gap.</td></tr><tr><td>Aug 2025 rev. 2026</td><td><a href=\"https://arxiv.org/abs/2508.15706\" rel=\"nofollow ugc noopener\">SparseLoCo</a></td><td>Method</td><td>Top-k sparsification, error feedback and low-bit quantization of pseudo-gradients.</td></tr><tr><td>Jan 2026</td><td><a href=\"https://pluralis.ai/blog/pluralis-multi-party-training-stack/\" rel=\"nofollow ugc noopener\">Pluralis Node0</a></td><td>Run, model-parallel</td><td>7.5B model trained on 36B tokens over three weeks by 303 active participants; each device held one block of the model.</td></tr><tr><td>Mar 2026</td><td><a href=\"https://arxiv.org/abs/2603.08163\" rel=\"nofollow ugc noopener\">Covenant-72B</a></td><td>Run</td><td>72B model pretrained on about 1.1T tokens with permissionless participation and SparseLoCo.</td></tr><tr><td>Apr 2026</td><td><a href=\"https://arxiv.org/abs/2604.21428\" rel=\"nofollow ugc noopener\">Decoupled DiLoCo</a></td><td>Method + systems</td><td>Asynchronous learners, quorum-based aggregation and heterogeneous hardware. Matches data-parallel quality up to 9B under simulated failures; a real 12B run spread eight learners across four US regions (<a href=\"https://deepmind.google/blog/decoupled-diloco\" rel=\"nofollow ugc noopener\">blog</a>).</td></tr><tr><td>Jun 2026</td><td><a href=\"https://arxiv.org/abs/2606.22768\" rel=\"nofollow ugc noopener\">Factored Gossip DiLoCo</a></td><td>Method</td><td>Replaces part of DiLoCo’s global averaging with pairwise gossip to reduce blocking communication.</td></tr><tr><td>Jul 2026</td><td><a href=\"https://arxiv.org/abs/2607.13332\" rel=\"nofollow ugc noopener\">Agora / Pluralis-8B</a></td><td>Run, model-parallel</td><td>8.6B model pretrained on 500B tokens over 40 days by 330 contributor nodes, mostly consumer GPUs.</td></tr><tr><td>Oct 2026</td><td><a href=\"https://arxiv.org/abs/2610.03457\" rel=\"nofollow ugc noopener\">Cross-facility HPC pretraining</a></td><td>Systems demo</td><td>A 0.6B model trained across three supercomputers on two continents, with an explicitly reported quality cost: perplexity 34.7 versus 28.2 for the centralized baseline.</td></tr></tbody></table></div>\n<h2 id=\"our-case-study-two-clusters-one-model\">Our case study: two clusters, one model</h2>\n<p>Early on, we set ourselves a concrete challenge: could our\ntraining stack turn two separate allocations into one useful\ntraining system? We used 160 B300 GPUs in the US and 160 GB300\nGPUs in Asia, roughly 9,000 miles apart.</p>\n<p>The setup. Each site trains on its own data shard with local\nexpert parallelism. Microbatch and gradient-accumulation\nsettings are tuned separately so both sites take similar time\nper step.</p>\n<p>The model was a conventional sparse mixture-of-experts (MoE) with\nabout 32B total parameters and an MoE layer in every transformer\nblock: 256 experts, top-8 routing and no shared experts.</p>\n<figure><img src=\"https://www.coreauto.com/assets/blog/compute-islands/training-results.png\" alt=\"Bits per byte against training compute on two evals. At equal compute, multi-cluster training is about 1% lower: 0.1995 versus 0.2017 on systems and kernels, 0.2525 versus 0.2549 on general coding.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>The star is our two-site run across the B300 and GB300 clusters,\nat the same compute as the largest standard run.</p>\n<p>We made many improvements. A few notable ones:</p>\n<ul><li><p><strong>Synchronization.</strong> Rather than syncing the whole</p><p>model at once, we synced it fragment by fragment, staggered\nacross each 50-step window. We borrowed fragment-wise,\noverlapped and asynchronous updates from\n<a href=\"https://arxiv.org/abs/2501.18512\" rel=\"nofollow ugc noopener\">Streaming</a> and\n<a href=\"https://arxiv.org/abs/2604.21428\" rel=\"nofollow ugc noopener\">Decoupled DiLoCo</a>.\nSites exchanged updates on a 50-step schedule, and a\nlate-arriving update can still be committed.</p></li><li><p><strong>Local optimizer.</strong> Each site used a variant of</p><p>Distributed Shampoo.</p></li><li><p><strong>Communication.</strong> Exchanged values were compressed</p><p>with stochastic rounding plus error feedback.</p></li><li><p><strong>MoE routing.</strong> The small routing statistics were</p><p>coordinated and synchronized separately, at every step.</p></li></ul>\n<h3 id=\"engineering-for-tail-latency\">Engineering for tail latency</h3>\n<p>We rewrote the networking layer to focus on completion time of\neach update, and make it predictable. Thus, we optimized directly\nfor tail latency, which involved implementing several strategies.\nEach of them was a\n<a href=\"https://research.google/pubs/the-tail-at-scale/\" rel=\"nofollow ugc noopener\">simple idea</a>, but the gains came from putting them together.</p>\n<p>i.</p>\n<p>Spread demand over time</p>\n<p>We staggered parameter updates over time, paced total\ntraffic and capped per-socket send rates, so workers that\nfinished together didn’t burst together. Control messages\nfor coordination and repair ran independently, so they were\nnever blocked.</p>\n<p>ii.</p>\n<p>Keep connections warm and verified</p>\n<p>We reused warm connections, watched them with heartbeats and\nreconnected with jitter. We also preallocated most of the\nmemory the transfers needed.</p>\n<p>iii.</p>\n<p>Chunk everything</p>\n<p>Payloads were split into independently scheduled chunks that\nany healthy connection could claim as soon as it was free,\nso slow links couldn’t hold an update back.</p>\n<p>iv.</p>\n<p>Repair only what’s missing</p>\n<p>When a write stalled, we re-requested only the missing\npieces and allowed a bounded number of duplicate sends. We\nalso launched redundant transfers when a deadline hit, which\ncut tail latency.</p>\n<p>At the same compute, the two-site run reached ≈0.2525 bits per\nbyte vs 0.2549 for standard training. Next we asked whether\npairwise exchanges could replace the all-reduce.</p>\n<h2 id=\"revisiting-butterfly-mixing\">Revisiting butterfly mixing</h2>\n<p>To find out, we split 64 GB300 GPUs at one site into eight\nindependent cells, each an eight-GPU learner training its own copy\nof the 32B-parameter MoE. Instead of averaging updates across all\neight cells at once, cells exchanged model fragments in pairs and\nkept training in between. After three rounds, every cell’s update\nhas reached all eight, with no all-reduce. The approach dates back\nto Zhao and Canny’s\n<a href=\"https://doi.org/10.1137/1.9781611972832.87\" rel=\"nofollow ugc noopener\"><em>butterfly mixing</em></a>. Training quality matched the baseline.</p>\n<figure><img src=\"https://www.coreauto.com/assets/blog/compute-islands/butterfly-mixing.gif\" alt=\"Animation: eight sites around the globe swap updates in three butterfly stages over fiber, subsea and orbital links. Slow links deliver later; no site waits.\" loading=\"lazy\" decoding=\"async\" referrerpolicy=\"no-referrer\" /></figure>\n<p>Butterfly mixing, imagined across eight very different sites and\nlink speeds. Slow links deliver later and nobody waits for them.\nIn our experiment, all eight cells ran at one site.</p>\n<p>The schedule for one fragment, unrolled. Follow cell 0’s update\n(blue) as it reaches 2, then 4, then all 8 cells, without any\nall-reduce. Hover a stage to colour the groups that have mixed\nby then, or hover a starting cell to trace its update.</p>\n<p>In practice, pairings differed by fragment, and each exchange was\nasynchronous from the cell’s point of view:</p>\n<pre><code># pairings differ per fragment\nfragment A:  0–5  1–3  2–6  4–7\nfragment B:  0–1  2–4  3–5  6–7\n\n# one exchange, from one cell&#39;s view\nstep t     build candidate from fragment, send to partner\nstep t+1   keep training\nstep t+2   live ← 0.5·live + 0.5·pair_result</code></pre>\n<p>The model’s parameters were divided into fragments, and each\nfragment was exchanged six times every 50 steps (two full\nbutterflies), or about once every eight steps. At each exchange, a\ncell built a fresh outer-update candidate from its local optimizer\nstate, sent it to its partner and kept training. Two steps later,\nit blended the pair’s result into its training weights at equal\nweight.</p>\n<p>Varying the pairing for each fragment also spread traffic across\npeers and over time: each cell traded different parts of the model\nwith different partners at the same time.</p>\n<h2 id=\"training-in-orbit\">Training in orbit</h2>\n<p>The boldest version of this idea is already being tried. Google\nannounced\n<a href=\"https://blog.google/technology/research/google-project-suncatcher/\" rel=\"nofollow ugc noopener\">Project Suncatcher</a>\nin November 2025, a research program exploring constellations of\nsolar-powered TPU satellites linked by optical connections (<a href=\"https://research.google/blog/exploring-a-space-based-scalable-ai-infrastructure-system-design/\" rel=\"nofollow ugc noopener\">system design</a>). On October 30, 2026, a month after its first prototype\nlaunched, Google\n<a href=\"https://blog.google/innovation-and-ai/models-and-research/google-research/project-suncatcher-satellite-response/\" rel=\"nofollow ugc noopener\">reported</a>\nthe first AI response from a data-center-class TPU in orbit: a\nhaiku from an onboard Gemma 4 model.</p>\n<p>That is inference on a single satellite. Training across many is a\nharder problem. Orbit is the extreme version of a compute island:\nlinks to the ground come and go, heat can only be radiated away,\nand radiation makes hardware fail more often. Those are the same\nproblems our approach is meant to handle: learn locally through\ninterruptions, and combine progress when a connection returns.</p>\n<p>Three orbital islands. Blue: fast optical links inside a\nformation. Dashed: intermittent links between formations and\ndown to a ground station.</p>\n<p>An ambitious first step would be to pretrain a small model across\na few satellites and compare it with a ground baseline. A\ntrillion-parameter orbital run is much further off. Science\nfiction has long imagined the von Neumann probe, a spacecraft that\ncopies itself and spreads across the galaxy; ours is a humbler\nidea, a model that keeps learning across machines that are far\napart and only sometimes connected. So we aim for the stars: if\ntraining works in orbit, it should work on any cluster on the\nground.</p>\n<h2 id=\"what-this-means-for-the-industry\">What this means for the industry</h2>\n<p>As training reliably spans multiple smaller clusters, much more of\nthe world’s compute becomes usable for frontier training, capacity\nthat would otherwise go to less valuable computation. Training\nruns can move fungibly across clusters, surviving outages and\ndelays that hit one region. A frontier run could even live on\nlow-priority, preemptible capacity, taking compute when it frees\nup and giving it back without losing progress, which makes it\nfault tolerant by design. And more research teams can attempt\nambitious work and scale up without first securing a gigawatt\ncampus.</p>\n<p>Today, the main barrier to frontier-scale training is assembling a\nsingle colocated GPU cluster, which only a few organizations can\ndo. The lines of work we explore here widen the set of actors who\ncan train highly capable models.</p>\n<p>Approaches in this direction can widen the usable training pool\nfor frontier training. Each island only runs on its own schedule,\nso a single run can mix accelerator generations and vendors, with\ndifferent memory sizes, interconnects and parallelism layouts, as\nour own run already mixed GB300 and B300 clusters. Older\ngenerations or less common chips could also participate in a\nfrontier run.</p>\n<p>It’s also possible that integrity and safety get harder too when\nmodels are trained across providers: one faulty or compromised\nsite can quietly degrade or tamper with the model, in ways that\nare harder to trace.</p>\n<p>If you have compute islands and want to make them useful for\nfrontier training runs, or you’re someone who wants to work with a\ncrew pushing the boundaries of learning methods, write to\n<a href=\"mailto:hello@coreauto.com\">hello@coreauto.com</a>. We’re\nhiring across research, systems and lab automation.</p>\n<h2 id=\"notes\">Notes</h2>\n<ol><li><p>Sources for the compute tiers:</p><p> <a href=\"https://www.businesswire.com/news/home/20251110221705/en\" rel=\"nofollow ugc noopener\">CoreWeave Q3 2025 results</a>;\n <a href=\"https://nebius.com/newsroom/nebius-announces-multi-billion-dollar-agreement-with-microsoft-for-ai-infrastructure\" rel=\"nofollow ugc noopener\">Nebius–Microsoft agreement</a>\n (<a href=\"https://www.bloomberg.com/news/articles/2025-09-08/microsoft-signs-nebius-cloud-deal-for-as-much-as-19-4-billion\" rel=\"nofollow ugc noopener\">Bloomberg</a>\n on the $17.4 billion and $19.4 billion figures);\n <a href=\"https://nebius.com/newsroom/nebius-reports-third-quarter-2025-financial-results\" rel=\"nofollow ugc noopener\">Nebius Q3 2025 results</a>\n (Meta);\n <a href=\"https://lambda.ai/blog/introducing-lambda-1-click-clusters-a-new-way-to-train-large-ai-models\" rel=\"nofollow ugc noopener\">Lambda 1-Click Clusters</a>;\n <a href=\"https://www.together.ai/blog/together-instant-clusters-ga\" rel=\"nofollow ugc noopener\">Together AI Instant Clusters</a>;\n <a href=\"https://aws.amazon.com/ec2/capacityblocks/\" rel=\"nofollow ugc noopener\">AWS Capacity Blocks</a>; <a href=\"https://www.sfcompute.com\" rel=\"nofollow ugc noopener\">SF Compute</a>;\n <a href=\"https://www.hpcwire.com/2026/10/08/national-compute-grid-launches-to-pool-idle-ai-capacity-across-providers/\" rel=\"nofollow ugc noopener\">National Compute Grid</a>\n (HPCwire). <a href=\"#ref-1\">↩</a></p></li></ol>\n<h2 id=\"bibtex\">BibTeX</h2>\n<pre><code>@article{anil2026computeislands,\n  author  = {Rohan Anil},\n  title   = {The Hitchhiker&#39;s Guide to Training on Compute Islands},\n  journal = {Core Automation Blog},\n  year    = {2026},\n  month   = {November}\n}</code></pre>","headings":[{"level":2,"text":"Stranded, or just inconvenient?","id":"stranded-or-just-inconvenient"},{"level":2,"text":"Compute comes in many shapes","id":"compute-comes-in-many-shapes"},{"level":2,"text":"A brief history of low-communication training","id":"a-brief-history-of-low-communication-training"},{"level":3,"text":"Foundations, before 2022","id":"foundations-before-2022"},{"level":3,"text":"Large-model era, 2022–2026","id":"large-model-era-2022-2026"},{"level":2,"text":"Our case study: two clusters, one model","id":"our-case-study-two-clusters-one-model"},{"level":3,"text":"Engineering for tail latency","id":"engineering-for-tail-latency"},{"level":2,"text":"Revisiting butterfly mixing","id":"revisiting-butterfly-mixing"},{"level":2,"text":"Training in orbit","id":"training-in-orbit"},{"level":2,"text":"What this means for the industry","id":"what-this-means-for-the-industry"},{"level":2,"text":"Notes","id":"notes"},{"level":2,"text":"BibTeX","id":"bibtex"}]}}