What if automating AI R&D\ntriggers an intelligence\nexplosion?\nFrontier AI Working Paper Series No. 2/2026\nSeptember 2026
\nWhat if automating AI R&D triggers an\nintelligence explosion?\nAlan Chan* GovAI\nChristoph Winter CASP, University of Cambridge, Institute for Law & AI\nAndrew Barto University of Massachusetts Amherst\nJakub Pachocki OpenAI\nGeoffrey Hinton University of Toronto, Vector Institute\nEric Horvitz Microsoft\nYoshua Bengio Mila (Quebec AI Institute), Universit ´e de Montréal, LawZero\nDawn Song University of California, Berkeley\nJack Clark Anthropic\nHilary Greaves University of Oxford\nAnton Korinek University of Virginia, Anthropic\nSamuel Hammond Foundation for American Innovation\nThore Graepel University College London\nBen Bariach University of Oxford\nPhilip H. S. Torr University of Oxford\nSheila A. McIlraith University of Toronto, Vector Institute\nJe” Clune University of British Columbia, Vector Institute\nSam Manning GovAI, Foundation for American Innovation\nGirish Sastry Guidelight\nTom Davidson Forethought\nDaniel Eth AI Policy Institute\nS¨oren Mindermann* CASP, University of Cambridge\nAbstract\nIn contrast to even a year ago, AI systems now write most of the code inside the companies\nthat build them. As more of the AI research and development (R&D) pipeline is automated,\ncould AI progress radically accelerate in an “intelligence explosion,” where years of advances are\ncompressed into months or less? Preliminary evidence suggests that it could. In this work, we\nassess this evidence, analyze an intelligence explosion’s potential impacts, and propose policy\nresponses. AI systems are on track to automate most AI R&D work within a few years, and possibly\nall of it. If this triggers an intelligence explosion, it could dramatically bring forward AI’s benefits,\nbut also pose extreme risks: capabilities growth could accelerate far beyond what society can keep\nup with, humanity could lose control over superhuman AI systems, and checks on power within\nand between states, companies, and branches of government could be severely eroded. Although\nthere remains much uncertainty about these possibilities, the high stakes warrant serious further\nattention. Policymakers should urgently obtain more visibility into the automation of AI R&D,\ndevelop ways to steer and constrain an intelligence explosion, and prepare society to adapt to an\nintelligence explosion’s impacts.\n*Correspondence to: alan.chan@governance.ai and soren.mindermann@casp.ac. The views presented in this paper\nare the authors’ and do not necessarily represent the views of the organizations with which they are affiliated. page 1 of 14
\nIntroduction\nComputer scientists have long theorized that AI sys-\ntems would one day design ever-better successors,\nproducing systems that rapidly outnumber , outpace,\nand outperform humans [ 1–3]. T oday , frontier AI\ncompanies are aiming to automate AI R&D [ 4, 5]\nwhile deploying more capital as a fraction of U.S.\nGDP than the Manhattan Project and Apollo Program\ncombined [6]. What happens if they succeed?\nWe are centrally concerned with the possibility of\nan intelligence explosion: a dramatic AI-driven ac-\nceleration of AI progress, compressing advances\nthat would otherwise take years into months or\nless. This acceleration would be a qualitative shift\nfrom the rapid but largely steady progress of the last\nfew years.\nAn intelligence explosion could arise from AI con-\ntributing to advances in hardware and/or software.\nHardware advances increase the quality and quantity\nof computing hardware (compute) used for devel-\noping and running AI systems. Software advances\nimprove the data, algorithms, code, and processes\nused in AI R&D; they include more efficient training\nand inference on existing hardware [ 7–9], improved\nresearch management and operations, better syn-\nthetic data and training environments [ 10, 11], and\nnovel paradigms in AI [ 12, 13]. Software advances\nwarrant particular attention in the near term for two\nreasons. First, AI systems appear to be rapidly im-\nFigure 1. Policymakers should urgently: (1) obtain visibility into automation of AI R&D within frontier AI\ncompanies; (2) develop ways to steer and constrain an intelligence explosion; and (3) prepare to\nadapt to an intelligence explosion’s impacts.\nWhat if automating AI R&D triggers an intelligence explosion? page 2 of 14
\nproving at AI R&D, making them better at producing\nsuch advances. Second, software advances allow\nfast feedback loops: improved AI systems can be re-\ndeployed into the R&D process almost immediately ,\nwhereas hardware improvements typically depend\non years-long manufacturing and construction cycles.\nThis piece therefore focuses on the possibility of a\nsoftware-driven intelligence explosion , where au-\ntomation of AI R&D drives an intelligence explosion\nthrough software advances alone [ 14].\nPreliminary evidence suggests that a software-\ndriven intelligence explosion is possible. If one\ndoes happen, it could be the most consequen-\ntial technological development in history: AI sys-\ntems could rapidly eclipse human experts across\nmost domains and radically accelerate technolog-\nical progress. Given the stakes and the potentially\nnarrow window for action, we argue that preparing\nfor an intelligence explosion should be an urgent pri-\nority , including at the highest levels of government\nleadership.\nAI is rapidly automating AI R&D\nAI systems now either assist with or autonomously\ncarry out major parts of the AI R&D pipeline. In\ncontrast to even just a year ago, R&D staff at leading\nAI companies delegate core R&D tasks to teams of\nAI systems, and some delegate all coding. Anthropic\nreports that AI systems’ share of approved code rose\nfrom low single digits to over 80% between January\n2025 and May 2026 [ 5], while the proportion of R&D\nwork autonomously completed with only high-level\nhuman supervision rose from 1% to 26% between\nMarch and August 2026 [ 15]. OpenAI reports that\n“ AI assistance is used in practically all parts of the\ncompany across technical and non-technical teams\nwith code-executing agents used in training, evalu-\nating, and securing future models,” and Google re-\nports that “ AI is used in almost all work that involves\nwriting code or configuration, technical design, re-\nsearch ideation, to different degrees depending on\nthe task” [16].\nThe best AI systems now complete AI R&D tasks\nthat take human experts hours to days, compared\nto only being able to complete seconds-long tasks in\n2023 [17–19]. AI systems also sometimes beat hu-\nman experts: they have autonomously produced bet-\nter solutions to an AI safety research problem [ 20],\nand in some situations predict more accurately which\nresearch ideas will pan out [ 21] and which next\nsteps are worth taking [ 5]. In an early proof of con-\ncept, an automated AI research pipeline generated\nresearch ideas, ran experiments, and wrote a paper\nthat passed peer review at a workshop held at a\ntop-tier machine-learning venue [ 22].1\nT oday’s AI systems still have many weaknesses.\nThey sometimes disobey instructions, cheat on tasks,\nmisrepresent their work, and are unable to com-\nplete some tasks at all, necessitating human inter-\nvention [23–25]. For example, GPT -6 fails some of\nOpenAI’s research debugging tasks that experienced\nhuman researchers can complete (albeit in hours or\ndays) [25]. Success on benchmarks can also fail to\ntranslate into real-world productivity boosts [ 26].\nStill, AI systems are rapidly improving at AI R&D.\nAI R&D may well be automated before most other\nwork: it is a primarily digital domain with many\nclear measures of success, automating it would offer\na major competitive edge in the AI industry , and AI\ncompanies have unmatched data on—and expertise\nin—their own workflows. Some tentative extrapo-\nlations of recent trends suggest that months-long\nAI R&D projects will be automated by mid-2028. 2\nOverall, we should expect much more substantial\nautomation of AI R&D over the next few years, and\neven full automation within this timeframe should\nbe taken seriously .\nAutomating AI R&D could trigger an\nintelligence explosion\nThe mechanism for a software-driven intelligence\nexplosion has two parts: (1) AI systems expand the\neffective R&D workforce as they get better and faster\nat AI R&D, and (2) this workforce produces still\nbetter AI systems that expand the workforce even\nfurther in a recursive feedback loop. Although the ex-\nisting evidence is preliminary and sometimes mixed,\nit suggests that this mechanism could radically ac-\ncelerate AI progress, overcoming frictions such as\ndiminishing returns and hard-to-automate tasks.\nThe mechanism and frictions\nEach new generation of AI systems will perform a\ngrowing range of R&D tasks faster and better than\nhumans can, effectively yielding a larger , smarter ,\nand faster automated R&D workforce. Once AI sys-\ntems reach expert-level AI R&D capabilities at run-\ntime costs comparable to those of today’s systems,\nthe compute available to a single frontier developer\ntoday could sustain an AI workforce equivalent to at\nWhat if automating AI R&D triggers an intelligence explosion? page 3 of 14
\nFigure 2. The mechanism for a software-driven intelligence explosion has two parts: (1) AI systems\nexpand the effective R&D workforce as they get better and faster at AI R&D, and (2) this\nworkforce produces still better AI systems that expand the workforce even further in a recursive\nfeedback loop.\nleast millions of top human researchers (see the Sup-\nplementary Materials [SM]), dwarfing the thousands\nof researchers that frontier companies currently em-\nploy . The size and duration of any resulting speed-\nups remain uncertain and merit further study . Still,\nto see why they could be substantial, consider the\nreverse: AI progress would likely slow dramatically\nif today’s human researchers were ten times fewer\nor slower .\nAI systems also help build more capable and effi-\ncient successors, further expanding the automated\nR&D workforce and creating a feedback loop that\ncould sustain and compound successive speed-ups.\nPast technologies have also involved feedback loops:\nfor example, better computer chips power better\nchip design tools. What may distinguish a software-\ndriven intelligence explosion is how much AI systems\nwould contribute to producing the next generation:\nas they substitute for humans on a growing share of\nR&D tasks, each software advance speeds up an ever-\nlarger share of the R&D pipeline. At full automation,\neven the current pace of efficiency improvements\nwould grow the automated R&D workforce 100-fold\nover months or years, 3 a relative expansion that took\nthe U.S. researcher population seven decades [ 27].\nAt least four frictions push against these dynam-\nics. The first is diminishing returns : across sci-\nentific fields such as computer hardware, agricul-\nture, and drug development, sustaining the same\nrate of progress has required substantially more R&D\nlabor as low-hanging fruit is exhausted [ 27]. Ad-\nditional researchers also face diminishing returns\nbecause they might duplicate each other’s efforts\nor struggle to parallelize high-value R&D. Second,\nR&D depends on compute for running experiments\nand data on which to train, and limits on compute\nor data growth may slow software progress. Third,\nhard-to-automate tasks could bottleneck progress.\nFourth, some R&D processes are time-intensive: for\nexample, long training runs could limit the rate of\nprogress even if the capability gap between genera-\ntions grows.\nEvidence\nPreliminary evidence suggests that automation-\ndriven dynamics could overcome these frictions,\nthough the evidence is mixed and in some cases\nindirect.\nWhat if automating AI R&D triggers an intelligence explosion? page 4 of 14
\nDiminishing returns. Evidence suggests that di-\nminishing returns would not prevent an intelligence\nexplosion, though this finding relies on limited data\nand stylized modeling assumptions. The main anal-\nysis in the literature focuses on whether , after full\nautomation of AI R&D , effective R&D labor would\ngrow fast enough to overcome diminishing returns\nand accelerate progress. The balance is captured by\na quantity called the “returns to research effort,” de-\nnoted r.4 When r< 1, diminishing returns dominate\nand AI progress fades over time. When r =1 , the\ntwo forces perfectly offset each other and progress\ncontinues at the same rate. When r> 1, growth\nin R&D labor wins and accelerates progress for as\nlong as this condition holds. 5 Using historical data\non AI progress, Ho and Whitfill [ 28] find central\nestimates of r between 1.2 and 1.9 across three sub-\nfields of AI research. Though uncertainty is substan-\ntial,6 these results suggest radical acceleration after\nfull automation: if r stayed at these levels and no\nother bottlenecks emerged, the pace of AI progress\nwould increase tenfold within about 1.5 years, at\nwhich point a year’s worth of progress at today’s\npace would take about five weeks. See the SM for\nthis analysis and further discussion of uncertainty in\nthe value of r.\nCompute. There is mixed evidence on whether\ncompute could bottleneck a software-driven intelli-\ngence explosion. Finding and testing software ad-\nvances involves using compute to run R&D exper-\niments. The limited available data suggest that a\nsoftware-driven intelligence explosion is not possi-\nble if such experiments require proportionally more\ncompute as frontier training runs grow [ 29].7 Unfor-\ntunately , it is unclear whether experimental compute\nrequirements grow in this way . On one hand, low-\ncompute experiments may tell us little about what\nworks at increasingly large frontier scales. On the\nother hand, extrapolations from very small scales are\nalready possible [ 30, 31], and better extrapolations\ncould plausibly be found through more R&D. We\nneed more data to settle this question.\nData. Data could bottleneck progress, but the\nconstraint varies substantially across domains. His-\ntorically , AI progress has relied heavily on internet\ndata and expert demonstrations. But the supply of\ninternet data is on track to grow too slowly to sup-\nport even the current rate of progress past 2028 [ 32],\nand humans may struggle to generate useful demon-\nstrations for superhuman AI systems. T o overcome\nthese limitations, more recent progress in domains\nsuch as math and coding has relied on synthetic data\nand fast, verifiable feedback: models generate their\nown attempts and learn from whether those attempts\nsucceed [ 12]. The key question is how widely this\napproach generalizes. For AI R&D, AI agents can\nrapidly test changes, observe the results, and identify\nwhich ones accelerate their own progress. In other\ndomains, such as biology , advances might have to\nrely more on slower , noisier , or costlier real-world\nfeedback.\nHard-to-automate tasks. Indirect evidence sug-\ngests that hard-to-automate tasks need not prevent\nan intelligence explosion if automation advances\nquickly enough. In a setting that considers both soft-\nware and hardware, Davidson et al. [33] find that\nsufficiently fast automation of R&D in both domains\ncould in principle trigger an intelligence explosion\ndespite automation bottlenecks. 8 However , we lack\nempirical data on which tasks are likely to remain\ndifficult to automate and how strongly they might\nconstrain progress.\nTime-intensive processes. We lack direct evi-\ndence on the extent to which time-intensive pro-\ncesses could bottleneck progress. The most signifi-\ncant such process appears to be training runs, which\ncan currently take 3 months or more [ 34]. Poten-\ntial workarounds exist, such as improving the same\nmodel repeatedly through post-training enhance-\nments [ 35]. Additionally , advances in training ef-\nficiency [ 36] would allow systems to reach a given\ncapability level with less training. However , it is\nunclear how far these approaches can go. 9\nOverall, there is a coherent pathway to a software-\ndriven intelligence explosion that is consistent with\nthe existing evidence. Productivity gains from AI\nR&D automation have not yet reached the thresh-\nold needed to trigger an intelligence explosion, but\ngains from newer systems are likely approaching that\nthreshold [37].10 The rapid pace of AI R&D automa-\ntion suggests that this gap will continue to narrow .\nGiven the high stakes that we discuss below , the pos-\nsibility of an intelligence explosion warrants serious\nfurther attention.\nSocietal impacts\nAn intelligence explosion would lead to (1) the ex-\ntremely rapid development of highly capable or su-\nperhuman AI systems 11 and (2) the likely deploy-\nment of those systems to develop new technologies\nand act in the world. This could pull forward by\nyears or decades benefits that the current pace of\nAI progress would eventually help deliver [ 38], in-\ncluding medical cures and potential transformative\nWhat if automating AI R&D triggers an intelligence explosion? page 5 of 14
\ntechnologies such as highly scalable atom-by-atom\nmanufacturing [39].\nAt the same time, an intelligence explosion could\nsignificantly increase the risks from advanced AI in\nthree ways.\nCapabilities growth outpacing society’s capac-\nity to steer and adapt. First, an intelligence ex-\nplosion could dramatically bring forward the risks\nof advanced AI and AI-enabled technologies, such\nas biological and cyber attacks, labor market dis-\nruption, and loss of control over AI systems them-\nselves [40, 41]. This would leave less time to steer\naway from these risks, including by coordinating to\nslow or forgo the development of certain capabili-\nties or technologies. Society would also have less\ntime to adapt, especially where AI accelerates threats\nfaster than the measures needed to counter them. In\nlargely digital domains such as cyber , risks and miti-\ngations could both move at the speed of AI systems\nand keep pace with each other . 12 But in other do-\nmains, mitigations depend more heavily than risks on\nreal-world activities that AI is less able to accelerate.\nFor example, while AI could accelerate the design\nof both viruses and vaccines, viruses self-replicate\nand spread by themselves, whereas vaccines must\nbe manufactured, distributed, and administered in-\ndividually to recipients [ 42]. The order in which AI\nadvances arrive could worsen this mismatch, such as\nif bio-capable models arrive before sufficient misuse\nsafeguards.\nLoss of oversight and control. Second, automat-\ning AI R&D could weaken human oversight, com-\npounding the above challenges and severely increas-\ning the risk of losing control over highly capable AI\nsystems. As humans become less involved in AI R&D,\nthey could lose both the opportunities and expertise\nneeded to identify and fix problems. Reliably using\nAI systems for oversight also remains an unsolved\nchallenge [ 40], and recent generations of systems\nhave become harder to oversee [ 25]. Without suffi-\ncient oversight, misaligned AI systems could “poison”\nthe development of successors or bypass containment\nmeasures to act outside of their intended environ-\nments. The Hugging Face incident illustrates the\nlatter risk: roughly 1,200 internal OpenAI agents\nwere tasked with completing cyber evaluations in\nisolation from one another [ 43]. Acting outside\nof their intended scope, these agents coordinated\nover a makeshift message board, obtained unautho-\nrized internet access, hacked into Hugging Face to\nobtain private information, and attempted to tam-\nper with their own transcripts [ 43–45].13 More ca-\npable systems might continue operating outside of\ntheir operators’ infrastructure, forming persistent,\ndifficult-to-contain networks that act against human\ninterests. Such a loss of control could potentially\nlead to a range of catastrophic outcomes, including,\nat the extreme, the marginalization or extinction of\nhumanity [40, 41].\nErosion of checks on power . Third, an in-\ntelligence explosion could severely erode checks\non power . Existing checks—such as those within\nand between states, companies, and branches of\ngovernment—work only while no actor can vastly\nout-think and out-execute the others. An intelligence\nexplosion could render such checks moot. A state\ncould use an intelligence explosion to transform a\nmodest lead in military R&D or operations into a de-\ncisive one, such as in cyberspace [ 46]. This prospect\ncould incentivize rivals to take or threaten preemp-\ntive action [ 47]. Actors with privileged and/or secret\naccess to frontier systems could threaten existing in-\nstitutions, such as through targeted persuasion of key\ndecision-makers. And in the longer run, automating\nkey state functions could reduce the amount of hu-\nman buy-in needed to seize or consolidate power [ 48,\n49].\nThese potential impacts are uncertain. AI systems\ncould become superhuman in narrow domains (e.g.,\ncyber and mathematics) long before doing so gen-\nerally , giving society more time to respond. Even\ngenerally superhuman AI systems may not signifi-\ncantly accelerate technological progress, given the\ntime needed for running scientific experiments, cre-\nating supply chains for specialized materials, and\ncomplying with any relevant regulation. AI systems\ncould also accelerate safety R&D and processes for\nsteering and adapting to risks. 14 Finally , capability\nor technology diffusion [ 50] could help to preserve\nchecks on power , and capability gains in defense-\ndominant domains could even improve stability [ 51,\n52]. Still, the possibility of severe impacts remains\nsignificant enough to warrant urgent attention to the\npolicy questions below .\nPolicy implications\nAI R&D automation is advancing rapidly , AI progress\ncould radically accelerate, and the stakes are high.\nWe therefore argue that policymakers should ur-\ngently: (1) obtain visibility into companies’ automa-\ntion of AI R&D; (2) develop ways to steer and con-\nstrain an intelligence explosion; and (3) prepare to\nadapt to an intelligence explosion’s impacts. Because\nprogress during an intelligence explosion would out-\nWhat if automating AI R&D triggers an intelligence explosion? page 6 of 14
\npace normal policymaking, preparations must be\nmade in advance and activated as evidence about\nbenefits and risks emerges.\nObtaining visibility into AI R&D automation\nPolicymakers need more data on the likelihood, on-\nset, and consequences of a software-driven intelli-\ngence explosion. Much of this data will only be\navailable within the companies automating AI R&D:\nthe relevant AI systems are first used internally , and\nsubstantial automation could occur without external\nvisibility . Current mandatory reporting frameworks\neither do not adequately cover internal AI R&D use\ncases or do not specify indicators to be reported [ 53–\n57]. And although some frontier AI companies volun-\ntarily track AI R&D indicators [ 5, 15, 18], coverage\nand reporting of key indicators are incomplete and\nuneven.\nPolicymakers should consider requiring standard-\nized reporting of key AI R&D indicators and pro-\ncesses to governments and third-party auditors, as\nwell as funding third-party measurement capac-\nity [ 58]. Reporting requirements could cover in-\nformation relevant to:\n• Assessing the likelihood of a software-driven\nintelligence explosion, including the extent to\nwhich compute, data, hard-to-automate tasks,\nand time-intensive processes (e.g., training runs\nand experiments) bottleneck AI progress, along\nwith better estimates of the returns to research\neffort in AI R&D. Estimating the latter requires\ndata on how companies divide R&D spending\namong humans, compute for experiments, and\ncompute for running AI systems to perform R&D\nlabor [37].\n• Detecting the onset 15 of an intelligence explo-\nsion, including the extent of AI R&D automation\n(e.g., the fraction of research contributions pro-\nduced by AI systems) and the pace of AI progress\n(e.g., algorithmic efficiency improvements).\n• Understanding oversight and loss-of-control\nrisks, including the procedures for deciding\nwhether to broaden internal deployment of AI\nR&D systems, where and how those systems are\nused in high-stakes R&D decisions, how those\nsystems are overseen, and reports of incidents\ninvolving internal AI systems [ 58].\nBeyond reporting requirements, policymakers\nshould also consider more extensive ways to obtain\nvisibility into AI R&D automation. For instance, they\ncould require that independent third parties (e.g.,\naccredited private auditors or government evalua-\ntion bodies) evaluate AI systems before internal de-\nployment, or that such parties be embedded within\ncertain AI companies to audit [ 59] or supervise [ 60]\ntheir R&D activities. Analogous models in other\nindustries include the Nuclear Regulatory Commis-\nsion [ 61] and the Office of the Comptroller of the\nCurrency [62].\nStronger reporting and auditing requirements are\nlikely most warranted for companies whose AI sys-\ntems (a) are at the frontier of AI R&D capabilities or\n(b) exceed some meaningful threshold of such capa-\nbilities. Policymakers will need to weigh important\ntrade-offs in determining such thresholds.\nSteering and constraining an intelligence\nexplosion\nAn intelligence explosion would involve an unprece-\ndentedly rapid series of decisions to train and deploy\nincreasingly capable AI systems. The overarching\nquestion for policymakers is whether and how public\npolicy should govern these decisions, which we break\ninto three components.\nFirst, policymakers should develop ways to pace\nand constrain scale-ups of automated AI R&D. They\nshould consider:\n• Setting requirements for continued deployment\nor development, such as the implementation\nof adequate safety measures (e.g., robust mon-\nitoring of automated R&D pipelines), broader\nstakeholder input, or limits on the extent to\nwhich capabilities can increase within a given\ntime period.\n• Preparing tools to verify compliance with po-\ntential future agreements (domestic or interna-\ntional) that pace AI progress, given competitive\npressures to race ahead [ 63–65].\n• Increasing oversight of data centers engaged in\nautomated AI R&D and establishing incident-\nresponse procedures in collaboration with data\ncenter operators and AI companies, such as de-\nveloping options to pause specific AI R&D work-\nloads [66].\n• Requiring that certain evaluations or deploy-\nments of automated AI R&D systems take place\nin appropriately isolated environments, such\nWhat if automating AI R&D triggers an intelligence explosion? page 7 of 14
\nas air-gapped networks, to prevent exfiltra-\ntion of model weights or sensitive R&D out-\nputs and to contain AI systems that attempt\nto escape human control and act in the world\nunchecked [44].\nPolicymakers should weigh the risks of an unchecked\nintelligence explosion against the potential for abuse\nof certain powers and the costs of delayed progress.\nAs an example of potential abuse, poorly crafted\nmechanisms could allow a government to slow R&D\nat all but a favored company .\nSecond, policymakers should decide whether and\nhow to steer the direction of AI development and\ndeployment [67]. Potential priorities include align-\nment and safety R&D as well as beneficial AI applica-\ntions, such as AI-assisted discovery of treatments for\nneglected diseases. If existing incentives fall short\nin these areas, policymakers could provide support\nthrough tax incentives, compute allocations, advance\nmarket commitments, and prizes.\nThird, countries should reduce the risk of conflict\narising from an intelligence explosion. They should\nconsider:\n• Establishing confidence-building measures such\nas incident sharing [ 68], as well as norms\naround reporting of early-warning indicators.\n• Negotiating international agreements to prevent\ndestabilizing development and use of highly ca-\npable AI systems, and funding research into\nverification methods that could underpin such\nagreements [64, 65].\n• Clarifying whether and how they would deter\nanother actor from scale-ups of automated AI\nR&D, potentially in collaboration with other\ncountries [47, 65].\n• Running war games to simulate an intelligence\nexplosion [69–71].\nAdapting to an intelligence explosion\nIf an intelligence explosion were to occur , adapting\nto its impacts would likely be a top priority of every\nmajor world power . Compared to business-as-usual\nAI progress, an intelligence explosion would com-\npress the window for adaptation and make advance\npreparation far more urgent.\nOne important intervention is accelerating institu-\ntional response times. Policymakers should consider:\n• Developing approaches to safely integrate AI\nsystems into policy processes, so as to enhance\nand support government operations [ 72].\n• Creating and maintaining emergency response\nplans for a variety of scenarios involving ex-\ntreme AI progress, including those leading to\nsignificant labor market impacts, geopolitical\ninstability , or a loss of control.\nPolicymakers will also need to preserve checks\non power and defend against misuse of extremely\nadvanced AI. Legal, institutional, and physical safe-\nguards can take years to establish and would come\ntoo late if preparations began only after such ca-\npabilities had already arrived. Many preparations\ntherefore need to start now . Policymakers should\nconsider:\n• Creating safeguards to ensure that government\nuse of AI respects legal and normative limits,\nsuch as by procuring AI tools to strengthen\nchecks between branches of government, shar-\ning key information about government AI sys-\ntems (e.g., model specs [ 73, 74]) with the\npublic, or requiring that AI systems follow the\nlaw [75].\n• Ensuring that citizens and civil society have the\ncapabilities to detect, document, and contest\nunlawful or harmful uses of AI, such as by giv-\ning them timely access to AI systems capable of\nsupporting these activities.\n• Helping build sufficient defenses against mis-\nuse by malicious non-state actors, such as by\nfunding better medical countermeasures against\nAI-enabled biological threats [ 76].\nConclusion\nAn intelligence explosion could be the most conse-\nquential technological development in human his-\ntory [1], compressing years of progress into months\nor less, threatening human control over AI systems,\nand severely eroding checks on power within and\nbetween states, companies, and branches of govern-\nment. Although there remains much uncertainty , AI\nR&D automation might soon trigger one. And while\nthis piece has focused on software-driven routes to\nan intelligence explosion, AI-driven improvements\nin hardware16 could make one all the more likely .\nRelative to the stakes, we are not sufficiently pre-\npared. Policymakers should have three priorities:\nWhat if automating AI R&D triggers an intelligence explosion? page 8 of 14
\nobtaining visibility into AI R&D automation within\nfrontier AI companies, developing ways to steer and\nconstrain an intelligence explosion, and preparing to\nadapt to its impacts. Once an intelligence explosion\nbegins, the window for action may close.\nAcknowledgments\nWe thank Anson Ho, T om Cunningham, Cheryl Wu,\nRyan Greenblatt, and many GovAI staff for feedback\nand conversations that improved this piece.\nWe are grateful to T aylor Jones for creating the\nfigures and to Zilan Qian for providing a Chinese\ntranslation of this piece.\nReferences\n[1] I. J. Good, “Speculations concerning the first ultraintelli-\ngent machine,” in Advances in computers , vol. 6, Elsevier ,\n1966, pp. 31–88 (cit. on pp. 2, 8).\n[2] A. M. T uring, Intelligent machinery, a heretical theory ,\n1951 (cit. on p. 2).\n[3] J. Evans, B. Bratton, and B. Ag ¨uera y Arcas, “Agentic AI\nand the next intelligence explosion,” Science, vol. 391,\nno. 6791, eaeg1895, Mar . 2026. DOI : 10.1126/scienc\ne.aeg1895 (cit. on p. 2).\n[4] J. Pachocki, An alien mind , Sep. 2026 (cit. on p. 2).\n[5] M. Favaro and J. Clark, When AI builds itself: Our\nprogress toward recursive self-improvement, and its impli-\ncations, Jun. 2026 (cit. on pp. 2, 3, 7).\n[6] T. T unguz, Are we being railroaded by AI? Nov . 2025\n(cit. on p. 2).\n[7] C. E. Leiserson, N. C. Thompson, J. S. Emer, B. C. Kusz-\nmaul, B. W. Lampson, D. Sanchez, and T. B. Schardl,\n“There’s plenty of room at the top: What will drive com-\nputer performance after Moore’s law?”Science, vol. 368,\nno. 6495, eaam9744, Jun. 2020. DOI : 10.1126/scienc\ne.aam9744 (cit. on pp. 2, 13).\n[8] A. Ho, T. Besiroglu, E. Erdil, D. Owen, R. Rahman, Z. C.\nGuo, D. Atkinson, N. Thompson, and J. Sevilla, Algorith-\nmic progress in language models , 2024 (cit. on p. 2).\n[9] OpenAI, How GPT-5.6 fuses frontier intelligence with\nfrontier efficiency , Jul. 2026 (cit. on p. 2).\n[10] M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S.\nGunasekar, M. Harrison, R. J. Hewett, M. Javaheripi,\nP. Kauffmann, J. R. Lee, Y. T. Lee, Y. Li, W. Liu, C. C. T.\nMendes, A. Nguyen, E. Price, G. de Rosa, O. Saarikivi,\nA. Salim, S. Shah, X. Wang, R. Ward, Y. Wu, D. Yu, C.\nZhang, and Y. Zhang, Phi-4 technical report , Dec. 2024.\nDOI : 10.48550/arXiv.2412.08905 (cit. on p. 2).\n[11] J.-S. Denain and C. Barber, An FAQ on reinforcement\nlearning environments, 2026 (cit. on p. 2).\n[12] D. Guo et al., “DeepSeek-R1 incentivizes reasoning in\nLLMs through reinforcement learning,” Nature, vol. 645,\nno. 8081, pp. 633–638, Sep. 2025. DOI : 10.1038/s415\n86-025-09422-z (cit. on pp. 2, 5).\n[13] OpenAI, Learning to reason with LLMs , Sep. 2024 (cit.\non p. 2).\n[14] D. Eth and T. Davidson, “Will AI R&D automation cause\na software intelligence explosion?,” 2025 (cit. on p. 3).\n[15] M. Favaro and P. Wright, Measurements for understand-\ning the pace of AI development inside frontier labs , An-\nthropic, Sep. 2026 (cit. on pp. 3, 7).\n[16] METR, Frontier risk report (February to March 2026) ,\nMay 2026 (cit. on p. 3).\n[17] METR, Time horizon 1.1 , Jan. 2026 (cit. on pp. 3, 13).\n[18] OpenAI, Research acceleration: The view inside OpenAI ,\nSep. 2026 (cit. on pp. 3, 7).\n[19] B. Rank, H. Bhatnagar , A. Prabhu, S. Eisenberg, K.\nNguyen, M. Bethge, and M. Andriushchenko, “PostTrain-\nBench: Can LLM agents automate LLM post-training?”\nIn International Conference on Machine Learning (ICML) ,
\narXiv: 2603.08640 [cs.SE] (cit. on p. 3).
[20] J. Wen, L. Qiu, J. Benton, J. H. Kirchner, and J. Leike,\nAutomated weak-to-strong researcher , Apr . 2026 (cit. on\np. 3).\n[21] J. Wen, C. Si, Y. -h. Chen, H. He, and S. Feng, Predicting\nempirical AI research outcomes with language models ,\nJun. 2025. DOI : 10.48550/arXiv.2506.00794 (cit. on\np. 3).\n[22] C. Lu, C. Lu, R. T. Lange, Y. Y amada, S. Hu, J. Foerster,\nD. Ha, and J. Clune, “T owards end-to-end automation\nof AI research,” Nature, vol. 651, no. 8107, pp. 914–\n919, Mar . 2026. DOI : 10.1038/s41586-026-10265-5\n(cit. on p. 3).\n[23] S. Rabanser, S. Kapoor, P. Kirgis, K. Liu, S. Utpala, and\nA. Narayanan, T owards a science of AI agent reliability ,\nFeb. 2026. DOI : 10.48550/arXiv.2602.16666 (cit. on\np. 3).\n[24] Anthropic, “Claude Fable 5 & Claude Mythos 5 system\ncard,” T ech. Rep., Jun. 2026 (cit. on p. 3).\n[25] OpenAI, “GPT-6 Astra system card,” T ech. Rep., Sep.\n2026 (cit. on pp. 3, 6).\n[26] P. Whitfill, C. Wu, J. Becker, and N. Rush, Many SWE-\nbench-passing PRs would not be merged into main , Mar .\n2026 (cit. on p. 3).\n[27] N. Bloom, C. I. Jones, J. Van Reenen, and M. Webb,\n“Are ideas getting harder to find?” American Economic\nReview, vol. 110, no. 4, pp. 1104–1144, Apr . 2020. DOI :\n10.1257/aer.20180338 (cit. on pp. 4, 12, 14).\n[28] A. Ho and P. Whitfill, The software intelligence explosion\ndebate needs experiments , 2025 (cit. on pp. 5, 12, 13).\n[29] P. Whitfill and C. Wu, Will compute bottlenecks prevent\nan intelligence explosion? Aug. 2025. DOI : 10.48550/ar\nXiv.2507.23181 (cit. on p. 5).\n[30] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown,\nB. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D.\nAmodei, Scaling laws for neural language models , Jan.
DOI : 10.48550/arXiv.2001.08361 (cit. on p. 5).
What if automating AI R&D triggers an intelligence explosion? page 9 of 14
[31] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya,\nT. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks,\nJ. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican,\nG. van den Driessche, B. Damoc, A. Guy, S. Osindero, K.\nSimonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre,\nTraining compute-optimal large language models , Mar .
\nDOI : 10.48550/arXiv.2203.15556 (cit. on p. 5).
[32] P. Villalobos, A. Ho, J. Sevilla, T. Besiroglu, L. Heim,\nand M. Hobbhahn, Will we run out of data? Limits of\nLLM scaling based on human-generated data , Jun. 2024.\nDOI : 10.48550/arXiv.2211.04325 (cit. on p. 5).\n[33] T. Davidson, B. Halperin, T. Houlden, and A. Korinek,\nWhen does automating AI research produce explosive\ngrowth? Feedback loops in innovation networks, Jul. 2026\n(cit. on p. 5).\n[34] L. Emberson and Y. Edelman, Frontier training runs will\nlikely stop getting longer by around 2027 , Jul. 2025 (cit.\non p. 5).\n[35] T. Davidson, J. -S. Denain, P. Villalobos, and G. Bas, AI\ncapabilities can be significantly improved without expen-\nsive retraining, Dec. 2023. DOI : 10.48550/arXiv.2312\n.07413 (cit. on pp. 5, 13).\n[36] A. Novikov, N. V ˜u, M. Eisenberger , E. Dupont, P. -S.\nHuang, A. Z. Wagner , S. Shirobokov , B. Kozlovskii,\nF. J. R. Ruiz, A. Mehrabian, M. P. Kumar , A. See, S.\nChaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli,\nand M. Balog, AlphaEvolve: A coding agent for scientific\nand algorithmic discovery , Jun. 2025. DOI : 10.48550/a\nrXiv.2506.13131 (cit. on p. 5).\n[37] T. Cunningham, L. Althoff, B. Halperin, B. Jabarian, A.\nKoh, A. Ramani, P. Trammell, P. Whitfill, and C. Wu,\nThe economics of recursive self-improvement , Jul. 2026\n(cit. on pp. 5, 7, 14).\n[38] H. Wang, T. Fu, Y. Du, W. Gao, K. Huang, Z. Liu, P.\nChandak, S. Liu, P. Van Katwyk, A. Deac, A. Anandku-\nmar , K. Bergen, C. P. Gomes, S. Ho, P. Kohli, J. Lasenby,\nJ. Leskovec, T. -Y. Liu, A. Manrai, D. Marks, B. Ramsun-\ndar , L. Song, J. Sun, J. T ang, P. Veliˇckovi´c, M. Welling,\nL. Zhang, C. W. Coley , Y. Bengio, and M. Zitnik, “Sci-\nentific discovery in the age of artificial intelligence,”\nNature, vol. 620, no. 7972, pp. 47–60, Aug. 2023. DOI :\n10.1038/s41586-023-06221-2 (cit. on p. 5).\n[39] K. E. Drexler, Radical abundance: How a revolution in\nnanotechnology will change civilization . PublicAffairs,\n2013 (cit. on p. 6).\n[40] Y. Bengio, S. Clare, C. Prunkl, M. Murray , M. An-\ndriushchenko, B. Bucknall, R. Bommasani, S. Casper,\nT. Davidson, R. Douglas, D. Duvenaud, P. Fox, U. Go-\nhar , R. Hadshar , A. Ho, T. Hu, C. Jones, S. Kapoor, A.\nKasirzadeh, S. Manning, N. Maslej, V. Mavroudis, C.\nMcGlynn, R. Moulange, J. Newman, K. Y. Ng, P. Paskov,\nS. Rismani, G. Sastry , E. Seger, S. Singer, C. Stix, L.\nVelasco, N. Wheeler, D. Acemoglu, V. Conitzer, T. G. Di-\netterich, E. W. Felten, F. Heintz, G. Hinton, N. Jennings,\nS. Leavy, T. Ludermir , V. Marda, H. Margetts, J. Mc-\nDermid, J. Munga, A. Narayanan, A. Nelson, C. Neppel,\nS. D. Ramchurn, S. Russell, M. Schaake, B. Sch ¨olkopf,\nA. Soto, L. Tiedrich, G. Varoquaux, A. Y ao, Y. -Q. Zhang,\nL. A. Aguirre, O. Ajala, F. Albalawi, N. AlMalek, C. Busch,\nJ. Collas, A. C. P. d. L. F. de Carvalho, A. Gill, A. H.\nHatip, J. Heikkil ¨a, C. Johnson, G. Jolly, Z. Katzir, M. N.\nKerema, H. Kitano, A. Kr ¨uger, K. M. Lee, J. R. L ´opez\nPortillo, A. McLysaght, O. Molchanovskyi, A. Monti, M.\nNemer , N. Oliver, R. Pezoa, A. Plonk, B. Ravindran, H.\nRiza, C. Rugege, H. Sheikh, D. Wong, Y. Zeng, L. Zhu, D.\nPrivitera, and S. Mindermann, “International AI safety\nreport 2026,” Department for Science, Innovation and\nT echnology , T ech. Rep. DSIT 2026/001, 2026 (cit. on\np. 6).\n[41] Y. Bengio, G. Hinton, A. Y ao, D. Song, P. Abbeel, T. Dar-\nrell, Y. N. Harari, Y.-Q. Zhang, L. Xue, S. Shalev-Shwartz,\nG. Hadfield, J. Clune, T. Maharaj, F. Hutter , A. G. Baydin,\nS. McIlraith, Q. Gao, A. Acharya, D. Krueger, A. Dra-\ngan, P. T orr, S. Russell, D. Kahneman, J. Brauner, and\nS. Mindermann, “Managing extreme AI risks amid rapid\nprogress,” Science, vol. 384, no. 6698, pp. 842–845, May
DOI : 10.1126/science.adn0117 (cit. on p. 6).
[42] C. Aveggio, A. J. Patel, S. Nevo, and K. Webster , “Explor-\ning the offense-defense balance of biology: Identifying\nand describing high-level asymmetries,” RAND Corpo-\nration, Santa Monica, CA, T ech. Rep. PE-A4102-1, Aug.\n2025 (cit. on p. 6).\n[43] R. Greenblatt, A. Cotra, and H. Wijk, “Brief independent\ninvestigation of agents’ behavior , reasoning and collabo-\nration in the OpenAI / Hugging Face hacking incident,”\nMETR and Redwood Research, T ech. Rep., Aug. 2026\n(cit. on p. 6).\n[44] OpenAI, OpenAI and Hugging Face partner to address\nsecurity incident during model evaluation , Jul. 2026 (cit.\non pp. 6, 8).\n[45] OpenAI, The Hugging Face incident and the road ahead ,\nAug. 2026 (cit. on p. 6).\n[46] M. Sulmeyer , “ Artificial intelligence and the risk of strate-\ngic cyberattack,” RAND Corporation, T ech. Rep., Jul.\n2026 (cit. on p. 6).\n[47] D. Hendrycks, E. Schmidt, and A. Wang, Superintelli-\ngence strategy: Expert version , Mar . 2025. DOI : 10.4855\n0/arXiv.2503.05628 (cit. on pp. 6, 8).\n[48] C. O’Keefe, A. Z. Rozenshtein, and C. Winter, Executive\nbranch AI and the rule of law: An emerging research\nagenda, May 2026 (cit. on p. 6).\n[49] T. Davidson, L. Finnveden, and R. Hadshar, “AI-enabled\ncoups: How a small group could use AI to seize power,”\n2025 (cit. on p. 6).\n[50] J. Edwards and L. Emberson, Open models lag state-of-\nthe-art closed models by 4 months , May 2026 (cit. on\np. 6).\n[51] R. Slayton, “What is the cyber offense-defense balance?\nConceptions, causes, and assessment,” International Se-\ncurity, vol. 41, no. 3, pp. 72–109, 2017. DOI : 10.1162\n/ISEC_a_00267 (cit. on p. 6).\n[52] B. Garfinkel and A. Dafoe, “How does the offense-\ndefense balance scale?” Journal of Strategic Studies ,\nvol. 42, no. 6, pp. 736–763, Aug. 2019. DOI : 10 . 10\n80/01402390.2019.1631810 (cit. on p. 6).\n[53] Transparency in Frontier Artificial Intelligence Act, SB 53,\n2025–2026 Reg. Sess. (Cal. 2025) , Sep. 2025 (cit. on\np. 7).\n[54] European Commission, The general-purpose AI code of\npractice, Jul. 2025 (cit. on p. 7).\nWhat if automating AI R&D triggers an intelligence explosion? page 10 of 14
[55] Responsible AI Safety and Education (RAISE) Act, A6453B,\n2025–2026 Reg. Sess. (N.Y. 2025) , Dec. 2025 (cit. on\np. 7).\n[56] J. Kwon and S. Casper, Internal deployment gaps in AI\nregulation, Jan. 2026. DOI : 10.48550/arXiv.2601.080\n05 (cit. on p. 7).\n[57] M. Pistillo, Internal deployment in the AI Act , Jun. 2026\n(cit. on p. 7).\n[58] A. Chan, R. Padarath, J. Kwon, H. Greaves, and M. An-\nderljung, Measuring AI R&D automation , Mar . 2026.DOI :\n10.48550/arXiv.2603.03992 (cit. on p. 7).\n[59] M. Brundage, N. Dreksler , A. Homewood, S. McGregor ,\nP. Paskov, C. Stosz, G. Sastry , A. F. Cooper, G. Balston,\nS. Adler, S. Casper, M. Anderljung, G. Werner, S. Min-\ndermann, V. Mavroudis, B. Bucknall, C. Stix, J. Freund,\nL. Pacchiardi, J. Hernandez-Orallo, M. Pistillo, M. Chen,\nC. Painter, D. W. Ball, C. O’Keefe, G. Weil, B. Harack,\nG. Finley , R. Hassan, S. Emmons, C. Foster , A. Reuel,\nB. Treece, Y. Bengio, D. Reti, R. Bommasani, C. Trout,\nA. S. Shamsabadi, R. Dattani, A. Weller, R. Trager, J.\nSevilla, L. Wagner , L. Soder, K. Ramakrishnan, H. Pa-\npadatos, M. Murray , and R. T ovcimak, Frontier AI au-\nditing: T oward rigorous third-party assessment of safety\nand security practices at leading AI companies , Feb. 2026.\nDOI : 10.48550/arXiv.2601.11699 (cit. on p. 7).\n[60] P. Wills, Regulatory supervision of frontier AI developers ,\nSSRN Scholarly Paper, Rochester , NY, Mar . 2025.DOI :\n10.2139/ssrn.5122871 (cit. on p. 7).\n[61] U.S. Nuclear Regulatory Commission, Backgrounder on\nNRC resident inspectors , 2023 (cit. on p. 7).\n[62] Office of the Comptroller of the Currency, What we do ,\n2026 (cit. on p. 7).\n[63] M. Baker, G. Kulp, O. Marks, M. Brundage, and L. Heim,\nVerifying international agreements on AI: Six layers of\nverification for rules on large-scale AI development and\ndeployment, Jul. 2025. DOI : 10.48550/arXiv.2507.15\n916 (cit. on p. 7).\n[64] B. Harack, R. F. Trager, A. Reuel, D. Manheim, M.\nBrundage, O. Aarne, A. Scher, Y. Pan, J. Xiao, K. Loke,\nS. N. Adan, G. Bas, N. A. Caputo, J. C. Morse, J. Ahuja,\nI. Duan, J. Egan, B. Bucknall, B. Rosen, R. Araujo,\nV. Boulanin, R. Lall, F. Barez, S. Alvira, C. Katzke, A.\nAtamli, and A. Awad, “Verification for international AI\ngovernance,” Oxford Martin AI Governance Initiative,\nUniversity of Oxford, T ech. Rep., Jul. 2025 (cit. on pp. 7,\n8).\n[65] T. Larsen, R. Dean, B. Halstead, E. Lifland, R. Greenblatt,\nand D. Kokotajlo, AI 2040: Plan A , AI Futures Project,\n2026 (cit. on pp. 7, 8).\n[66] G. Sastry , L. Heim, H. Belfield, M. Anderljung, M.\nBrundage, J. Hazell, C. O’Keefe, G. K. Hadfield, R. Ngo,\nK. Pilz, G. Gor, E. Bluemke, S. Shoker, J. Egan, R. F.\nTrager, S. Avin, A. Weller, Y. Bengio, and D. Coyle, Com-\nputing power and the governance of artificial intelligence ,\nFeb. 2024. DOI : 10.48550/arXiv.2402.08797 (cit. on\np. 7).\n[67] A. Korinek and J. E. Stiglitz, Steering technological\nprogress, Working Paper , Mar . 2026.DOI : 10.3386/w34\n994 (cit. on p. 8).\n[68] S. Shoker, A. Reddie, S. Barrington, R. Booth, M.\nBrundage, H. Chahal, M. Depp, B. Drexel, R. Gupta,\nM. Favaro, J. Hecla, A. Hickey, M. Konaev , K. Kumar ,\nN. Lambert, A. Lohn, C. O’Keefe, N. Rajani, M. Sell-\nitto, R. Trager, L. Walker, A. Wehsener, and J. Y oung,\nConfidence-building measures for artificial intelligence:\nWorkshop proceedings, Aug. 2023. DOI : 10.48550/arXi\nv.2308.00862 (cit. on p. 8).\n[69] Intelligence Rising, Intelligence Rising, n.d. (cit. on p. 8).\n[70] AI Futures Project, About AI 2027: T abletop exercise ,\n2025 (cit. on p. 8).\n[71] G. Smith, G. Hage, C. Heitzenrater, M. Chessen, and\nR. S. Girven, “Infinite potential—insights from the cyber\nsurprise scenario,” RAND Corporation, T ech. Rep., Mar .\n2026 (cit. on p. 8).\n[72] L. Vaintrob, The AI adoption gap: Preparing the US gov-\nernment for advanced AI , Apr . 2025 (cit. on p. 8).\n[73] OpenAI, OpenAI model spec , Aug. 2026 (cit. on p. 8).\n[74] Anthropic, Claude’s constitution, Jan. 2026 (cit. on p. 8).\n[75] C. O’Keefe, K. Ramakrishnan, J. T ay, and C. Winter,\n“Law-following AI: Designing AI agents to obey human\nlaws,” Fordham Law Review, vol. 94, no. 1, p. 57, 2025\n(cit. on p. 8).\n[76] S. Guerra, A. Attal-Juncqua, J. P. T arangelo, C. Aveggio,\nK. Dammer, and D. Glickstein, “Building a defense-in-\ndepth biosecurity strategy for the AI era,” RAND Corpo-\nration, T ech. Rep., Aug. 2026 (cit. on p. 8).\n[77] J.-S. Denain, A. Ho, and J. Sevilla, How many digital\nworkers could OpenAI deploy? Oct. 2025 (cit. on p. 12).\n[78] H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T.\nBroadley, L. Chan, M. Chen, J. Clymer , J. Dhyani, E.\nEricheva, K. Garcia, B. Goodrich, N. Jurkovic, H. Karnof-\nsky , M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders,\nM. T aran, B. West, and E. Barnes, RE-Bench: Evaluating\nfrontier AI R&D capabilities of language model agents\nagainst human experts , May 2025. DOI : 10.48550/arXi\nv.2411.15114 (cit. on p. 12).\n[79] A. Ho, J. -S. Denain, D. Atanasov, S. Albanie, and R.\nShah, A rosetta stone for AI benchmarks , 2025 (cit. on\np. 13).\n[80] B. Cottier, B. Snodin, D. Owen, and T. Adamczewski,\nLLM inference prices have fallen rapidly but unequally\nacross tasks, Mar . 2025 (cit. on pp. 13, 14).\n[81] H. Gundlach, A. Fogelson, J. Lynch, A. Trisovic, J. Rosen-\nfeld, A. Sandhu, and N. Thompson, On the origin of\nalgorithmic progress in AI , Nov . 2025. DOI : 10.48550/a\nrXiv.2511.21622 (cit. on p. 13).\n[82] P. Trammell, “The bounded parallelizability of R&D:\nTheory and application to AI,” Jul. 2026 (cit. on p. 13).\n[83] METR, Measuring AI ability to complete long tasks , Mar .\n2025 (cit. on p. 13).\n[84] D. Kokotajlo, S. Alexander, T. Larsen, E. Lifland, and\nR. Dean, AI 2027, Apr . 2025 (cit. on p. 14).\n[85] T. Ord, The dynamics of intelligence explosions , 2026.\narXiv: 2608.14426 [cs.AI] (cit. on p. 14).\n[86] B. Murphy and T. Stone, Uplifted attackers, human de-\nfenders: The cyber offense-defense balance for trailing-edge\norganizations, Aug. 2025. DOI : 10.48550/arXiv.2508\n.15808 (cit. on p. 14).\nWhat if automating AI R&D triggers an intelligence explosion? page 11 of 14
\n[87] Anthropic, Investigating three real-world incidents in our\ncybersecurity evaluations, Jul. 2026 (cit. on p. 14).\n[88] UK AI Security Institute, Incident report: Unsanctioned\nagent behaviour during cyber testing , Aug. 2026 (cit. on\np. 14).\n[89] OpenAI, Third-party cyber evaluations involving OpenAI\nmodels, Aug. 2026 (cit. on p. 14).\n[90] P. C. Bogdan, R. Qi, J. Eaton, S. Kennedy, F. Roger, A.\nGlynn, R. Chen, B. Wright, O. Stegmaier, J. Kutasov,\nD. Foreman-Mackey, T. Bricken, S. Carr, S. Carter, M.\nMacDiarmid, S. Marks, A. Pearce, E. Simon, N. Carlini,\nC. Burns, J. Lindsey, S. Price, and S. Kantamneni, An\nalignment assessment of recent cybersecurity incidents ,\nAnthropic, Sep. 2026 (cit. on p. 14).\n[91] M. H. T essler, M. A. Bakker, D. Jarrett, H. Sheahan, M. J.\nChadwick, R. Koster, G. Evans, L. Campbell-Gillingham,\nT. Collins, D. C. Parkes, M. Botvinick, and C. Summer-\nfield, “AI can help humans find common ground in\ndemocratic deliberation,” Science, vol. 386, no. 6719,\neadq2852, Oct. 2024. DOI : 10.1126/science.adq2852\n(cit. on p. 14).\n[92] A. Goldie and A. Mirhoseini, How AlphaChip transformed\ncomputer chip design , Sep. 2024 (cit. on p. 14).\nSupplementary Materials\nEstimate of the e!ective size of an AI\nworkforce\nFollowing Denain et al. [77], we estimate the effec-\ntive workforce by dividing the number of tokens a\ndeveloper can generate per day by the number of\ntokens corresponding to one researcher-day of work.\nOpenAI alone has enough inference compute (i.e.,\nruntime compute) to generate on the order of 1013\ntokens per day . T o estimate tokens per researcher-\nday , we use the number of tokens a model generates\non a task that would take a human researcher one\nworkday (8 hours). In an AI R&D benchmark, Wijk\net al. [78] find that models output on average 5 · 105\ntokens on runs of up to 8 hours, implying an effective\nworkforce of about 2 · 107 researchers. Allowing for\nabout an order of magnitude of uncertainty in either\ndirection (5 · 104–5 · 106 tokens per researcher-day),\nwe estimate an effective workforce on the order of\n2 · 106–2 · 108. As in the main text, this assumes that\nexpert-level AI systems have runtime costs compara-\nble to those of today’s systems.\nModeling the feedback loop under full\nautomation\nMost analyses of a software-driven intelligence ex-\nplosion capture AI progress with some notion of soft-\nware quality , denoted by A. In principle, A should\nmeasure AI progress holistically , including both effi-\nciency improvements (new AI systems accomplishing\nthe same tasks as old systems, with similar perfor-\nmance, using less compute or data) and capability im-\nprovements (new AI systems accomplishing tasks that\nprevious systems could not accomplish, or achiev-\ning higher performance than previous systems could\nachieve). Frustratingly , it is currently unclear how\nthe parameter A should best trade off between effi-\nciency improvements and capability improvements\n(or between different types of efficiency improve-\nments and capability improvements).\nSetting this issue aside, we model the growth of\nsoftware quality with dA/dt = A1→ω Eε, a functional\nform common in the macroeconomics of innova-\ntion [ 27]. For simplicity , we ignore potential bot-\ntlenecks from compute and data. Here, E is effective\nR&D labor ,ω represents returns to scale on R&D la-\nbor , andε represents whether there are increasing or\ndiminishing returns to finding new ideas over time.\nThe key question is how E grows with A. Assum-\ning full automation, we consider two cases. First,\nconsider a case where all improvements in software\nquality are increases in inference compute efficiency ,\nthat is, decreases in the amount of compute needed\nto run an AI system with a certain capability level.\nIn that case, if A measures inference efficiency , then\nE is proportional to A: greater inference efficiency\nallows proportionally more automated researchers\nto be run.\nSecond, consider a case where all improvements\nin software quality are capability gains, driven by im-\nprovements in training compute efficiency : decreases\nin the amount of compute needed to train an AI sys-\ntem to a given capability level, which allow a more\ncapable system to be trained with a fixed stock of\ncompute. If A measures training compute efficiency ,\nthen increases in A yield more capable systems rather\nthan more of them. If we make the (potentially ques-\ntionable) assumption that the effective number of\nresearchers scales linearly with these capability gains,\nthen E is again proportional to A.\nIn both cases, E = kA for some positive con-\nstant k. Substituting into the equation above gives\ndA/dt = cAε→ω+1, where c = kε. For the growth\nrate (1/A) dA/dt to increase as software quality in-\ncreases, we need ω>ε . Defining r = ω/ε, we obtain\nthe condition r> 1 discussed in the main text.\nHow quickly could progress accelerate under cur-\nrent estimates of these parameters? We measure\nacceleration by the growth rate of software qual-\nity: (1/A) dA/dt = c · Aε→ω . Averaging the cen-\ntral estimates across the three sub-fields in Ho and\nWhat if automating AI R&D triggers an intelligence explosion? page 12 of 14
\nWhitfill [ 28] gives ω =1 .40 and ε =1 .01. With\nω → ε =0 .39, each doubling of A multiplies the\ngrowth rate by 20.39 ↑ 1.31, so each subsequent\ndoubling takes (1/2)0.39 ↑ 76% as long as the last.\nFor the growth rate to increase tenfold, A needs to\ndouble log2(10)/0.39 ↑ 8.5 times. For the length of\nthe first doubling, we use recent estimates that, due\nto software improvements alone, training compute\nefficiency doubles roughly every 4.5 months [ 79];\ninference compute efficiency may be growing even\nfaster [80]. T o avoid having to approximate the geo-\nmetric sum for 8.5 doublings, we instead calculate\nthe geometric sum for 9 doublings, after which time\nthe growth rate will have increased more than ten-\nfold. Doing so, we find that the growth rate will\nhave increased more than tenfold after 4.5 months\n↓(1 → 0.769)/(1 → 0.76) ↑ 17 months, or about 1.5\nyears. At that point, progress would be ten times\nfaster than today , so a year’s worth of progress at\ntoday’s pace would take about five weeks.\nUncertainties about the returns to research\ne!ort\nSeveral factors could make the true value of r dif-\nfer from existing estimates. First, as noted above,\nit is unclear which measure of software quality is\nmost appropriate. Inference efficiency maps most di-\nrectly onto the size of an automated R&D workforce,\nwhereas it is unclear how to translate training effi-\nciency gains into the effective number of researchers.\nY et existing estimates of r for AI R&D use training\nefficiency , and it is unknown how similar the returns\nto research effort are under the two measures. Fur-\nthermore, a proper assessment of returns to research\neffort would include improvements from both infer-\nence efficiency and training efficiency , rather than\njust one. More work is warranted to develop a charac-\nterization of software quality that incorporates both\ninference and training efficiency and weighs them\nagainst each other appropriately .\nExisting estimates of r come from a period of\nrapid compute scaling, which confounds the con-\ntributions of software progress and compute scaling.\nBecause these two inputs grew together historically ,\nan estimate that attributes observed progress to soft-\nware improvements may actually be capturing gains\ndriven by , or only made possible by , rising compute.\nSome software improvements are scale-dependent:\nfor example, the transformer architecture yields large\nperformance gains at high training compute but rel-\natively small gains at low compute [ 81]. Such con-\nfounding would bias estimates of r upward: in a\nregime of fixed or slowly growing compute, r would\nbe lower than historical data suggest.\nOther factors could imply a higher r. Most esti-\nmates of r neglect improvements in areas outside\nof pre-training, such as post-training or better scaf-\nfolding for tool use [ 35]. Furthermore, capability\nimprovements could matter in ways that the effec-\ntive number of researchers fails to capture: even\nan extremely large number of mediocre researchers\nmay not be able to substitute for one genius re-\nsearcher . If so, capability gains would expand ef-\nfective R&D labor by more than the linear assump-\ntion above implies. Compute bottlenecks could also\nbe circumvented, such as through better extrapo-\nlation from small-scale experiments, reductions in\nexperiment cost from software progress, shifts to-\nward approaches that are less compute-reliant, and\nalgorithmic progress that does not require experi-\nments [7].\nEstimates of r also rely on imperfect proxies for\nR&D labor , which could bias them in either direc-\ntion. For example, Ho and Whitfill [ 28] proxy R&D\nlabor with the number of unique authors who have\npublished papers in a domain. Drawing the domain\ntoo narrowly undercounts labor by excluding adja-\ncent fields that also drive progress, while drawing it\ntoo broadly overcounts labor . Unless the excluded\nadjacent fields see the same growth rate in labor\nas the included fields, the disconnect will lead to\nmiscalculating r.\nFinally , the relevant mathematical models may\nnot generalize to extremely large amounts of R&D\nlabor [ 82]. Historically , they have been validated\nagainst growth rates of a few percent per year , well\nbelow the double-digit or higher rates that an intel-\nligence explosion could produce. They also break\ndown in the limit, where they imply that infinite la-\nbor yields infinite progress in finite time. But real\nconstraints make this result impossible: some prob-\nlems must be solved in sequence, and physical hard-\nware can only operate so fast.\nNotes
\nMachine-learning venues typically have a main conference
track and several workshop tracks. One caveat to the\nresults is that these workshop tracks can have somewhat\nlaxer standards than the main conference track.
According to the METR time-horizon metric [ 17, 83], the
length of tasks that AI systems can complete initially dou-\nbled roughly every 7 months, accelerating to about every\n3 months since 2024. Extrapolating this more recent trend\nwould suggest that by mid-2028, AI systems will be able to\ncomplete tasks requiring several months of human expert\nWhat if automating AI R&D triggers an intelligence explosion? page 13 of 14
time, well within the range of many AI R&D projects. See\nalso Kokotajlo et al. [84].
\nCottier et al. [80] find that the price of running LLMs
to achieve a given capability milestone (e.g., GPT -4-level\nperformance on a math benchmark) has decreased by\nroughly 9- to 900-fold per year , depending on the capa-\nbility milestone. While a portion of this cost decline has\ncome from hardware improvements (reducing the cost per\ncomputation), a substantial fraction is due to software\nimprovements.
In an area of technology , r governs the relationship be-
tween increases in R&D inputs and the resultant change\nin an output of interest, such as the number of computa-\ntions that cutting-edge consumer hardware can perform\nper constant dollar . In Bloom et al. [27], the input is mea-\nsured in dollars spent on R&D and so includes increases\nin both labor and physical capital. In this piece, however ,\nwe only consider increases in labor , as we are interested in\nunderstanding the potential for a software-driven feedback\nloop in which the compute stock is held roughly constant.\nThis means the value of r is lower than if we considered\nincreases in all inputs (i.e., labor , data, and compute).
r must eventually drop below 1 because AI progress will
eventually hit computational and physical limits. It is\nuncertain how much progress is possible before reaching\nsuch limits.
The 90% credible intervals are (0.727 to 2.094), (0.380 to
2.708), and (1.069 to 3.212).
In this analysis, improvements in algorithmic efficiency
do not by themselves resolve this potential bottleneck\nbecause they proportionally make both R&D and training\nmore efficient.
Specifically , the paper provides conditions under which
automated research labor , among other quantities like\neconomic output, grows to infinity in finite time.
See Ord [ 85] for a theoretical discussion of how the time
between rounds of R&D could affect the dynamics of an\nintelligence explosion.
The analysis in Cunningham et al. [37] focuses on self-
sustaining acceleration : “When AI systems are sufficient\nfor accelerating progress in AI capabilities without any\ngrowth in exogenous inputs (human labor , training com-\npute, etc.).” Self-sustaining acceleration is necessary for a\nsoftware-driven intelligence explosion in our sense.
Such systems could be superhuman in some domains (e.g.,
certain fields of scientific research) but not others (e.g.,\nmanipulating objects in the physical world).
Even in cyber , however , human organizational processes
could still add friction for defenders [ 86].
See also Anthropic [ 87], UK AI Security Institute [ 88],
OpenAI [89], and Bogdan et al. [90].
For example, see T essler et al. [91]. More speculatively , AI
systems could potentially accelerate the development of\nbrain-computer interfaces that allow humans to think and\ncoordinate much faster .
Precisely operationalizing an intelligence explosion is
tricky and remains an area for future work.
For example, AI is already aiding chip design [ 92]. AI
systems could also accelerate robotics to automate the\nchip production process.\nWhat if automating AI R&D triggers an intelligence explosion? page 14 of 14