![\"\"](\"https://substackcdn.com/image/fetch/$s_!2uht!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcf3996a1-6e32-47fb-9120-8713b3e90565_741x719.png\")
AI companies like Anthropic and OpenAI have long mentioned the possibility that their products could be existentially dangerous, but this has recently crescendoed into even more dire warnings, including suggestions that we have to pace the frontier, ie coordinate to slow AI progress.
\nIn response, cynical commentators have suggested this is a conspiracy to boost the companies’ stocks. The exact motives imputed vary from person to person. Some say the companies are hyping the product by talking about how dangerous it is; if AI is smart enough to kill all humans, it must be smart enough for your business workflow. Others say they’re trying to distract from near-term harms; every inch of newspaper space devoted to human extinction is one less inch that can be spent on plagiarism and water use. But the most popular story involves regulatory capture: if the product is unsafe, maybe it can only be trusted to a few big politically-connected companies, and everyone else should be banned.
\nMany debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
\nTaking these in order:
\nGrace et al surveyed 1,580 AI researchers who presented papers at big AI-related conferences, mostly academics with no company affiliation. Over half thought there was at least a 10% chance AI would cause “human extinction or similarly permanent and severe disempowerment of the human species”. 72% said we needed to be prioritizing AI safety research more. This survey was in 2024; current numbers would probably be higher.
\nGeoffrey Hinton, Yoshua Bengio, and Yann LeCunn are collectively considered “the godfathers of AI” for inventing deep learning; in 2018, they received the Turing Award, sometimes called the Nobel Prize of computer science, for their accomplishment. In 2023, Geoffrey Hinton quit his job at Google to work full-time raising awareness of the potential existential dangers of AI; he said there was “between 10% and 50% chance” that his invention would destroy humanity. In the same year, Yoshua Bengio also switched to activism, saying that he “regretted his life’s work” and that AI could have a “catastrophic impact”. Neither was affiliated with a major AI company.
\n(the third godfather, Yann LeCunn, runs a $3 billion AI company, and says he isn’t worried)
\nAlong with these independent researchers, there are many others who formerly worked at companies, then quit to whistleblow about the dangers, or left to pursue anti-AI activism full-time. Paul Christiano left OpenAI in 2021 to found an AI alignment nonprofit; he later said there was a “50/50 chance of doom shortly after you have AI systems that are human level.” Daniel Kokotajlo quit OpenAI in 2024, refusing to sign a non-disparagement clause even though it could have cost him $2 million in equity, in order to start a nonprofit warning about risks from AI (he says he thinks there’s about an 80% chance AI kills everyone). Jacob Coxon quit Anthropic in September, also likely giving up millions of dollars in equity, because he thought it was “gambling with our lives”. There are dozens of others.
\nThe people at AI companies are no more likely to raise the alarm about AI dangers than people who were never at AI companies, or people who were previously at AI companies but have since quit.
\nDARIO AMODEI first started discussing the risk of superintelligence destroying humanity in 2013, before OpenAI or Anthropic was founded, and when Amodei wasn’t employed in tech at all. Here’s a 62 page debate between Amodei, Eliezer Yudkowsky, and other AI safety enthusiasts about strategy, dated October 27, 2013:
\n
In 2017, when Amodei was an ordinary employee at OpenAI, he spoke about his work the on the podcast 80,000 Hours (July 21, 2017):
\n\nI mainly work on safety. We have a safety team that’s so far, myself and Paul Christiano. Paul was co-author of mine on the [Concrete Problems In AI Safety] paper and has also written a lot online on his blog about AI. He’s probably one of the people who’s I think done the most to promote clear thinking about the problem and tie it to current AI. We have a third person joining in a few weeks who I’m super excited about. We’re trying to build up a team that focuses on technical safety. We also do a little bit of strategy stuff which is how do we get different organizations that are working on AI to cooperate with each other, how do we cooperate with policy makers on questions like these. We’re also thinking a little bit about those issues but mainly technical safety.
Dario expanded on the cooperation point - he told the host about his fear that AI would lead to an “arms race” in which companies competed to develop the technology first and let safety slide. His solution was making companies cooperate:
\n\nWe all think about AGI and think, that safety issues are important. . . then the question is, how does that scale to there being a lot of organizations? How does that scale to others who get involved once they see how powerful AI is? Can we make them cooperate as well? My hope is that we can.
This became an obsession with him, and he hired an advisor to help OpenAI work on the problem. According to the New Yorker:
\n\nIn 2017, Amodei hired Page Hedley, a former public-interest lawyer, to be OpenAI’s policy and ethics adviser. In an early PowerPoint presentation to executives, Hedley outlined how OpenAI might avert a “catastrophic” arms race—perhaps by building a coalition of A.I. labs that would eventually coördinate with an international body akin to NATO, to insure that the technology was deployed safely.
\nAs Hedley recalled it, [OpenAI president Greg] Brockman didn’t understand how this would help the company beat its competitors. “No matter what I said,” Hedley told us, “Greg kept going back to ‘So how do we raise more money? How do we win?’
This incident and several like it turned Dario against Greg, leading to an era of constant backstabbing office politics. Dario led a pro-safety faction against Greg’s pro-business faction; Greg usually won. It didn’t help that Dario was anxious to the point of paranoia, constantly insisting that their work was too dangerous to release:
\n![\"\"](\"https://substackcdn.com/image/fetch/$s_!mU0u!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b046705-8904-4b4f-9e43-b758a6e2c8c1_952x362.png\")
Sam Altman tried to play peacemaker by promising Dario various concessions, but eventually Dario noticed that Altman lied constantly and never came through on any of them, so he and the rest of the safety faction left to found Anthropic. Here’s how Fortune described the split:
\n\nAmodei, along with a small group of employees, became concerned about OpenAI’s direction and what he perceived as a lack of safety involved in scaling up ChatGPT.
\nIn 2021, Amodei, along with his sister and cofounder Daniela Amodei and a dozen other ex-OpenAI employees started Anthropic with a mission to put safety first. The company released its large language model, Claude, in 2023.
\nThe employees who jumped ship from OpenAI believed two things, Amodei told Fortune’s Jeremy Kahn at the Fortune Brainstorm Tech conference in 2023. First, they believed if you pour more compute into large language models, they will greatly improve, and there’s almost no limit to how advanced they can get. Second, they believed—in addition to scaling up the models—there needed to be a strong set of guardrails and safety considerations.
SAM ALTMAN said in February 2015 - that is, ten months before he started OpenAI - that “development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.” He continued:
\n\nThe US government, and all other governments, should regulate the development of SMI [superhuman machine intelligence]. In an ideal world, regulation would slow down the bad guys and speed up the good guys—it seems like what happens with the first SMI to be developed will be very important.
\nAlthough my general belief is that technology is often over-regulated, I think some regulation is a good thing, and I’d hate to live in a world with no regulation at all. And I think it’s definitely a good thing when the survival of humanity is in question. (Incidentally, there is precedent for classification of privately-developed knowledge when it carries mass risk to human life. SILEX is perhaps the best-known example.)
He also famously said in June of that year that “AI will most likely lead to the end of the world, but in the meantime there will be great companies”. Some people say this was a “joke”. I think it was the one time in his life that Sam Altman was ever completely 100% honest.
\nELON MUSK, another AI company leader who has been talking about extinction risk, has an equally long relationship with the topic. His first statement on AI safety was in 2014, before he had any business ventures in AI, when he said that “We are summoning the demon with artificial intelligence”. He continued by calling it “our biggest existential threat”, and admitted that “I’m increasingly inclined to think there should be some regulatory oversight, maybe at the national and international level just to make sure that we don’t do something very foolish”.
\nHere’s a photo from the first big AI Safety conference in Puerto Rico. Elon is circled in orange. He was a rare famous billionaire who agreed to hang out with powerless nerds to debate the safety of a technology that technically didn’t exist yet, just for the love of the game.
\n![\"\"](\"https://substackcdn.com/image/fetch/$s_!6WUN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd0cd842e-337d-4773-936d-172dbc8eac8e_897x430.png\")
The green circle is Demis Hassabis, the purple is Ilya Sutskever, and the red is Eliezer Yudkowsky.
\nAfter the conference, Elon agreed to donate $10 million to the fledgling cause. He continued to be a regular at AI safety conferences over the next few years. I saw him speak at the Asilomar conference in 2017, where he gave a rambling talk on how brain-computer interfaces could solve the problem (they didn’t).
\nSHANE LEGG and DEMIS HASSABIS are two of the three co-founders of DeepMind, now Google DeepMind. In 2008 - a year before founding Deep Mind - Legg wrote his PhD thesis on “Machine Super Intelligence”, where he wrote that it could be “a nightmare scenario” (although he also said that “if we carefully prepare for this possibility in advance, not only might we avert disaster, we might bring about an age of prosperity unlike anything seen before.”) His other optimistic predictions include “If by suffering you mean prolonged suffering, then I think this is quite unlikely. If a super intelligent machine (or any kind of super intelligent agent) decided to get rid of us, I think it would do so pretty efficiently.”
\nHassabis was also concerned at least as early as 2012, when he reportedly tried to convince Elon Musk that AI was an existential risk. He attended the same 2015 AI safety conference as Musk (see photo above) and included as a condition of DeepMind’s acquisition by Google that Google set up a powerful AI safety oversight board (this went wrong in approximately every way possible).
\nTHE CORPORATE PLOT hypothesis requires these CEOs to be the most prescient people in history. In 2008 or 2013 or 2015, when they were all in other industries and AI was just an academic curiosity, they would have had to realize that one day AI would become a trillion dollar business, that they would be the CEO of one of the companies, and that the company would benefit from regulatory capture. Then they would have had to devote a significant proportion of their life, and in some cases donate millions of dollars, to start laying the groundwork ahead of time, just in case it became necessary. Dario Amodei, while finishing his biology postdoc, would have to have moonlighted as an AI safety enthusiast just in case he became a multi-billion dollar tech CEO one day. Shane Legg would have had to choose his PhD thesis topic based on the possibility that he would own shares in a Google AI subsidiary sixteen years later.
\nThe alternative is that all these people believed it then, and still believe it now, for the same reason that all of the independent experts and academics believe it. They are working in the field partly because they think that being closer to the development of AI gives them more of a chance to control it and keep it safe than if they were sitting on the sidelines.
\nHaving shown that these men would have to be the smartest people in the world to pull this off, it’s worth noting that the plot they apparently dedicated their lives to is really dumb.
\nFor example, if the Pacing The Frontier consortium attempt was a plot to pump stock prices, you might expect that, upon it being announced, stock prices would go up. This didn’t happen.
\n![\"\"](\"https://substackcdn.com/image/fetch/$s_!GP_e!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fddc43691-5d75-4fc6-8e86-18a3ca5e5e3d_979x647.png\")
Anthropic and OpenAI aren’t publicly traded companies, so they don’t have an easily-tracked stock price. But there are complicated cryptocurrency bets called “perpetual futures” that traders use to approximate the stock prices of private companies (usually these are some kind of prediction-market like instrument on eventual IPO price). When the slowdown call happened, Anthropic’s value on the biggest cryptocurrency futures platform dropped 4.57%, and OpenAI’s stock dropped 13.42%.
\n(two other AI companies that endorsed the slowdown proposal, Google DeepMind and X.AI, are subsidiaries of larger publicly traded companies; Google and SpaceX respectively. Both companies went down modestly the day after the announcement, although they have so many other things going on that I take this less seriously than the AI company futures.)
\nWhy wouldn’t the stock market (or at least this particular cryptocurrency simulation of the stock market) approve of the slowdown as a potential regulatory moat? It seems like traders thought the first-level effects (slowing down) were more important than any secondary regulatory effects that might or might not materialize.
\nLibertarian economist Alex Tabarrok is head of the Center for Study of Public Choice, the subfield of economics that studies things like regulatory capture. He agrees with the traders. He wrote a good post, What Regulatory Capture Actually Looks Like, arguing that a regulatory capture story doesn’t fit:
\n\nGo ahead: argue that Amodei and other AI experts are wrong about AI risk. Ask whether his proposals favor Anthropic. But calling “our product might kill you” a clever marketing and regulatory-capture strategy isn’t sophisticated analysis. The facts don’t fit regulatory capture theory and trying to make them fit requires epistemically painful Ptolemaic epicycles. Even a dull Ockham’s razor cuts through that story to the obvious alternative: Amodei actually believes what he’s saying.
Or consider another story - trying to distract from near-term harms. Is this really how things work? Are there vicious fights between environmentalists who want to save the rainforest and environmentalists who want to fight climate change, with the former arguing that the latter’s focus on longer-term risks detracts from environmental problems that are happening right now? No, obviously this is stupid; these groups are natural allies. Both donate to the Sierra Club and vote for the Green Party. Even if a certain rainforest supporter didn’t care about climate change at all, they would be grateful for the climate change activist’s votes and donations.
\nThis is how things are shaping up with AI also. Last summer, Ted Cruz sponsored a bill to ban states from regulating AI. It had strong support from AI money and was expected to pass. A coalition of effective altruist groups worried about human extinction, artists’ groups worried about intellectual property, family groups worried about harms to children, and many others united to defeat it.
\nBut maybe it’s true for newspaper space in particular? Maybe concerns about human extinction are preventing journalists from writing about nearer-term harms?
\nBoth people interested in human extinction and people interested in near-term harms fund the Tarbell Center for AI Journalism, which helps fund reporters on both category of risk (Tarbell received an ACX Grant two years ago). They highlight articles they’ve helped with on issues across the spectrum, from Worried Anthropic Researchers Warn That AI Could Kill All Humans to Hundreds Gather To Urge Scottish Government To Pause Data Center Boom to AI Is More Likely Than Humans To Form Biases When Hiring.
\nIn retrospect, this alliance should be obvious. Both groups want the same thing: the ability to slow AI. And both have the same enemy: the tsunami of Silicon Valley money which is trying to elect Congressmen who refuse to regulate AI in any way. Both groups are working together against their common foe. Signal-boosting long-term risks to distract from short-term risks would only empower this coalition, and so - like the other version of this plot - be implausibly stupid.
\nAs for the claim that AI companies are trying to hype their products, this seems unnecessary: AI is the fastest-adopted technology in history:
\n![\"AI](\"https://substackcdn.com/image/fetch/$s_!yVUT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F76e23b5c-2dc6-4942-9fdb-adce23b64ef6_619x443.png\" "\\"AI")
…so it hardly seems worth it to hype it up a little more at the cost of getting a large number of Americans to think your product will kill them personally, and a plurality of voters in both parties to want to ban your industry from creating new products:
\n![\"\"](\"https://substackcdn.com/image/fetch/$s_!XudH!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F66ddfc96-408d-4181-a942-8be13be674bf_830x769.png\")
If the last section required AI company leaders to be the smartest people in the world, this one requires them to be the dumbest.
\nWe live in an age of cynicism and nihilism. When someone endorses a good cause, we accuse them of virtue-signaling; if they donate to it, we accuse them of trying to whitewash their role in capitalism. We lionize corrupt politicians and insipid bling-wielding celebrities because, as people who have never made any claim to moral acceptability, they are the only people who can entirely escape the suspicion of faking it.
\nThis is idiotic and requires otherwise smart people to constantly contort themselves to figure out the latest less-than-entirely-depraved person’s angle. Maybe some people are so venal that they can’t imagine anyone truly caring about the welfare of humanity, but they should stop projecting this on everyone else. The experts are warning you that AI could kill you because they think AI could kill you. Many of them have made personal sacrifices to get the message out. Stop adding thumbtacks to your conspiracy corkboard and take their message seriously.
","headings":[{"level":3,"text":"I. The AI Experts Who Aren’t Affiliated With Big Companies Are Just As Concerned As The Ones Who Are","id":"i-the-ai-experts-who-aren-t-affiliated-with-big-companies-are-ju"},{"level":3,"text":"II. The AI Company Leaders Giving These Warnings Were Concerned Long Before They Ever Had AI Companies.","id":"ii-the-ai-company-leaders-giving-these-warnings-were-concerned-l"},{"level":3,"text":"III. If This Is A Plot, It’s A Really Dumb One","id":"iii-if-this-is-a-plot-it-s-a-really-dumb-one"}]}}