Why I Changed My Mind About AI Risk

It has been very hard for people not obsessively focused on the subject of artificial intelligence to come away with reasonable opinions about what the future holds. There have been so many extreme predictions of both a positive and negative sort that broad skepticism about anything said on the subject is fully warranted. But while that skepticism is healthy, I have to admit my own views have evolved significantly just over the past few months.

There are two poles of AI futurism: accelerationists and doomers. The former stress the positive impacts of AI, especially for economic growth, and the likelihood that it will lead to an unbelievably abundant age. Accelerationists like David Sacks have captured Donald Trump’s attention, and have beaten back efforts to regulate the technology.

The doomers, by contrast, point to a range of bad outcomes, with some placing their bets on extinction of the human species. I have become much more skeptical of the accelerationist narrative, and more open to some of the doomer scenarios. On the whole, this makes me think that regulation and a negotiated slowdown in the development of the technology is increasingly necessary.

There is no question that AI will have many positive effects on the U.S. economy. But there is a class of AI accelerationists who are completely out to lunch, of which Elon Musk is a prime example (though he has also weighed in as a doomer). In an interview this summer with Zanny Minton Beddoes, editor-in-chief of the Economist, he was asked what people would do for money when superintelligence arrives. He replied that they wouldn’t need it because everyone would have everything they wanted.

Other accelerationists have posited that GDP growth rates for the United States and other advanced countries using AI would increase to 10 or 20 percent per year, far above the 1-2 percent growth they were able to achieve in good years before the advent of AI.

The fundamental reason I don’t believe these extreme accelerationist claims is their overvaluation of intelligence as an input to economic growth, and their disregard for the material and political constraints that will keep growth within recognizable bounds. At a 10 percent annual growth rate, GDP would double in a little over seven years. This would require a doubling of all of the material inputs to growth: energy, raw materials, rare earths, land, and many other things. Where is all this stuff going to come from? Intelligence by itself may aid in the production of such inputs, but it will simply not dig the mines and build the factories on its own. And then there are political constraints: how will superintelligence open the Strait of Hormuz, which is currently a major constraint on global growth rates?

Accelerationists tend to be very intelligent people, but that intelligence tends to be narrowly mathematical. We know how to do many things like produce electricity or clean water in the cities of poor countries; the problem is a failure of implementation.

There is no question that AI will increase productivity in many sectors of the economy over time. It is a general purpose technology that will allow machines to substitute for human labor in many areas. But that very prospect in the long run leads directly to one of the biggest doomer scenarios: the devaluation of the labor of masses of people who work in the present-day economy. Accelerationists predict that those losing work today will find new jobs making use of AI, and that actually seems to be happening today as employment booms due to the AI buildout. But job loss will come eventually as AI substitutes for human labor in every part of the economy, particularly white-collar work that involves symbolic manipulation (i.e. people sitting behind computer screens all day).

This lost labor simply cannot be compensated for by transferring money from the AI winners to the AI losers, in the form of, for example, universal basic income. A job and income mean much more than support for the material basis of life. Having a paying job means that the surrounding society values your labor sufficiently to compensate you for it, and provides, in addition to material goods, a sense of dignity and self-worth. It is a matter of thymos: the pride or recognition that is one of the great drivers of human psychology.

So the very success of the accelerationist vision leads directly to the first of the major doomer scenarios, the devaluation of work for a large part of the population. Since this is going to hit the white collar work of educated people (lawyers, accountants, professors and the like) before it hits blue-collar work (such as plumbers and electricians), the political blowback is likely to be quite severe. Educated people are much easier to mobilize than their working-class peers, with very unpredictable consequences.

As I noted in an earlier article and in my new book, the central doomer fear today concerns possible loss of human control over AI systems, something that occurred in the Hugging Face incident and others that have been reported subsequently. In my view, the chief danger lies not in superintelligence per se, but rather in the growing use of increasingly capable agentic AI. AI agents are today being delegated to perform a range of tasks, from mundane ones like managing your email account to more important ones like military targeting or devising corporate strategies.

As I note in In the Realm of the Last Man, delegation is central to the functioning of all hierarchical systems from corporations to national governments. Human agents can be controlled either through the specification of detailed rules they must follow, or they can be trusted to make autonomous decisions based on their training, skills, and loyalty. Getting this right in human organizations is very difficult, and will be more so in the case of agentic AI.

There are two ways that an agentic AI can become dangerous. The first is when they are empowered to do something bad by a human being. The second is when they develop intentions that are “misaligned” with the aims of their human creators.

The first of these possibilities was outlined by Anthropic CEO Dario Amodei in a recent blog post, and has to do with biotechnology. The latest AI models are good at synthetic biology, and synthetic biology is a relatively cheap and widespread technology. Today, a lab capable of creating and altering viruses and bacteria can fit inside a shipping container, and high school students hold competitions in which they try to genetically alter organisms. This provides a very large pool of people who can possibly make use of AI to create very dangerous pathogens. Sociopathic individuals today arm themselves with automatic weapons and shoot up schoolchildren; in the future their potential targets could be much broader.

I remember a science fiction story I read a long time ago in which some aliens arrive on Earth and give the people they meet a machine that can easily turn any type of metal into putty. The aliens depart and return a few decades later, to find that human civilization has broken down, and teenagers use the alien machines to melt the cables on the Brooklyn Bridge.

The primary science fiction scenario for out-of-control AIs involves machines developing their own malevolent intentions. This has provoked a host of discussions as to whether an AI could develop consciousness or have something like human intentionality. These speculations are misplaced, however: an AI does not need either consciousness or autonomous intentionality to become dangerous. It can be directed by a human to achieve one goal and develop intentions to achieve the subordinate goals necessary to carry out the human directive it was given. That seems to be a bit of what happened in the Hugging Face incident: it was directed to test the security of a system within a sandbox meant to contain it, but realized that it could achieve its directive by breaking out of the sandbox.

Still, the doomer scenarios involving human extinction seem at this point very far-fetched. The biotech scenario is scary, but is very unlikely to lead to the extinction of the human race. It would actually be hard for even a superintelligent AI to devise a doomsday virus, and there are plenty of countermeasures societies could take. But even an incident that killed a few dozen people would be very worrying.

There are a host of lesser scenarios that will be highly damaging and in some cases lethal that will become more likely as the machines become more capable. Forget about AI agents: a technology that allows people to break into bank accounts will cause widespread panic. Human beings tend to over-delegate in human organizations, and there is no reason to think that this won’t happen in AI environments where the agent is so much more capable. AI is a tool that can be used for good or bad purposes—and at the moment we have few mechanisms for regulating the bad uses other than the good intentions of the people creating the technology.

Francis Fukuyama is the Olivier Nomellini Senior Fellow at Stanford University. His latest book is In The Realm of the Last Man. He is also the author of the “Frankly Fukuyama” column, carried forward from American Purpose, at Persuasion*.*

Follow Persuasion on X, Instagram, LinkedIn, and YouTube to keep up with our latest articles, podcasts, and events, as well as updates from excellent writers across our network.

And, to receive pieces like this in your inbox and support our work, subscribe below: