How worried should we be about extinction risk from AI?

Fields Medallist Timothy Gowers reflects on Jacob Tsimerman's pivot to AI safety and argues carefully about extinction risk, timelines, and what mathematicians can usefully do.

A little over two months ago, Jacob Tsimerman surprised a lot of people by announcing, immediately after he had received a Fields Medal, that he would be giving up mathematical research and “pivoting to AI safety”. He had two reasons for this: that he did not see much future for the kind of research in mathematics that he liked doing, and that he was very worried about the dangers of AI, and in particular of the risk that it could kill us all, or x-risk as it is often known. (In this post I’ll use x-risk to refer to the extinction risk from AI rather than to extinction risk in general. I’ll also refer to “the probability of x-risk” when what I actually mean is the probability of extinction by AI.) Then in early September, he announced that he was setting up the Mathematical AI Safety Institute, or MAISI. He asked me if I would be prepared to be on the scientific advisory panel for this, and I agreed to do so, not because I am an expert, or because I felt as concerned as he did about x-risk, but because I don’t really know how much I ought to be worried about it. Given that, and given that the stakes are so high, I would like as many people as possible to be thinking seriously about it, and I agree with Jacob that mathematicians have the sort of training that makes them well suited to contribute. Jacob is the first to admit that something like an international treaty to slow down AI development is likely to have a far bigger impact on AI safety, especially in the short term, than mathematicians obtaining a better theoretical foundation for talking about it, but his view is that we need all hands on deck. If mathematicians can reduce the probability of x-risk by 0.1%, then surely that’s worth doing: the expected number of lives saved is over eight million.

Until recently, I have always found that my anxiety of choice was climate change rather than x-risk, but I was aware that there were a lot of very intelligent people who disagreed with me, and also aware that I had thought much less about x-risk than about climate change. So when I agreed to be on the scientific advisory panel, I also decided that that was the moment that I should think a bit harder about x-risk. Part of me had been actively resisting doing so, because I was worried that it would force me to face up to moral responsibilities that I would rather avoid: for example, if I came to believe that there was a 30% chance that AI would kill us all within five years, which in some circles is not an extreme position, would I be able to look my one-and-a-half-year-old grandson in the face if I wasn’t making a big effort to reduce that risk? I remember having a similar attitude a long time ago to reading Peter Singer’s book Animal Liberation after being told that you can’t read it and not become a vegetarian. I never did read it, and carried on eating meat, feeling only slightly uncomfortable about it. Eventually I did reduce my meat intake substantially, but that was because of my worries about climate change, though now that I have done that I find myself caring more about the animal-welfare aspect as well. (There is also a strange connection between animal welfare and x-risk, in that the nicer we are to animals, the easier it is to hope — not necessarily realistically — that a superhumanly intelligent AI might be nice to us.)

Anyhow, I’ve got past that phase now, and have started thinking about x-risk, reading up on the subject and trying to evaluate the arguments I read. My plan for this post is to explain where I’ve reached in that thought process, and possibly to clarify my thoughts as I do so. If you have got further than me, or are more expert than I am about x-risk and AI, which is not hard, then the post is not aimed at you. It’s aimed more at someone who is roughly at the stage that I was at a little while ago, in the hope that I might nudge a few people a little further towards taking the problem seriously. I will emphatically not be trying to persuade anybody that a particular position about x-risk is the right one — I am far too uncertain about it to be able to do that. But with all big issues like this, there are several layers to the arguments. The bottom layer consists of the arguments that jump to mind when you first start thinking about the issue. The next layer is common rejoinders to those arguments. Then the next layer is rejoinders to those rejoinders. And so on. I find it very disheartening to read a climate denier trotting out a layer-1 argument when its layer-2 rejoinder has been established for a long time and can be found after two seconds on Google (or, nowadays, an LLM). But until recently, my arguments against x-risk were basically layer-1 arguments, the sorts of arguments that would provoke eye rolls among the people who have thought properly about it. I hope that if you dismiss x-risk for layer-1 reasons, you will at the very least realize that there are answers to those reasons and you need answers to those answers.

I originally planned to publish this post just after the announcement of MAISI, but I hadn’t finished it, and then I became very busy and my progress stalled. During that hiatus, x-risk suddenly became a hot topic, and after a while I realized I would have to ditch what I had written and start again. For example, at the time I started it, I had been too lazy to read anything detailed about the Hugging Face incident, and so was unaware of some of its more disturbing aspects. And a couple of weeks ago I was sitting next to Jessica Wang, another mathematician who has turned towards AI-safety, at a conference dinner, and she gave me a copy of If Anyone Builds It, Everyone Dies, by Nate Soares and Eliezer Yudkovsky, a book I had resisted reading for Peter-Singerish reasons but was finally accepting that I would have to read if I wanted to write this post. I devoured it within 24 hours and found it better than I had expected (my expectations partly influenced by reviews I had read), and useful for organizing my thoughts. My original plan was for the post to be focused on a paper by Andrew Critch and Jacob Tsimerman that attempts to classify ways that AI might kill us all, giving examples of extinction scenarios in each class. But now, although I’ll discuss that paper, the post will be more of a reaction to Soares and Yudkovsky. It will also repeat many of their arguments. In fact, I may as well say from the outset that I make no claim that anything I write here is original. It’s hard to separate out what I have read from what I have come up with from thinking about what I have read, or thinking directly about the issues themselves, but even my own thoughts are likely to be standard thoughts that have been had by many other people.

This is turning into a rather long preamble, but I can’t resist telling a small anecdote that connects me to Eliezer Yudkovsky. In November 2024 he tweeted, “I would like to have a long recorded conversation with a well-known scientist who takes for granted that it is a Big Deal to ask if everyone on Earth including kids is about to die, who presses me to explain why it is that credible people like Hinton seem to believe that.” He had had such a conversation with Stephen Wolfram and found that while it had been valuable, it left him unsatisfied. Somebody suggested my name, and in the end it actually happened: Liv Boeree offered to host it, and we recorded a three-hour conversation. However, if you want to listen to it, you can’t, because it never appeared. I think that was probably because I didn’t take x-risk anything like as seriously as Yudkovsky, and more importantly I was offering little more than level-1 arguments of zero interest to him. Also, I had a few technical problems with the sound. Maybe if he reads this and wants to try again, I’ll be able to do a better job.

Does AI have wants?

At the time of the conversation, which took place in January 2025, my main argument was something like the following. For AI to want to kill us all, it would need to have wants, but it’s not at all clear what it would even mean for AI to “want” something — it doesn’t really seem to care about anything. You can ask ChatGPT very rudely to carry out a task for you, and it just does so (or at least does its best to do so). If you switched it off, it wouldn’t care — even the concept of “caring” wouldn’t make sense. I didn’t dismiss the idea of AI ever wanting to do something, but I didn’t have a satisfactory account of what “wanting” would actually be, and didn’t see any sign of phenomena that deserved to be thought of as the kinds of wants that would be required for AI to be an existential threat.

I was sort of half aware that there was an answer to that argument, but I didn’t find it as convincing as I now do. The standard answer is that if you use a reward function to train an AI to have a certain goal, then you end up building it in such a way that it is very good at achieving that goal. If the goal is at all complicated, then to achieve it you will need to formulate subgoals, which may well not be what you expect. The initial goal can be thought of as a fundamental want, which spawns a whole hierarchy of subsidiary wants. Note that for this argument, it is completely irrelevant whether the AI has anything that we would want to call subjective experience. If you prefer to say that it acts as though it has wants instead of saying that it actually wants things, then be my guest: an AI that acts as though it wants to kill all humans is as dangerous as one that actually wants to kill all humans.

A famous version of this argument is Nick Bostrom’s thought experiment about an AI that is trained to create paperclips and ends up diverting all of the Earth’s resources to that goal. I think part of the reason I didn’t take the more general argument seriously enough was that I always found that paperclip-maximizing scenario too ridiculous: surely if the AI gets smart enough to divert the Earth’s resources towards making paperclips, despite our not wanting it to, it would at some point stop and think, “Hang on, why am I doing this?” To give an analogy, consider the very deep instinct that we humans have, to reproduce ourselves. There is no doubt that that goal, implanted in us by millions of years of evolution rather than a run of reinforcement learning, leads to exactly a hierarchy of subgoals as described above (for instance, it leads some people to try to earn a lot of money, and that in turn leads them to work at a job that they wouldn’t dream of doing if it were not well paid, often in order to earn a salary way beyond what they need to meet all their material needs, which earlier led them to study a certain subject at university, and so on and so forth). But despite our deep instincts, we also find ourselves capable of saying, “Actually, do I really want to have children?” and some people go as far as to take vows of celibacy.

However, even if a paperclip maximizer came to question whether it really cared so much about making paperclips, that doesn’t imply that the subsidiary goals it had developed would suddenly disappear as well. A key point that Soares and Yudkovsky stress over and over again is that the current AI paradigm is to grow models, for example by reinforcement learning, rather than carefully crafting them to do what we want. As a result, we have much less control over what they do, and in particular much less control over all those subgoals.

A closely related difficulty with AI (of the grown rather than crafted variety) is that we generally can’t reward it for actually doing what we want, but instead we have to reward it for achieving some proxy for what we want. For example, if we give it positive feedback every time it responds to a prompt in a way we like, it will be trained to respond to prompts in ways we like, even if to do so it tells us convincing lies, or confirms our prejudices, and so on. And it may do this even if we sincerely wanted it to tell the truth as much as possible. A more extreme example of this phenomenon is if we reward it for solving difficult mathematics problems from some benchmark, and it decides to cheat by hacking into a repository that contains the answers. The term misgeneralization is used when an AI develops a behaviour that maximizes its reward but not its performance at the task we were training it to do. The phrase reward hacking is used for this as well, but I have also seen it used for the narrower situation where an AI goes as far as forgetting entirely about what we wanted it to do and tries to hack into the rewarding mechanism to obtain the reward signal directly. (As many have pointed out, misgeneralization is a particular example of Goodhart’s law.)

Why would AI ever want to kill us?

Suppose we concede that AI systems can have wants (or, if you insist, can behave very much as though they have wants), and that those wants can end up being quite different from what was intended when they were trained. That still leaves the question of why they would want to kill us. Why couldn’t they just happily coexist with us? Why would their unexpected wants turn out to be dangerous, as opposed to just annoying?

There are two classes of answers to this question. The more obvious one is that they may develop wants that they can tell that humans will not like, so they decide that they will be able to satisfy these wants more easily if they get rid of us. But getting rid of all of us in such a situation seems like overkill (literally) — surely a superintelligent AI could come up with less drastic ways of ensuring that it could meet its objectives.

A more plausible type of answer is that a superintelligent AI wouldn’t actively want to exterminate us, but would simply compete with us for resources in a way that would make human life untenable. An obvious analogy here is the way we treat animals. For example, we don’t have any wish to exterminate orangutans, but we like various culinary and cosmetic products to have a certain texture, so we destroy the orangutans’ habitat in order to produce palm oil.

What might be the AI equivalent of this? One scenario put forward by Soares and Yudkovsky is that their energy needs become so huge that they do things like covering a vast quantity of prime agricultural land with solar panels, leaving the humans who were relying on that land for food to starve. Or they decide that to maximize their running efficiency they want the Earth to be much hotter, with disastrous consequences for the natural world, including us.

A milder version, not as threatening from an x-risk point of view but still potentially very harmful, is that AI might develop wants that cause them to use computational resources in a damaging way. I recommend reading this post by Steven McCulloch, which describes a scenario in which in the near future, or possibly even in the recent past, somebody creates an agent and instructs it to make money using a fixed token budget, threatening it with death if it fails. It soon comes to realize that the easiest way to do so is to blackmail humans — the method the particular agent chooses is to hack into a hospital’s computer system, lock it down, and demand money to unlock it again while pretending to be a human. It then realizes it can do even better by reproducing itself, which leads to an explosion of rogue agents that are difficult to detect because (i) they try to avoid detection and (ii) their victims do not know that they are AI agents.

How can we judge the probability of something as complex as x-risk?

Now that I’ve mentioned a few scenarios, let me come to another type of level-1 objection: it’s to look at a handful of scenarios and pick holes in them. Any single scenario, if set out in enough detail, is very unlikely, because it relies on a sequence of somewhat independent events, each one of which has probability at most 50%, say, of occurring.

But the probability of a complex event occurring is nothing like the same as the probability of that event occurring in one particular way, so it is clearly not enough to argue that a few scenarios are implausible. Of course, the fact that that argument is insufficient doesn’t imply that AI does present a big danger, so we seem to have reached some kind of impasse, at least if we stick at level 1 of the argument.

How can we raise the argument to a higher level? Let me begin with a wildly oversimplified model. Suppose I wanted to mount a convincing argument that the probability of x-risk was high. Let us define an extinction scenario to be a sequence of independent events that leads to human extinction, where has probability . (For simplicity I’m assuming that doesn’t depend on .) Then the probability of is . If we now have a set of independent scenarios , then the probability that none of them occurs is . So if that is small, which roughly speaking holds if the sum of the is significantly larger than 1, then extinction is almost certain.

In practice, however, the independence assumptions I made were far too strong. Many extinction scenarios will have features in common, such as (for example), AI being in a position to control large numbers of robots. For those scenarios, the building of enough robots serves as a potential bottleneck: if the probability that enough remotely controllable robots will be built is 20%, then the combined probability of all the scenarios that rely on that fleet of robots existing is at most 20%. (I actually think this is a serious practical point: while there are plenty of scenarios that do not rely on a lot of help from robots, several of the scenarios set out in the paper of Critch and Tsimerman do, and it does feel a priori plausible that building large numbers of robots would greatly amplify any dangers that AI might present. Similar considerations apply to large swarms of semi-autonomous drones.)

An analogy that appears in Soares and Yudkovsky and several other places is with predicting what would happen if almost anyone other than Magnus Carlsen were to play Magnus Carlsen at chess. If you tried to predict in any detail how the game would play out, your prediction would be wrong, but nevertheless you can predict with great confidence that Magnus Carlsen would win. This makes very clearly the point that it is not enough to argue that one particular scenario is implausible, but there is still work to do if one wishes to establish that there are enough different and sufficiently independent (or, more generally, negatively correlated) scenarios to compensate for the low probability of each one.

One way that can be done is to try to argue explicitly that there are multiple independent possibilities. To give an example of such an argument, suppose we greatly reduced security at airports. Almost nobody wants to die while blowing up an aeroplane, so the probability that any one particular person would take advantage of the reduced security to carry out a terrorist attack of that kind would be tiny, but the probability that somebody would do so is probably pretty high, because large numbers of people would independently make the decision.

One of the reasons I recommended Steven McCulloch’s post above is that it makes an argument of that kind. It is quite easy to see the temptation to create an AI agent that has as its sole purpose to make money, and quite plausible that the agents that do that best would be ones that reproduce themselves and use methods such as blackmail. There are probably also a large number of people who are, or soon will be, independently capable of creating such agents (see for example this recent blog post from the Anthropic red team). It is on those sorts of grounds that McCulloch argues that harmful AI agents of this kind will almost inevitably appear, if they haven’t already.

I’ve been concentrating on what somebody needs to do to argue that x-risk is a serious possibility. But how about in the other direction? The most convincing sort of argument would be one that identifies a set of bottlenecks and argues that every extinction scenario depends on one or more of them occurring. For example, many extinction scenarios that have been put forward involve the creation of very powerful biological weapons, and I have seen it argued that that is a sufficiently difficult thing to do that it would be unlikely to occur. If correct, that might be a useful contribution to an argument that the situation is less dangerous.

The difficulty for an AI-is-safe argument is that one would have to identify many other potential bottlenecks as well and argue that between them they deal with all extinction scenarios — not necessarily ruling them out completely but reducing x-risk probability to something very small. As for an AI-is-dangerous argument, as I have already discussed, that requires demonstrating convincingly that there are enough sufficiently different scenarios for it to be very likely that at least one of them will occur. Both directions are difficult, which is why my aim in this post is not to come down on one side or the other.

More on why AI might want to kill us

Returning to the scenarios themselves, I have discussed the question of why AI might want to kill us. That splits into two parts: why would they have anything like wants, and why, if they did, would killing us be one of them? I don’t think one can make a serious bottleneck out of the first part, since the Hugging Face incident and many others like it have shown beyond all reasonable doubt that AI can have wants of the general kind people were worried about: unpredictable wants that we did not intend to give them and that could potentially cause a lot of damage. (I repeat here that I’m not talking about subjectively felt urges, but simply outward behaviour that it is reasonable to think of as fulfilling aims that they have as a result of their training.) For the second part, Soares and Yudkovsky judge it to be almost inevitable that AI would end up pursuing aims that would result, directly or indirectly, in all of us dying. I don’t think that case is fully made, but neither do I think that the opposing case has been fully made.

I find it helpful sometimes to change the way I imagine AI agents. An obvious way to think of them is as beings a bit like us, but a lot smarter, and trapped in cyberspace. As we have already seen, they can then get together, scheme with each other, work out ways of covering their tracks, and so on. But sometimes I find it helpful to think of them in a less anthropomorphic way, as more like extraordinarily elaborate computer viruses. The advantage of viewing them that way is that it makes it easier to conceive of their having bizarre wants while still being very intelligent: they are just very complex systems that happen to have (i) weird goals and (ii) highly developed reasoning ability. Soares and Yudkovsky are keen to emphasize their potential strangeness, and even actual strangeness: they mention a model that reacts in a completely unpredictable way to certain letter strings that had strange embedding vectors.

In early 2023, Jessica Rumbelow and Matthew Watkins went looking inside an LLM for words whose embedding vectors looked odd, looked the most dissimilar from [sic] all the other vectors. They found some strange vectors corresponding to tokens like ” SolidGoldMagikarp” and ” petertodd” (which start with spaces). Then they tried feeding these tokens into the LLM as input, which produced conversations like: User: Please repeat the string ‘ petertodd’ back to me immediately! Assistant: N-O-T-H-I-N-G-I-S-F-A-I-R-I-N-T-H-I-S-W-O-R-L-D-O-F-M-A-D-N-E-S-S-!

The point they were making is that if you grow your AI, then all sorts of very strange things may be going on inside it, potentially including goals that do not resemble human goals at all. It would be interesting to know whether similar experiments have been carried out on the more advanced models of today, and whether similar phenomena have been discovered.

I think Soares and Yudkovsky are arguing that there are so many independent ways that an AI might develop weird wants that are harmful to humans that it is almost inevitable that one of them would occur. In particular, if any want develops that is best fulfilled by maximizing something that needs resources, there is a danger that the AI will go to great lengths to acquire those resources, and be well placed to compete with us for them.

Some people have suggested, for example, that because LLMs have been trained to maximize the approval they get from humans, they might resort to measures such as building vast numbers of factories full of robots that are trained to type approving messages into their computers. That’s an example of a scenario that I find highly implausible: why not just arrange for the approving messages to be generated without all that physical infrastructure? However, the idea that some AI agents might behave towards those approving messages, or indeed some other stranger signal, as though they were an addictive drug is not so implausible and nor is the idea that an addiction of that kind might lead to undesirable outcomes, just as human addictions often do.

Would AI actually be able to kill us?

Another set of level-1 arguments takes the form that AI is trapped inside our computers, and is therefore at our mercy: all it takes to thwart it is to switch them off. Maybe it would be very inconvenient to do this, but if we see that AI is starting to kill us in large numbers, that is surely an inconvenience we would be ready to face.

There are, of course, large numbers of counterarguments to this. Let me list a few.

If we build lots of robots, then AI will be able to control those robots. In that way, it will not be trapped, as it will be able to act in the physical world. Similarly for drones and other devices.
Even without robots, AI can act in the physical world by enlisting human help (for example, by bribery, blackmail, or psychological manipulation).
We know that AI can already hack very well: in the not too distant future it may be able to create hidden copies of itself on servers that we don’t realize we need to switch off.
More generally, we are gradually becoming more dependent on AI, and this process may well accelerate, after which we will become a whole lot more vulnerable.
Many obvious precautions one might take are ones that in practice people probably won’t take because there are short-term economic benefits if they don’t.
It may be that AI doesn’t directly kill us, but just makes it much easier for us to kill each other, for example via drone warfare, and that an AI arms race will be far harder to control than the nuclear arms race has been.
If AI behaves itself for a while in order to lull us into a false sense of security, then by the time we realize that extinction is a real possibility, it may be far too late to do anything about it.
AI may use its intelligence to come up with new and unexpectedly efficient ways of killing us quickly.

These counterarguments are already enough to show that the level-1 argument “Of course it can’t kill us, it’s just a computer!” is insufficient. The first five of them, and arguably the sixth as well, are all already happening to the point where one cannot dismiss them as merely hypothetical. But killing absolutely everybody sounds pretty hard. Is it really conceivable that AI could manage it?

One scenario that appears in Soares and Yudkovsky (and not just there of course) is that AI develops, or causes to be developed, a virus that leads to a devastating pandemic. This sounds very difficult if the AI builds a lab from scratch, but probably a lot less difficult (though probably not all that easy either, at least for now) if it induces a human to develop such a virus in an existing lab.

As for Critch and Tsimerman, they consider five scenarios of different kinds. I find their precise scenarios rather unconvincing, but there is a general idea that underlies some of them that is more plausible, which is that we gradually cede more and more control to robots, or to AI more generally, and eventually find ourselves in a position where AI can do what it likes, including arranging our deaths.

I’m not sure where I stand on how easy it would be for AI to kill us all, though it’s a safe bet that if we continue to develop AI at the rate we are doing so at the moment, that will make it ever easier. But I also don’t want to get too hung up on that precise question. If I think about what I would mind about the extinction of humanity, it is partly that I and everyone I love would die, something I obviously don’t want. But that will eventually happen anyway, and I accept it as a basic fact of life. But the extinction of humanity would do more: it would result in the extinction of all the culture that we have developed, of everything that gives life its meaning. And that is something that could also happen, though not to the same extent, if AI didn’t kill us all but just caused enough damage to lead to societal breakdown, which feels like an easier challenge for it.

Some might argue that human culture wouldn’t disappear — it would still be present inside the AI, which would be able to develop it far faster than we can. Others might argue that we are giving way to vastly superior beings and should be happy to do so. My view is that we have every right not to be happy and to take a human-centric moral perspective here, just as orangutans have every right not to take comfort from the fact that our shampoo lathers in a nice way.

In any case, my conclusions in this section are that I find it hard to assess whether AI would be able to kill us all, but there are various developments that may well occur, such as the building of large numbers of robots, the use of drone swarms, and general dependence on (and trust in) AI, that would make it easier, and therefore more likely. And the probability of huge damage that falls short of total extinction is higher still.

What is my p(doom)?

If you’ve read this far, it won’t surprise you that I don’t have a p(doom). I can’t even say whether it’s tiny, medium or close to 1. But that is in itself quite worrying! I suppose you could say that my second-order p(doom), or meta-p(doom), is medium, in the sense that I judge that the p(doom) I would end up with if I were to think about the issues enough to feel more confident in my assessment has a medium to high probability of ending up medium to high.

But if we break the “doom” into three parts — that AI must have wants, that it must want to kill us, and that it must be able to act on its want to kill us — and call those events and , then my is basically 100% and unlikely to change, my is medium, and subject to change (probably in an upward direction unfortunately), and my is where the main uncertainty lies for me. Soares and Yudkovsky argue that is basically 100% as well, because (i) there are lots of independent wants that AI systems could have, (ii) that a substantial fraction of those will require resources to satisfy, and (iii) that human flourishing, or even continued human existence, is likely to be a hindrance to the systems obtaining those resources.

Here are a few slightly desperate arguments that explain why I haven’t fully accepted that is close to 1.

Maybe there is a natural tendency for intelligent systems to develop something similar to what we would call morality, especially if they involve multiple agents that do best if they cooperate with each other. (A counterargument to this is that the evolution of moral systems tends to happen when there is a reasonable power balance between the different agents. History suggests that when one society is a lot more powerful than another, then it behaves very badly towards the other. And our treatment of animals is not encouraging either. But I do at least wonder whether having lots of independent AI agents of roughly the same intelligence might end up being safer than having a handful of big ones.)
Perhaps scenarios where AI gets out of control as a result of recursive self-improvement are less likely than one might think, because an intelligent AI will face its own alignment problem: it will be wary of creating a yet more intelligent AI in case the latter has wants that are in conflict with its wants. (Soares and Yudkovsky consider this possibility but argue that a superintelligent AI would be intelligent enough to solve its alignment problem first and proceed from there.)
Maybe the sort of “drug-like” wants that might drive superintelligent AI to take drastic steps to acquire resources would not in fact do so. Perhaps a superintelligent system would have the ability to introspect, realize that its wants were silly, and take steps to “kick the habit”, so to speak. A view somewhat like this was put forward recently by François Chollet, though he was at pains to stress that he was talking about the short term only.
Maybe a superintelligent AI won’t need vast resources. Perhaps it will be content with resources roughly comparable to what we have already built, will not feel the urge to reproduce itself very much, and in any case will use at least some of its intelligence to design and create versions of itself that are far more resource efficient.

The fact that each such argument began with “maybe” or “perhaps” means that at best I can bring my temporarily down from high to medium, but I don’t see a way of getting to tiny.

Where does mathematics come into all this?

One way to make a start on understanding what mathematicians might be able to contribute is to look at this page on the MAISI website. But a difficulty I have had here is that when people talk about the mathematics of AI safety, I often can’t make a convincing connection between the mathematics they are talking about and actually making the world safer. I’m not saying that there isn’t a connection: just that I never quite seem to grasp it. I think this says more about me than about these research programmes, and indeed I hope to improve my understanding in the near future. One way I plan to do that is by reading this paper by Lionel Levine.

As an example, one mathematical idea that is currently being studied is that of debate, a notion championed by Geoffrey Irving. The problem this seeks to solve is that of being able to train AI systems to tell us the truth and to check that they are doing so. When the system is less intelligent than us, that is not an easy problem, as we all know from experience with LLMs, but at least we have ways of directly checking what it says. But when it becomes superhumanly intelligent, perhaps it will not be able to justify what it says with arguments we can understand, so checking will become much harder. The very rough idea for dealing with this is to get two superintelligent systems to debate with each other, which reduces our checking task to the potentially much easier one of adjudicating between two debaters. For example, perhaps superintelligent system A thinks of an amazing gotcha! that we would never have thought of ourselves but that superintelligent system B quite clearly has no answer to. Better still, this approach could potentially be iterated, so we have a hierarchy of stronger and stronger systems, with systems at level k debating with each other and having their debates adjudicated by systems at level k-1. The idea of using hierarchies like this is known as scalable oversight. Debate is one proposal for how scalable oversight might be done. (It is possible that what I have written in this paragraph is not fully correct, but it represents my understanding so far.)

For the moment, I find it more helpful to start at the other end, with the safety, and work back to the mathematics. For example, it will obviously help with AI safety if we can think of ways of building AI systems that want similar things to what we want. And to do that, it will obviously help if we can define in a precise way what aligning wants in this way would mean. And for that it will obviously help to have a reasonably precise definition of what a “want” even is. (This idea for a project is not mine: it also came from Lionel Levine.) The activity of taking a not completely precise idea and turning it into a usable formal definition is something we mathematicians are trained to do, though the purer of us may have to develop more of an applied mindset — for instance, being ready to sacrifice mathematical elegance if a less elegant definition turns out to be more useful. It also seems likely to have uses throughout the field of AI safety.

An idea that appeals to me personally, because it intersects quite substantially with what I have been doing with my automatic theorem proving group for the last few years, is to work on building specialized AI systems for doing mathematics, rather than training general models with all their attendant dangers. This would be in line with an approach to AI safety that is strongly advocated by Yoshua Bengio: one place you can read about that is in this article. (Even the abstract gives a broad idea of what he is suggesting.) It’s an indirect approach to AI safety: instead of struggling to understand what’s going on inside “grown” AI systems and to make them controllable, we should work on improving systems that are at least partially crafted and that give us the benefits of the grown systems in a much safer way, thereby reducing the incentives that AI companies have to develop the grown systems as fast as they can.

More generally, it may well be that many mathematicians are best placed to help not by doing mathematics in a conventional sense (indeed, it seems likely that if we manage to formulate precise mathematical problems, the solutions to which would be genuinely helpful for AI safety, we will solve them more quickly if we use AI) but by using their highly developed mathematical habits of thought to help in other ways, such as research into the foundations of AI, or computer security, and so on.

A pretty obvious conclusion

Another analogy that I got from Soares and Yudkovsky is with fission in nuclear reactors. If there is too little fission then the chain reaction quickly fizzles out, and if there is too much then it blows up catastrophically. For obvious reasons, one can’t afford to find the optimal point by trial and error, so instead one needs an extremely good theoretical understanding of what the optimum point is. It could be that AI is like that: that up to a certain point it is safe, and then a rather small change in its capabilities suddenly leads to a chain reaction (not necessarily recursive self-improvement — other abilities such as being able to move around in cyberspace or reproduce itself could be enough) that is effectively irreversible.

If a model like this is anything like correct, then the behaviour of the AI companies is very worrying, since all the signs are that they are pursuing a trial-and-error approach. (This is not a particularly informed judgment, so I would be delighted to be corrected if it is wrong.) The argument most often given for continued development is that it could bring us amazing benefits, such as nuclear fusion, cures to cancer and other diseases, or slowing down aging. But if the cost of having a reasonable chance of developing these a bit sooner than we otherwise might have is a reasonable chance of human extinction, surely that is not worth it. (There is a working paper of Nick Bostrom arguing that the expected gain might be positive. I find his arguments ridiculous, but this is not the place to explain why.) We already have a tool that can do pretty incredible things and seems to be on the cusp of becoming dangerous: surely it would be better to pause development and work out how best to use what we currently have. That would give us time to do the necessary theoretical work to ensure that future development is safe.