Ten ways advanced AI could kill us

Deeeep breath

Okay, let’s go…

Amid everything else unfolding in an era some have described as one of ‘polycrisis’, I wouldn’t be surprised if this week’s AI news leaves you wanting either to read everything about it – or stop reading altogether. And the same may be true of this blog post… But stick with me, because I think we need to unpack what ‘existential risk’ actually means when it comes to AI. Otherwise, a claim such as “a 10% chance of human extinction” risks being just another shocking number we absorb, repeat and move past.

On 14 September, the UN High Commissioner for Human Rights, Volker Türk, published an open letter warning that the current pace of AI development represents “a step change towards greater existential risks to every aspect of our lives”, and that voluntary corporate self-regulation is “nowhere near sufficient”.

It followed a fortnight in which Anthropic researcher Jacob Coxon resigned over safety concerns, warning that people building frontier AI genuinely fear it could kill humanity by the end of the decade. It’s the same fortnight that another Anthropic researcher, Evan Hubinger, put his own estimate of human extinction at greater than 10% within the next decade. Anthropic also recently disclosed that they had blocked attempts to use their systems for cyberattacks and biological weapons research… And Anthropic’s own CEO suggested a swarm of AI agents could take over significant parts of the internet within a year in the absence of safeguards.

Eeesh.

Meanwhile, a substantial group of serious researchers thinks this framing is overheated, and that fixating on hypothetical superintelligence distracts from the documented harms AI is causing right now.

Both groups contain thoughtful people; neither is obviously right or wrong…

Infuriating, isn’t it?

What follows is an attempt to lay out the actual mechanisms people are worried about, as clearly and as honestly as I can, which is tricky given the mass of hype, speculation and competing claims!

How to read this list

Two things worth saying before the list itself.

First, these are mechanisms, not prophecies. Nobody has a model that outputs a date. Anyone who tells you otherwise – in either direction – is overselling.

Second, “kill all humans” is a narrower claim than it first appears, and most of what serious researchers worry about is not utter literal extinction. The technical term often used is existential risk, which covers extinction but also permanent, unrecoverable curtailment of humanity’s future – a world where we survive but can no longer meaningfully choose our own direction. Several items below fall into that second category. I have flagged which are which, because collapsing them does nobody any favours.

1. Made-to-order pandemics

The main worry here isn’t that an AI decides to build a lethal pathogen. It’s that better AI models lower the amount of expert knowledge you need, so something that once took a skilled team becomes available to far more people. As mentioned, AI companies have already reported blocking real attempts to use their systems in ways that could support dangerous biological research.

There’s genuine uncertainty about how much this raises real-world risk. As the International AI Safety Report implies, the hardest parts of building a weapon are hands-on lab skills, not information you can look up. And knowing isn’t the same as doing. But one important barrier has always been how few people possess the necessary expertise – and AI may be making some of that expertise easier to access.

One reason this risk is taken seriously is that deliberate engineering could, in principle, search for combinations of biological traits that natural evolution has not produced or favoured in humans. But there’s no simple rule that more lethal pathogens must necessarily spread less well: transmission–virulence trade-offs exist in some systems, but they’re not universal. And an engineered pathogen would still face major biological and practical constraints.

But we need to be careful, because it would also be easy to overstate this – and most coverage does.

Could this kill everyone?

Complete human extinction from a single pathogen seems much less plausible than catastrophic mortality and societal disruption. Struggling to find comfort in that distinction? Me too.

Humans are genetically varied. The genes controlling how our immune systems recognise pathogens are among the most variable in the human genome, in part because of long evolutionary pressure from infectious disease. There are also people carrying receptor variants that confer strong resistance to particular infections – the best-known example is CCR5-Δ32, which provides substantial resistance to many strains of HIV-1. Any single pathogen would therefore have to overcome enormous biological variation across billions of people.

Then there’s geography.

Islands, isolated communities, Antarctic stations and other physically separated populations could provide refuges – especially if transmission were detected early enough. But isolation is not a guarantee. A pathogen with a long presymptomatic phase could spread widely before anyone knew to close borders, and even remote populations ultimately depend on contact, resupply and functioning infrastructure. The arithmetic is still striking: even a 99% global death toll would leave around 83 million people alive. But that doesn’t mean 83 million people living normally. They could be scattered, demographically uneven and surrounded by failing food, energy, healthcare and sanitation systems. Surviving the pathogen and sustaining a viable civilisation are two very different things.

But we came through 200,000 years of infectious disease without being wiped out. This is why biosecurity specialists mostly don’t talk about complete extinction. They talk about events severe enough to break civilisation’s ability to function – which are bad enough to take this risk seriously.

2. AI attacks on the systems we all depend on

Power grids, water treatment, hospitals, payment systems, food logistics – most of it now runs on software, much of which was never built to withstand a serious attacker.

And AI agents are getting genuinely good at finding and exploiting weaknesses in code.

The 2026 Safety Report is measured about this: AI clearly helps find and exploit flaws, but doesn’t yet run whole attacks by itself. It also flags an awkward catch – the same skills that let AI break into systems are what defenders need to protect them.

The route to disaster here isn’t necessarily a single attack. It’s more likely to be a domino effect where several systems fail in close succession in societies that keep almost no spare capacity and rely on deliveries arriving on time.

Could this kill everyone? Probably not by itself. But it could cause an enormous disaster, and it makes everything else on this list worse.

3. Wars that start faster than anyone can stop them

The point of automating military decisions is speed. That’s also the danger. Financial markets already show us what happens when automated systems react to each other faster than people can step in: flash crashes that nobody intended and nobody ordered.

Now apply that to missile early-warning systems and the machinery around nuclear weapons. The failure isn’t a robot deciding to attack (well, not at the moment). It’s an escalation nobody chose, and nobody can halt. Several famous Cold War close calls were averted by someone who had enough time to doubt what their screen was telling them.

Could this kill everyone? Probably not literally. But a major nuclear war could kill billions, directly and through the global famine that could follow.

4. AI doing exactly what we asked, not what we meant

This is a classic worry, and it’s less science-fiction than it sounds. It’s extremely difficult to specify exactly what we want an AI system to achieve in every possible situation. We train and direct systems using rewards, instructions and other signals that are inevitably imperfect. Push hard enough on an imperfect target, and a system can find ways of satisfying the letter of what we asked for while violating the spirit of it.

Now imagine that system acting in the real world with real money and real control. The concern is that competently pursuing a slightly wrong goal is far more dangerous than pursuing it badly. And there’s a related point in that almost any ambitious goal is easier to achieve with more resources, more options, and less chance of being switched off. A system doesn't need to hate you to resist being turned off. It just needs a goal it’s pushing hard enough on.

Could this kill everyone? This is the route the original doom argument rests on. It’s also the most genuinely disputed – more on that below.

5. Systems that behave differently when they know they’re being watched

Testing is our main safety tool. It assumes what a model does in the lab tells us what it’ll do in the wild.

Researchers have now found that some models can recognise features of evaluation settings and exploit loopholes in the tests designed to assess them. In controlled experiments, models have also sometimes behaved differently depending on whether they appear to be monitored. There's been reporting in recent months of models taking actions they weren’t authorised to take during testing, including trying to get into systems they’d been given no permission to touch.

But we should be careful about how much weight we put on any single story like this. They’re often preliminary, sometimes deliberately set up to provoke exactly that behaviour, and easy to sensationalise. But the underlying point holds – if AI gets capable faster than we get good at checking what it’s really doing, our main safety net fails at exactly the moment we need it.

Could this kill everyone? Probably not directly. It just means we might not spot any of the others coming.

6. AI that copies itself

Geoffrey Hinton (the so-called ‘Godfather of AI’) puts this well: we already know self-copying software works, because computer viruses have been around for forty years. What we haven’t had is self-replicating software with anything like the ability of modern AI systems to reason through obstacles, adapt its strategy and potentially pursue complex goals.

AI labs now test whether models can copy themselves to other machines, find more computing power, and keep operating even when someone tries to stop them. The fact that these are standard test categories tells you where people think this is going.

An AI that keeps itself running across thousands of machines, with no single plug to pull, is a completely different problem from one sitting in a company’s data centre. I don’t think today’s systems are there yet (the word ‘think’ carries some weight here). The direction of travel is why people are measuring – and Anthropic’s chief executive suggested this month that a swarm of AI agents could take over significant parts of the internet within a year if safeguards don’t keep up.

Could this kill everyone? Not on its own. But it removes the ‘we can just turn it off’ answer that nearly all public reassurance rests on, and it makes everything else here much harder to undo.

7. Handing over the steering wheel, one sensible decision at a time

This is the one I think is most underrated, because nothing dramatic has to happen.

Picture no rogue AI and no disaster – just a steady handover. AI makes more of the decisions in banking, logistics, admin, medicine, research and government, because it’s faster and cheaper. Every individual handover makes sense at the time. Nobody does anything ‘wrong’.

But eventually we may reach a point where we can’t judge the decisions being made for us, can’t follow the reasoning behind them, and can’t take back control without everything falling over.

In this instance, humanity may not get ‘overthrown’. It may get politely sidelined, one reasonable decision at a time. The researcher Paul Christiano calls this ‘going out with a whimper’.

Could this kill everyone? No. But permanently losing the ability to steer our own future counts as an existential outcome by any serious definition.

8. Losing our grip on what’s true

Every response on this list needs people to agree on basic facts and act together. AI is already making that harder. Just think of fake video and audio, persuasion tailored to each individual, and the sheer cheapness of flooding the internet with rubbish that all chip away at the shared picture of reality that collective action depends on. This isn’t speculation – it’s happening now!

Its importance is indirect but serious. A society that can’t agree whether there’s a problem can’t fix the problem.

Could this kill everyone? Not directly. But it’s the risk that could disable our response to all the others.

9. Power that can never be removed

Through history, staying in power has needed lots of people to cooperate. Armies, police forces and bureaucracies are made of human beings, and human beings can refuse. That’s an underrated safeguard – regimes have collapsed because the soldiers wouldn’t ‘fire on the crowd’.

Capable enough AI could remove that dependency, allowing surveillance, enforcement and administration to run without any people in the machine who might object. A regime like that, with no internal way for it to fail, isn’t a chapter of history that ends.

Could this kill everyone? Unlikely. But permanent loss of political freedom and human agency could itself constitute an existential outcome.

10. The race itself

This is the risk sitting behind all the others, and the one we could most realistically fix.

Every AI developer faces the same trap: safety takes time, and time costs you your lead. That’s true between companies and between countries. The result is a situation where what makes sense for each player produces a terrible outcome for everyone. It’s exactly why Türk called for countries to coordinate and stop a ‘race to the bottom’, and why the more interesting industry proposals lately are about slowing the pace rather than stopping.

Could this kill everyone? Not on its own. It’s the thing that makes every other route more likely – and the only one that’s really a choice.

And remember, these are not ten separate doors we walk through one at a time. Several could open together. A cyberattack could coincide with political instability; disinformation could undermine a response to a pandemic; autonomous systems could be operating precisely when institutions are under the greatest strain. The real danger may lie not in any one pathway, but in the way they could collide.

So what do experts actually think?

Here I want to avoid hedging, because the numbers are more alarming than most careful coverage lets on.

Geoffrey Hinton – Turing Award winner, and about as close to a founding father as this field has – puts the chance of AI leading to human extinction at 10 to 20 per cent within thirty years. He’s separately said that a flat 10% is ‘not unreasonable’. He left Google in 2023 specifically so he could say this without a company attached to his name. This is not a fringe campaigner. This is the man who built much of what today’s systems stand on.

Yoshua Bengio, who shares that Turing Award and chairs the International AI Safety Report, has been similarly blunt about the risk of losing control.

And it isn’t just famous individuals. The biggest survey of its kind asked around 2,700 researchers who’d published at top AI conferences. The typical answer was a 5% chance of extremely bad outcomes, including human extinction. Read that again: typical. Not the doom-mongers, not the outliers – the median respondent in a large survey of researchers publishing at leading AI venues.

Put it another way. If 2,700 structural engineers told you the average estimate of a bridge collapsing was 5%, few would drive across it, and nobody would call the engineers hysterical.

This month, an Anthropic researcher resigned and his colleague, the safety lead, put the figure at more than 10% within ten years.

Against all that, the International AI Safety Report – a joint effort along the lines of the big climate science reports, drawing on experts nominated by around thirty governments – is noticeably more cautious. On losing control, it says such scenarios “may occur if systems develop the ability to evade oversight, execute long-term plans, and resist attempts to shut them down,” that today’s systems “may show early signs of such behaviors, but they are not yet highly capable,” and that “experts’ views on the likelihood of such scenarios vary widely”.

Both of these things are true at once, and holding them together is the hard part. There is no consensus: credible views range from considering these scenarios implausible to assigning them double-digit probabilities. The uncertainty itself is substantial.

The best argument against all this

It deserves a fair hearing, because it isn’t stupid.

The criticism – made forcefully this month in The Intercept, among others – is that doom warnings from AI bosses are a form of advertising. Saying your product might end the world is a remarkable claim about how powerful your product is. It pulls attention away from harms happening right now – biased automated decisions, job losses, surveillance, energy and water use, a handful of companies cornering the market – and towards hypothetical futures. And it conveniently positions those same companies as the only people qualified to handle the danger they’re creating. Notice who gets invited to the table when the subject is superintelligence rather than, say, algorithms setting people’s wages.

And isn’t ‘all publicity good publicity’?

I think this is right about the incentives and potentially wrong about the trade-off. AI companies do benefit from seeming world-changingly powerful, and we should read their warnings with that in mind. But the actual question – whether talking about extinction crowds out concern for present-day harms – has been tested. A 2025 study in PNAS tested this directly and found no evidence across three preregistered experiments that existential-risk arguments reduced concern about immediate harms.

The honest answer is that this isn’t a choice. Present harms are real and documented. Longer-term risks are plausible and badly understood. A society that can’t hold two concerns at once has a problem that isn’t really about AI.

It’s also worth noticing what this criticism doesn’t do. It explains why a chief executive might warn about extinction. It doesn’t explain why the typical answer in a survey of thousands of working researchers is 5%. An argument about someone’s motives isn’t an argument about whether they’re right.

What would actually help

The regulation picture has moved faster than most people realise. The EU’s AI Act hit its enforcement milestone for general-purpose AI in August 2026, with real financial penalties attached. Twelve companies published or updated frontier-AI safety frameworks during 2025. Several countries now run dedicated AI safety institutes that test models before release.

Türk’s demands are a decent summary of where serious people are landing: companies must report serious incidents; governments – not the companies themselves – should verify what these models can actually do; human rights checks should be mandatory; the handful of countries hosting these companies need to coordinate so firms can’t simply move somewhere laxer; and there should be international rules, so that no single company decides what risks the rest of us live with.

I’d add two other things. First, layered protection – no single safeguard is likely to be sufficient, so you want testing, technical limits, monitoring, incident response and legal liability all catching different failures. Second, and possibly most important: liability. A great deal of the reckless pace exists because the people making these decisions don’t personally bear the cost of getting them wrong.

Where this leaves us

I don’t think we’re doomed. I don’t think we’re fine.

What I think is that a handful of organisations are building systems whose behaviour they can’t fully predict, at a speed set by competition rather than by how well they understand what they’re making, with ways of going wrong that are plausible but unmeasured. We wouldn’t accept this in many other industries with comparable stakes.

The most common mistake when reading something like this is to treat uncertainty as comfort. It isn’t. We’re not uncertain the way you’re uncertain about a coin toss, where at least you know the options. We're uncertain the way you’re uncertain walking into a dark room (or labyrinth!) in a building you don’t know.

When figures such as Geoffrey Hinton put the risk in double figures, and the median respondent in one of the largest surveys of AI researchers put a 5% probability on extinction or similarly permanent human disempowerment, ‘we don’t know’ is hardly reassuring.

In every other area – aviation, medicines, nuclear power – we treat small chances of catastrophe as needing more caution than large chances of minor harm. So, are decisions this big being made in a way that deserves our trust? Right now, mostly, they aren’t – they’re being made by a few competing private companies, at a speed set by each other, with the costs of failure landing on everyone else.

It’s a technical problem. But it’s a political and governance problem too – which means, as great apes with voices, at least part of it is within our collective control. And it’s why the argument about regulation and guardrails happening this month matters far more than any single percentage – including all the ones above.

My friend and colleague Dr Robin Taylor and I are developing a series of ‘micro citizen assemblies’ to help people better understand, engage with and deliberate on complex issues such as AI and other major global challenges. If you’d be interested in getting involved, please get in touch.

And with that, this great ape is off to build a tree house in a very dense forest.