The Mythology Of Conscious AI

The cultural history of synthetic consciousness is both long and mostly unhappy. From Yossele the Golem, to Mary Shelley’s “Frankenstein,” HAL 9000 in “2001: A Space Odyssey,” Ava in “Ex Machina,” and Klara in “Klara and The Sun,” the dream of creating artificial bodies and synthetic minds that both think and feel rarely ends well — at least, not for the humans involved. One thing we learn from these stories: If artificial intelligence is on a path toward real consciousness, or even toward systems that persuasively seem to be conscious, there’s plenty at stake — and not just disruption in job markets.

Some people think conscious AI is already here. In a 2022 interview with The Washington Post, Google engineer Blake Lemoine made a startling claim about the AI system he was working on, a chatbot called LaMDA. He claimed it was conscious, that it had feelings, and was, in an important sense, like a real person. Despite a flurry of media coverage, Lemoine wasn’t taken all that seriously. Google dismissed him for violating its confidentiality policies, and the AI bandwagon rolled on.

But the question he raised has not gone away. Firing someone for breaching confidentiality is not the same as firing them for being wrong. As AI technologies continue to improve, questions about machine consciousness are increasingly being raised. David Chalmers, one of the foremost thinkers in this area, has suggested that conscious machines may be possible in the not-too-distant future. Geoffrey Hinton, a true AI pioneer and recent Nobel Prize winner, thinks they exist already. In late 2024, a group of prominent researchers wrote a widely publicized article about the need to take the welfare of AI systems seriously. For many leading experts in AI and neuroscience, the emergence of machine consciousness is a question of when, not if.

How we think about the prospects for conscious AI matters. It matters for the AI systems themselves, since — if they are conscious, whether now or in the future — with consciousness comes moral status, the potential for suffering and, perhaps, rights.

It matters for us too. What we collectively think about consciousness in AI already carries enormous importance, regardless of the reality. If we feel that our AI companions really feel things, our psychological vulnerabilities can be exploited, our ethical priorities distorted, and our minds brutalized — treating conscious-seeming machines as if they lack feelings is a psychologically unhealthy place to be. And if we do endow our AI creations with rights, we may not be able to turn them off, even if they act against our interests.

Perhaps most of all, the way we think about conscious AI matters for how we understand our own human nature and the nature of the conscious experiences that make our lives worth living. If we confuse ourselves too readily with our machine creations, we not only overestimate them, we also underestimate ourselves.

The Temptations Of Conscious AI

Why might we even think that AI could be conscious? After all, computers are very different from biological organisms, and the only things most people currently agree are conscious are made of meat, not metal.

The first reason lies within our own psychological infrastructure. As humans, we know we are conscious and like to think we are intelligent, so we find it natural to assume the two go together. But just because they go together in us doesn’t mean that they go together in general.

Intelligence and consciousness are different things. Intelligence is mainly about doing: solving a crossword puzzle, assembling some furniture, navigating a tricky family situation, walking to the shop — all involve intelligent behavior of some kind. A useful general definition of intelligence is the ability to achieve complex goals by flexible means. There are many other definitions out there, but they all emphasize the functional capacities of a system: the ability to transform inputs into outputs, to get things done.

An artificially intelligent system is measured by its ability to perform intelligent behavior of some kind, though not necessarily in a humanlike form. The concept of artificial general intelligence (AGI), by contrast, explicitly references human intelligence. It is supposed to match or exceed the cognitive competencies of human beings. (There’s also artificial superintelligence, ASI, which happens when AI bootstraps itself beyond our comprehension and control. ASI tends to crop up in the more existentially fraught scenarios for our possible futures.)

Consciousness, in contrast to intelligence, is mostly about being. Half a century ago, the philosopher Thomas Nagel famously offered that “an organism has conscious mental states if and only if there is something it is like to be that organism.” Consciousness is the difference between normal wakefulness and the oblivion of deep general anesthesia. It is the experiential aspect of brain function and especially of perception: the colors, shapes, tastes, emotions, thoughts and more, that give our lives texture and meaning. The blueness of the sky on a clear day. The bitter tang and headrush of your first coffee.

AI systems can reasonably lay claim to intelligence in some form, since they can certainly do things, but it is harder to say whether there is anything-it-is-like-to-be ChatGPT.

The propensity to bundle intelligence and consciousness together can be traced to three baked-in psychological biases.

The first is anthropocentrism. This is the tendency to see things through the lens of being human: to take the human example as definitional, rather than as one example of how different properties might come together.

The second is human exceptionalism: our unfortunate habit of putting the human species at the top of every pile, and sometimes in a different pile altogether (perhaps closer to angels and Gods than to other animals, as in the medieval Scala naturae). And the third is anthropomorphism. This is the tendency to project humanlike qualities onto nonhuman things based on what may be only superficial similarities.

Taken together, these biases explain why it’s hardly surprising that when things exhibit abilities we think of as distinctively human, such as intelligence, we naturally imbue them with other qualities we feel are characteristically or even distinctively human: understanding, mindedness and consciousness, too.

One aspect of intelligent behavior that’s turned out to be particularly effective at making some people think that AI could be conscious is language. This is likely because language is a cornerstone of human exceptionalism. Large Language Models (LLMs) like OpenAI’s ChatGPT or Anthropic’s Claude have been the focus of most of the excitement about artificial consciousness. Nobody, as far as I know, has claimed that DeepMind’s AlphaFold is conscious, even though, under the hood, it is rather similar to an LLM. All these systems run on silicon and involve artificial neural networks and other fancy algorithmic innovations such as transformers. AlphaFold, which predicts protein structure rather than words, just doesn’t pull our psychological strings in the same way.

The language that we ourselves use matters too. Consider how normal it has become to say that LLMs “hallucinate” when they spew falsehoods. Hallucinations in human beings are mainly conscious experiences that have lost their grip on reality (uncontrolled perceptions, one might say). We hallucinate when we hear voices that aren’t there or see a dead relative standing at the foot of the bed. When we say that AI systems “hallucinate,” we implicitly confer on them a capacity for experience. If we must use a human analogy, it would be far better to say that they “confabulate.” In humans, confabulation involves making things up without realizing it. It is primarily about doing, rather than experiencing.

When we identify conscious experience with seemingly human qualities like intelligence and language, we become more likely to see consciousness where it doesn’t exist, and to miss seeing it where it does. We certainly should not just assume that consciousness will come along for the ride as AI gets smarter, and if you hear someone saying that real artificial consciousness will magically emerge at the arbitrary threshold of AGI, that’s a sure sign of human exceptionalism at work.

There are other biases in play, too. There’s the powerful idea that everything in AI is changing exponentially. Whether it’s raw compute as indexed by Moore’s Law, or the new capabilities available with each new iteration of the big tech foundation models, things surely are changing quickly. Exponential growth has the psychologically destabilizing property that what’s ahead seems impossibly steep, and what’s behind seems irrelevantly flat. Crucially, things seem this way wherever you are on the curve — that’s what makes it exponential. Because of this, it’s tempting to feel like we are always on the cusp of a major transition, and what could be more major than the creation of real artificial consciousness? But on an exponential curve, every point is an inflection point.

Finally, there’s the temptation of the techno-rapture. Early in the movie “Ex Machina,”the programmer Caleb says to the inventor Nathan: “If you’ve created a conscious machine — it’s not the history of man, that’s the history of Gods.” If we feel we’re at a techno-historical transition, and we happen to be one of its architects, then the Promethean lure must be hard to resist: the feeling of bringing to humankind that which was once the province of the divine. And with this singularity comes the signature rapture offering of immortality: the promise of escaping our inconveniently decaying biological bodies and living (or at least being) forever, floating off to eternity in a silicon-enabled cloud.

Perhaps this is one reason why pronouncements of imminent machine consciousness seem more common within the technorati than outside of it. (More cynically: fueling the idea that there’s something semi-magical about AI may help share prices stay aloft and justify the sky-high salaries and levels of investment now seen in Silicon Valley. Did someone say “bubble”?)

In his book “More Everything Forever,” Adam Becker describes the tendency to project consciousness into AI as a form of pareidolia — the phenomenon of seeing patterns in things, like a face in a piece of toast or Mother Teresa in a cinnamon bun (Figure 1). This is an apt description. But helping you recognize the power of our pareidolia-inducing psychological biases is just the first step in challenging the mythology of conscious AI. To address the question of whether real artificial consciousness is even possible, we need to dig deeper.

Consciousness & Computation

The very idea of conscious AI rests on the assumption that consciousness is a matter of computation. More specifically, that implementing the right kind of computation, or information processing, is sufficient for consciousness to arise. This assumption, which philosophers call computational functionalism, is so deeply ingrained that it can be difficult to recognize it as an assumption at all. But that is what it is. And if it’s wrong, as I think it may be, then real artificial consciousness is fully off the table, at least for the kinds of AI we’re familiar with.

Challenging computational functionalism means diving into some deep waters about what computation means and what it means to say that a physical system, like a computer or a brain, computes at all. I’ll summarize four related arguments that undermine the idea that computation, at least of the sort implemented in standard digital computers, is sufficient for consciousness.

1: Brains Are Not Computers

First, and most important, brains are not computers. The metaphor of the brain as a carbon-based computer has been hugely influential and has immediate appeal: mind as software, brain as hardware. It has also been extremely productive, leading to many insights into brain function and to the vast majority of today’s AI. To understand the power and influence of this metaphor, and to grasp its limitations, we need to revisit some pioneers of computer science and neurobiology.

Alan Turing towers above everyone else in this story. Back in the 1950s, he seeded the idea that machines might be intelligent, and more than a decade earlier, he

formulated a definition of computation that has remained fundamental to our technologies, and to most people’s understanding of what computers are, ever since.

Turing’s definition of computation is extremely powerful and highly (though, as we’ll see, not completely) general. It is based on the abstract concept of a Turing machine: a simple device that reads and writes symbols on an infinite tape according to a set of rules. Turing machines formalize the idea of an algorithm: a mapping, via a sequence of steps, from an input (a string of symbols) to an output (another such string); a mathematical recipe, if you like. Turing’s critical contribution was to define what became known as a universal Turing machine: another abstract device, but this time capable of simulating any specific Turing machine — any algorithm — by taking the description of the target machine as part of its input. This general-purpose capability is one reason why Turing computation is so powerful and so prevalent. The laptop computer I’m writing with, as well as the machines in the server farms running whatever latest AI model, are all physical, concrete examples of (or approximations to) universal Turing machines, bounded by physical limitations such as time and memory.

Another major advantage of this framework, from a practical engineering point of view, is the clean separation it licenses between abstract computation (software) and physical implementation (hardware). An algorithm (in the sense described above) should do the same thing, no matter what computer it is running on. Turing computation is, in principle, substrate independent: it does not depend on any particular material basis. In practice, it’s better described as substrate flexible, since you can’t make a viable computer out of any arbitrary material — cheese, for instance, isn’t up to the job. This substrate-flexibility makes Turing computation extremely useful in the real world, which is why computers exist in our phones rather than merely in our minds.

At around the same time that Turing was making his mark, the mathematician Walter Pitts and neurophysiologist Warren McCulloch showed, in a landmark paper, that networks of highly simplified abstract neurons can perform logical operations (Figure 2). Later work, by the logician Stephen Kleene among others, demonstrated that artificial neural networks like these, when provided with a tape-like memory (as in the Turing machine), were “Turing complete” — that they could, in principle, implement any Turing machine, any algorithm.

Put these ideas together, and we have a mathematical marriage of convenience and influence, and the kind of beauty that accompanies simplicity. On the one hand, we can ignore the messy neurobiological reality of real brains and treat them as simplified networks of abstract neurons, each of which just sums up its inputs and produces an output. On the other hand, when we do this, we get everything that Turing computation has to offer — which is a lot.

The fruits of this marriage are most evident in its children: the artificial neural networks powering today’s AI. These are direct descendants of McCulloch, Pitts and Kleene, and they also implement algorithms in the substrate-flexible Turing sense. It is hardly surprising that the seductive impressiveness of the current wave of AI reinforces the idea that brains are nothing more than carbon-based versions of neural network algorithms.

But here’s where the trouble starts. Inside a brain, there’s no sharp separation between “mindware” and “wetware” as there is between software and hardware in a computer. The more you delve into the intricacies of the biological brain, the more you realize how rich and dynamic it is, compared to the dead sand of silicon.

Brain activity patterns evolve across multiple scales of space and time, ranging from large-scale cortical territories down to the fine-grained details of neurotransmitters and neural circuits, all deeply interwoven with a molecular storm of metabolic activity. Even a single neuron is a spectacularly complicated biological machine, busy maintaining its own integrity and regenerating the conditions and material basis for its own continued existence. (This process is called autopoiesis, from the Greek for “self-production.” Autopoiesis is arguably a defining and distinctive characteristic of living systems.)

Unlike computers, even computers running neural network algorithms, brains are the kinds of things for which it is difficult, and likely impossible, to separate what they do from what they are.

Nor is there any good reason to expect such a clean separation. The sharp division between software and hardware in modern computers is imposed by human design, following Turing’s principles. Biological evolution operates under different constraints and with different goals. From the perspective of evolution, there’s no obvious selection pressure for the kind of full separation that would allow the perfect interoperability between different brains as we enjoy between different computers. In fact, the opposite is likely true: Maintaining a sharp software/hardware division is energetically expensive, as is all too apparent these days in the vast energy budgets of modern server farms.

This matters because the idea of the brain as a meat-based (universal) Turing machine rests precisely on this sharp separation of scales, on the substrate independence that motivated Turing’s definition in the first place. If you cannot separate what brains do from what they are, the mathematical marriage of convenience starts to fall apart, and there is less reason to think of biological wetware as there simply to implement algorithmic mindware. Evidence that the materiality of the brain matters for its function is evidence against the idea that digital computation is all that counts, which in turn is evidence against computational functionalism.

Another consequence of the deep multiscale integration of real brains — a property that philosophers sometimes call “generative entrenchment” — is that you cannot assume it is possible to replace a single biological neuron with a silicon equivalent, while leaving its function, its input-output behavior, perfectly preserved.

For example, the neuroscientists Chaitanya Chintaluri and Tim Vogels found that some neurons fire spikes of activity apparently to clear waste products created by metabolism. Coming up with a perfect silicon replacement for these neurons would require inventing a whole new silicon-based metabolism, too, which just isn’t the kind of thing silicon is suitable for. The only way to seamlessly replace a biological neuron is with another biological neuron — and ideally, the same one.

This reveals the weakness of the popular “neural replacement” thought experiment, most commonly associated with Chalmers, which invites us to imagine progressively replacing brain parts with silicon equivalents that function in exactly the same way as their biological counterparts. The supposed conclusion is that properties like cognition and consciousness must be substrate independent (or at least silicon-substrate-flexible). This thought experiment has become a prominent trope in discussions of artificial consciousness, usually invoked to support its possibility. Hinton recently appealed to it in just this way, in an interview where he claimed that conscious AI was already with us. But the argument fails at its first hurdle, given the impossibility of replacing any part of the brain with a perfect silicon equivalent.

There is one more consequence of a deeply scale-integrated brain that is worth mentioning. Digital computers and brains differ fundamentally in how they relate to time. In Turing-world, only sequence matters: A to B, 0 to 1. There could be a microsecond or a million years between any state transition, and it would still be the same algorithm, the same computation.

By contrast, for brains and for biological systems in general, time is physical, continuous and inescapable. Living systems must continuously resist the decay and disorder that lies along the trajectory to entropic sameness mandated by the inviolable second law of thermodynamics. This means that neurobiological activity is anchored in continuous time in ways that algorithms, by design, are not. (This is another reason why digital computation is so energetically expensive. Computation exists out of time, but computers do not. Making sure that 1s stay as 1s and 0s stay as 0s takes a lot of energy, because not even silicon can escape the tendrils of entropy.)

What’s more, many researchers — especially those in the phenomenological tradition — have long emphasized that conscious experience itself is richly dynamic and inherently temporal. It does not stutter from one state to another; it flows. Abstracting the brain into the arid sequence space of algorithms does justice neither to our biology nor to the phenomenology of the stream of consciousness.

Metaphors are, in the end, just metaphors, and — as the philosopher Alfred North Whitehead pointed out long ago — it’s always dangerous to confuse a metaphor with the thing itself. Looking at the brain through “Turing glasses” underestimates its biological richness and overestimates the substrate flexibility of what it does. When we see the brain for what it really is, the notion that all its multiscale biological activity is simply implementation infrastructure for some abstract algorithmic acrobatics seems rather naı̈ve. The brain is not a Turing machine made of meat.

2: Other Games In Town

In the previous section, I noted that Turing computation is powerful but limited. Turing computations — algorithms — map one finite range of discrete numbers (more generally, a string of symbols) onto another, with only the sequence mattering. Turing algorithms are powerful, but there are many kinds of dynamics, many other kinds of functions, that go beyond this kind of computation. Turing himself identified various non-computable functions, such as the famous “halting problem,” which is the problem of determining, in general, whether an algorithm, given some specific input, will ever terminate. What’s more, any function that is continuous (infinitely divisible) or stochastic (involving inherent randomness), strictly speaking, lies beyond Turing’s remit. (Turing computations can approximate or simulate these properties to varying extents, but that’s different from the claim that such functions are Turing computations. I’ll return to this distinction later.)

Biological systems are rife with continuous and stochastic dynamics, and they are deeply embedded in physical time. It seems presumptuous at the very least to assume that only Turing computations matter for consciousness, or indeed for many other aspects of cognition and mind. Electromagnetic fields, the flux of neurotransmitters, and much else besides — all lie beyond the bounds of the algorithmic, and any one of them may turn out to play a critical role in consciousness.

These limitations encourage us to take a broader view of the brain, moving beyond what I sometimes call “Turing world” to consider how broader forms of computation and dynamics might help explain how brains do what they do. There is a rich history here to draw on, and an exciting future too.

The earliest computers were not digital Turing machines but analogue devices operating in continuous time. The ancient “Antikythera mechanism,” used for astronomical purposes and dating back to around 2,000 BCE, is an excellent example. Analogue computers were again prominent at the birth of AI in the 1950s, in the guise of the long-neglected discipline of cybernetics, where issues of control and regulation of a system are considered more important than abstract symbol manipulation.

Recently, there’s been a resurgence in neuromorphic computation, which leverages more detailed properties of neural systems, such as the precise timing of neuronal spikes, than the cartoon-like simulated neurons that dominate current artificial neural network approaches. And then there’s the relatively new concept of “mortal computation” (introduced by Hinton), which stresses the potential for energy saving offered by developing algorithms that are inseparably tied to their material substrates, so that they (metaphorically) die when their particular implementation ceases to exist. All these alternative forms of computation are more closely tied to their material basis — are less substrate-flexible — than standard digital computation.

Many systems do what they do without it being reasonable or useful to describe them as being computational at all. Three decades ago, the cognitive scientist Tim van Gelder gave an influential example, in the form of the governor of a steam engine (Figure 3). These governors regulate steam flow through an engine using simple mechanics and physics: as engine speed increases, two heavy cantilevered balls swing outwards, which in turn closes a valve, reducing steam flow. A “computational governor,” sensing engine speed, calculating the necessary actions and then sending precise motor signals to switch actuators on or off, would not only be hopelessly inefficient but would betray a total misunderstanding of what’s really going on.

The branch of cognitive science generally known as “dynamical systems,” as well as approaches that emphasize enactive, embodied, embedded and extended aspects of mind (so-called 4E cognitive science), all reject, in ways relating to van Gelder’s insight, the idea that mind and brain can be exhaustively accounted for algorithmically. They all explore alternatives based on the mathematics of continuous, dynamical processes — involving concepts such as attractors, phase spaces and so on. It is at least plausible that those aspects of brain function necessary for consciousness also depend on non-computational processes like these, or perhaps on some broader notion of computation.

These other games in town are all still compatible with what in philosophy is known as functionalism: the idea that properties of mind (including consciousness) depend on the functional organization of the (embodied) brain. One of the factors contributing to confusion in this area has been a tendency to conflate the rather liberal position of functionalism-in-general, since functional organization can include many things, with the very specific claim of computational functionalism, which implies that the type of organization that matters is computational and which in turn is often assumed to relate to Turing-style algorithms in particular.

The challenge for machine consciousness here is that the further we venture from Turing world, the more deeply entangled we become in randomness, dynamics and entropy, and the more deeply tied we are to the properties of a particular material substrate. The question is no longer about which algorithms give rise to consciousness; it’s about how brain-like a system has to be to move the needle on its potential to be conscious.

3: Life Matters

My third argument is that life (probably) matters. This is the idea — called biological naturalism by the philosopher John Searle— that properties of life are necessary, though not necessarily sufficient, for consciousness. I should say upfront that I don’t have a knock-down argument for this position, nor do I think any such argument yet exists. But it is worth taking seriously, if only for the simple reason mentioned earlier: every candidate for consciousness that most people currently agree on as actually being conscious is also alive.

Why might life matter for consciousness? There’s more to say here than will fit in this essay ( I wrote an entire book, “Being You,” and a recent research paper on the subject), but one way of thinking about it goes like this.

The starting point is the idea that what we consciously perceive depends on the brain’s best guesses about what’s going on in the world, rather than on a direct readout of sensory inputs. This derives from influential predictive processing theories that understand the brain as continually explaining away its sensory inputs by updating predictions about their causes. In this view, sensory signals are interpreted as prediction errors, reporting the difference between what the brain expects and what it gets at each level of its perceptual hierarchies, and the brain is continually minimizing these prediction errors everywhere and all the time.

Conscious experience in this light is a kind of controlled hallucination: a top-down inside-out perceptual inference in which the brain’s predictions about what’s going on are continually calibrated by sensory signals coming from the bottom-up (or outside-in).

This kind of perceptual best-guessing underlies not only experiences of the world, but experiences of being a self, too — experiences of being the subject of experience. A good example is how we perceive the body, both as an object in the world and as the source of more fundamental aspects of selfhood, such as emotion and mood. Both these aspects of selfhood can be understood as forms of perceptual best-guessing: inferences about what is, and what is not, part of the body, and inferences about the body’s internal physiological condition (the latter is sometimes called “interoceptive inference”; interoception refers to perception of the body from within).

Perceptual predictions are good not only for figuring out what’s going on, but (in a call back to mid-20th century cybernetics) also for control and regulation: When you can predict something, you can also control it. This applies above all to predictions about the body’s physiological condition. This is because the primary duty of any brain is to keep its body alive, to keep physiological quantities like heart rate and blood oxygenation where they need to be. This, in turn, helps explain why embodied experiences feel the way they do.

Experiences of emotion and mood, unlike vision (for example), are characterized primarily by valence — by things generally going well or going badly.

This drive to stay alive doesn’t bottom out anywhere in particular. It reaches deep into the interior of each cell, into the molecular furnaces of metabolism. Within these whirls of metabolic activity, the ubiquitous process of prediction error minimization becomes inseparable from the materiality of life itself. A mathematical line can be drawn directly from the self-producing, autopoietic nature of biological material all the way to the Bayesian best-guessing that underpins our perceptual experiences of the world and of the self.

Several lines of thought now converge. First, we have the glimmers of an explanatory connection between life and consciousness. Conscious experiences of emotion, mood and even the basal feeling of being alive all map neatly onto perceptual predictions involved in the control and regulation of bodily condition. Second, the processes underpinning these perceptual predictions are deeply, and perhaps inextricably, rooted in our nature as biological systems, as self-regenerating storms of life resisting the pull of entropic sameness. And third, all of this is non-computational, or at least non-algorithmic. The minimization of prediction error in real brains and real bodies is a continuous dynamical process that is likely inseparable from its material basis, rather than a meat-implemented algorithm existing in a pristine universe of symbol and sequence.

Put all this together, and a picture begins to form: We experience the world around us and ourselves within it — with, through and because of our living bodies. Perhaps it is life, rather than information processing, that breathes fire into the equations of experience.

4: Simulation Is Not Instantiation

Finally, simulation is not instantiation. One of the most powerful capabilities of universal, Turing-based computers is that they can simulate a vast range of phenomena — even, and perhaps especially, phenomena that aren’t themselves (digitally) computational, such as continuous and random processes.

But we should not confuse the map with the territory, or the model with the mechanism. An algorithmic simulation of a continuous process is just that — a simulation, not the process itself.

Computational simulations generally lack the causal powers and intrinsic properties of the things being simulated. A simulation of the digestive system does not actually digest anything. A simulation of a rainstorm does not make anything actually wet. If we simulate a living creature, we have not created life. In general, a computational simulation of X does not bring X into being — does not instantiate X — unless X is a computational process (specifically, an algorithm) itself. Making the point from the other direction, the fact that X can be simulated computationally does not justify the conclusion that X is itself computational.

In most cases, the distinction between simulation and instantiation is obvious and uncontroversial. It should be obvious and uncontroversial for consciousness, too. A computational simulation of the brain (and body), however detailed it may be, will only give rise to consciousness if consciousness is a matter of computation. In other words, the prospect of instantiating consciousness through some kind of whole-brain emulation, at some arbitrarily high level of detail, already assumes that computational functionalism is true. But as I have argued, this assumption is likely wrong and certainly should not be accepted axiomatically.

This brings us back to the poverty of the brain-as-computer metaphor. If you think that everything that matters about brains can be captured by abstract neural networks, then it’s natural to think that simulating the brain on a digital computer will instantiate all its properties, including consciousness, since in this case, everything that matters is, by assumption, algorithmic. This is the “Turing world” view of the brain.

If, instead, you are intrigued by more detailed brain models that capture the complexities of individual neurons and other fine-grained biophysical processes, then it really ought to be less natural to assume that simulating the brain will realize all its properties, since these more detailed models are interesting precisely because they suggest that things other than Turing computation likely matter too.

There is, therefore, something of a contradiction lurking for those who invest their dreams and their venture capital into the prospect of uploading their conscious minds into exquisitely detailed simulations of their brains, so that they can exist forever in silicon rapture. If an exquisitely detailed brain model is needed, then you are no more likely to exist in the simulation than a hailstorm is likely to arise inside the computers of the U.K. meteorological office.

But buckle up. What if everything is a simulation already? What if our whole universe — including the billions of bodies, brains and minds on this planet, as well as its hailstorms and weather forecasting computers — is just an assemblage of code fragments in an advanced computer simulation created by our technologically godlike and genealogically obsessed descendants?

This is the “simulation hypothesis,” associated most closely with the philosopher Nick Bostrom, and still, somehow, an influential idea among the technorati.

Bostrom notes that simulations like this, if they have been created, ought to be much more numerous than the original “base reality,” which in turn suggests that we may be more likely to exist within a simulation than within reality itself. He marshals various statistical arguments to flesh out this idea. But it is telling that he notes one necessary assumption, and then just takes it as a given. This, perhaps unsurprisingly, is the assumption that “a computer running a suitable program would be conscious” (see page 2 of his paper). If this assumption doesn’t hold, then the simple fact that we are conscious would rule out that we exist in a simulation. That this strong assumption is taken on board without examination in a philosophical discussion that is all about the validity of assumptions is yet another indication of how deeply ingrained the computational view of mind and brain has become. It is also a sign of the existential mess we get ourselves into when we fail to distinguish our models of reality from reality itself.

Let’s summarize. Many social and psychological factors, including some well-understood cognitive biases, predispose us to overattribute consciousness to machines.

Computational functionalism — the claim that (algorithmic) computation is sufficient for consciousness — is a very strong assumption that looks increasingly shaky as the many and deep differences between brains and (standard digital) computers come into view. There are plenty of other technologies (e.g., neuromorphic computing, synthetic biology) and frameworks for understanding the brain (e.g., dynamical systems theory), which go beyond the strictly algorithmic. In each case, the further one gets from Turing world, the less plausible it is that the relevant properties can be abstracted away from their underlying material basis.

One possibility, motivated by connecting predictive processing views of perception with physiological regulation and metabolism, is that consciousness is deeply tied to our nature as biological, living creatures.

Finally, simulating the biological mechanisms of consciousness computationally, at whatever grain of detail you might choose, will not give rise to consciousness unless computational functionalism happens anyway to be true.

Each of these lines of argument can stand up by itself. You might favor the arguments against computational functionalism while remaining unpersuaded about the merits of biological naturalism. Distinguishing between simulation and instantiation doesn’t depend on taking account of our cognitive biases. But taken together, they complement and strengthen each other. Questioning computational functionalism reinforces the importance of distinguishing simulation from instantiation. The availability of other technologies and frameworks beyond Turing-style algorithmic computation opens space for the idea that life might be necessary for consciousness.

Collectively, these arguments make the case that consciousness is very unlikely to simply come along for the ride as AI gets smarter, and that achieving it may well be impossible for AI systems in general, at least for the silicon-based digital computers we are familiar with.

At the same time, nothing in what I’ve said rules out the possibility of artificial consciousness altogether.

Given all this, what should we do?

What (Not) To Do?

When it comes to consciousness, the fact of the matter matters. And not only because of the mythology of ancestor simulations, mind-uploading and the like. Things capable of conscious experiences have ethical and moral standing that other things do not. At least, claims to this kind of moral consideration are more straightforward when they are grounded in the capacity for consciousness.

This is why thinking clearly about the prospects for real artificial consciousness is of vital importance in the here and now. I’ve made a case against conscious AI, but I might be wrong. The biological naturalist position (whether my version or any other) remains a minority view. Other theories of consciousness propose accounts framed in terms of standard computation-as-we-know-it. These theories generally avoid proposing sufficient conditions for consciousness. They also generally sidestep defending computational functionalism, being content instead to assume it.

But this doesn’t mean they are wrong. All theories of consciousness are fraught with uncertainty, and anyone who claims to know for sure what it would take to create real artificial consciousness, or for sure what it would take to avoid doing so, is overstepping what can reasonably be said.

This uncertainty lands us in a difficult position. As redundant as it may sound, nobody should be deliberately setting out to create conscious AI, whether in the service of some poorly thought-through techno-rapture, or for any other reason. Creating conscious machines would be an ethical disaster. We would be introducing into the world new moral subjects, and with them the potential for new forms of suffering, at (potentially) an exponential pace. And if we give these systems rights, as arguably we should if they really are conscious, we will hamper our ability to control them, or to shut them down if we need to.

Even if I’m right that standard digital computers aren’t up to the job, other emerging technologies might yet be, whether alternative forms of computation (analogue, neuromorphic, biological and so on) or rapidly developing methods in synthetic biology. For my money, we ought to be more worried about the accidental emergence of consciousness in cerebral organoids (brain-like structures typically grown from human embryonic stem cells) than in any new wave of LLM.

But our worries don’t stop there. When it comes to the impact of AI in society, it is essential to draw a distinction between AI systems that are actually conscious and those that persuasively seem to be conscious but are, in fact, not. While there is inevitable uncertainty about the former, conscious-seeming systems are much, much closer.

As the Google engineer Lemoine demonstrated, for some of us, such conscious-seeming systems are already here. Machines that seem conscious pose serious ethical issues distinct from those posed by actually conscious machines.

For example, we might give AI systems “rights” that they don’t actually need, since they would not actually be conscious, restricting our ability to control them for no good reason. More generally, either we decide to care about conscious-seeming AI, distorting our circles of moral concern, or we decide not to, and risk brutalizing our minds. As Immanuel Kant argued long ago in his lectures on ethics, treating conscious-seeming things as if they lack consciousness is a psychologically unhealthy place to be.

The dangers of conscious-seeming AI are starting to be noticed by leading figures in AI, including Mustafa Suleyman (CEO of Microsoft AI) and Yoshua Bengio, but this doesn’t mean the problem is in any sense under control.

One overlooked factor here is that even if we know, or believe, that an AI is not conscious, we still might be unable to resist feeling that it is. Illusions of artificial consciousness might be as impenetrable to our minds as some visual illusions. The two lines in the Müller-Lyer illusion (Figure 5) are the same length, but they will always look different. It doesn’t matter how many times you encounter the illusion; you cannot think your way out of it. The way we feel about AI being conscious might be similarly impervious to what we think or understand about AI consciousness.

What’s more, because there’s no consensus over the necessary or sufficient conditions for consciousness, there aren’t any definitive tests for deciding whether an AI is actually conscious. The plot of “Ex Machina” revolves around exactly this dilemma. Riffing on the famous Turing test (which, as Turing well knew, tests for machine intelligence, not consciousness), Nathan — the creator of the robot Ava — says that the “real test” is to reveal that his creation is a machine, and to see whether Caleb — the stooge — still feels that it, or she, is conscious. The “Garland test,” as it’s come to be known, is not a test of machine consciousness itself. It is a test of what it takes for a human to be persuaded that a machine is conscious.

The importance of taking an informed ethical position despite all these uncertainties spotlights another human habit: our unfortunate track record of withholding moral status from those that deserve it, including from many non-human animals, and sometimes other humans. It is reasonable to wonder whether withholding attributions of consciousness to AI may leave us once again on the wrong side of history. The recent calls for attention to “AI welfare” are based largely on this worry.

But there are good reasons why the situation with AI is likely to be different. Our psychological biases are more likely to lead to false positives than false negatives. Compared to non-human animals, the apparent wonders of AI may be more similar to us in ways that do not matter for consciousness, like linguistic ability, and less similar in ways that do, like being alive.

Soul Machine

Despite the hype and the hubris, there’s no doubt that AI is transforming society. It will be hard enough to navigate the clear and obvious challenges AI poses, and to take proper advantage of its many benefits, without the additional confusion generated by immoderate pronouncements about a coming age of conscious machines. Given the pace of change in both the technology itself and in its public perception, developing a clear view of the prospects and pitfalls of conscious AI is both essential and urgent.

Real artificial consciousness would change everything — and very much for the worse. Illusions of conscious AI are dangerous in their own distinctive ways, especially if we are constantly distracted and fascinated by the lure of truly sentient machines. My hope for this essay is that it offers some tools for thinking through these challenges, some defenses against overconfident claims about inevitability or outright impossibility, and some hope for our own human, animal, biological nature. And hope for our future too.

The future history of AI is not yet written. There is no inevitability to the directions AI might yet take. To think otherwise is to be overly constrained by our conceptual inheritance, weighed down by the baggage of bad science fiction and submissive to the self-serving narrative of tech companies laboring to make it to the next financial quarter. Time is short, but collectively we can still decide which kinds of AI we really want and which we really don’t.

The philosopher Shannon Vallor describes AI as a mirror, reflecting back to us the incident light of our digitized past. We see ourselves in our algorithms, but we also see our algorithms in ourselves. This mechanization of the mind is perhaps the most pernicious near-term consequence of the unseemly rush toward human-like AI. If we conflate the richness of biological brains and human experience with the information-processing machinations of deepfake-boosted chatbots, or whatever the latest AI wizardry might be, we do our minds, brains and bodies a grave injustice. If we sell ourselves too cheaply to our machine creations, we overestimate them, and we underestimate ourselves.

Perhaps unexpectedly, this brings me at last to the soul. For many people, especially modern people of science and reason, the idea of the soul might seem as outmoded as the Stone Age. And if by soul what is meant is an immaterial essence of rationality and consciousness, perfectly separable from the body, then this isn’t a terrible take.

But there are other games in town here, too. Long before Descartes, the Greek concept of psychē linked the idea of a soul to breath, while on the other side of the world, the Hindu expression of soul, or Ātman, associated our innermost essence with the ground-state of all experience, unaffected by rational thought or by any other specific conscious content, a pure witnessing awareness.

The cartoon dreams of a silicon rapture, with its tropes of mind uploading, of disembodied eternal existence and of cloud-based reunions with other chosen ones, is a regression to the Cartesian soul. Computers, or more precisely computations, are, after all, immortal, and the sacrament of the algorithm promises a purist rationality, untainted by the body (despite plentiful evidence linking reason to emotion). But these are likely to be empty dreams, delivering not posthuman paradise but silicon oblivion.

What really matters is not this kind of soul. Not any disembodied human-exceptionalist undying essence of you or of me. Perhaps what makes us us harks even further back, to Ancient Greece and to the plains of India, where our innermost essence arises as an inchoate feeling of just being alive — more breath than thought and more meat than machine. The sociologist Sherry Turkle once said that technology can make us forget what we know about life. It’s about time we started to remember.