{"article":{"slug":"watch-a-brain-learn-a-visual-explanation-of-reinforcement-learning","title":"Watch a Brain Learn: A visual explanation of reinforcement learning","subtitle":null,"summary":"An interactive visual explainer that builds a simple digital creature with a brain and watches it learn a new behaviour, illustrating how reinforcement learning works in brains and machines.","content_type":"essay","language":"en","canonical_url":"https://aatishb.com/myriad/how-brains-learn/","author":{"name":"Aatish Bhatia and Arkarup Banerjee","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Myriad","url":"https://aatishb.com/myriad","listing_slug":null,"listing":null},"topics":[{"name":"machine-learning","slug":"machine-learning","url":"https://listedarticles.com/topics/machine-learning"},{"name":"science","slug":"science","url":"https://listedarticles.com/topics/science"},{"name":"education","slug":"education","url":"https://listedarticles.com/topics/education"},{"name":"research","slug":"research","url":"https://listedarticles.com/topics/research"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":2071,"reading_minutes":9,"published_at":"2026-11-13T00:00:00.000Z","added_at":"2026-11-15T01:49:21.970Z","updated_at":"2026-11-15T01:49:21.970Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/watch-a-brain-learn-a-visual-explanation-of-reinforcement-learning","markdown_url":"https://listedarticles.com/articles/watch-a-brain-learn-a-visual-explanation-of-reinforcement-learning.md","example":false,"citation":"Aatish Bhatia and Arkarup Banerjee, Myriad. \"Watch a Brain Learn: A visual explanation of reinforcement learning.\" 13 Nov 2026. https://aatishb.com/myriad/how-brains-learn/ (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://aatishb.com/myriad/how-brains-learn/"},"body_markdown":"# Watch a Brain Learn\n\nA visual explanation of reinforcement learning\n\n \n\nBy [Aatish Bhatia](https://aatishb.com/) and [Arkarup Banerjee](https://www.arkabanerjeelab.com/team)\n\n**How does your brain help you learn a new skill?**\n\nTo try to answer this question, we’ll build a digital being with a simple brain, and watch it learn a new behavior.\n\n[In our previous article](https://aatishb.com/myriad/lets-build-a-brain/), we created a creature called a spark that could steer towards light. And we saw how small changes to its neural circuitry gave rise to very different behaviors.\n\nNeural  \ncircuit\n\nBack then, we’d wired these circuits into the spark’s brain by hand. But now, we want it to learn the same behaviors *by itself*.\n\nAfter all, you didn’t come into this world knowing how to walk, talk or read. While some of your abilities, such as breathing or swallowing, are wired by evolution, you gained others through an arduous process of trial and error.\n\nWe want our spark to do the same. To give you an intuition for this learning process, we’re going to hand over the controls to you. **We’d like** ***you*** **to wire the spark’s brain.**\n\n### You wire its brain\n\nBut first, we need to tell you what you’ll be wiring. The spark’s brain has just four neurons — two neurons serving as its eyes, and two neurons that wiggle its flippers.\n\nThere are **straight connections** between neurons on the same side of its body, **crossed connections** between neurons on opposite sides, and **drive**, which controls the spark’s speed when there’s no light.\n\nstraight\n\ncrossed\n\ndrive\n\nSo its brain wiring boils down to just three numbers — the strength of each kind of connection, and the drive.\n\nNow let’s set this spark loose. Use the sliders below to wire its brain as it moves. We’ll keep score.\n\nYour goal: Make the spark love the light by **getting a score above 0.5.**\n\n### Can you make the spark spend time inside the light?\n\nNeural  \ncircuit\n\n#### Score\n\nTarget 0.11\n\nTries\n\nstraight\n\ncrossed\n\ndrive\n\nThe score represents the average amount of light the spark encounters over 100 trials with these settings.\n\nFinding the right configuration takes a lot of trial and error. (If you get stuck, click the hint button above for a nudge in the right direction.)\n\nLet’s visualize what you just did. To guide the spark to the light, you tweaked three numbers. We can represent these numbers as the three axes of a cube.\n\nThe payoff of this picture is that we can visualize your spark’s learning trajectory as a path within this cube.\n\n#### Your learning path...\n\nTo guide the spark, you probably nudged a slider one way, saw whether this made things better or worse, and then used this feedback to decide your next move.\n\nLet’s distill this into a **learning algorithm** — a step-by-step procedure by which the spark can rewire its brain through practice.\n\nHere’s how it’ll work: The spark gets many tries to steer towards the light. Before every trial, its brain wiring will change a little at random, just like you wiggled the sliders to and fro.\n\nIf the trial goes better than previous ones, then the spark gets a reward and its wiring shifts in the direction of this random change. And if the trial goes worse, the spark gets a punishment, and its wiring shifts the other way.\n\nThis isn’t that different from how you learned how to walk or ride a bike. Falling down was your punishment, and the thrill of balancing was your reward.\n\nCan this simple algorithm help the spark find the light?\n\nFollow the spark’s journey in [Aatish’s newsletter](https://aatishb.com/blog/).\n\n### A spark learns by itself\n\nIn the beginning, the spark is a novice, with a brain that’s wired together at random. Unsurprisingly, it doesn’t get very far.\n\n### Before learning: Stuck\n\nClick to move the light around.\n\nDrag the light to move it around.\n\nThe baby spark can spin in place, but it can’t yet move around. But it’s about to get better…\n\n### After 50 tries: Meandering\n\nClick to move the light around.\n\nDrag the light to move it around.\n\nAfter 50 tries of this learning algorithm, our infant spark has taken its first steps. And while it won’t break any speed records, it’s now on the move…\n\n### After 250 tries: Avoiding\n\nClick to move the light around.\n\nDrag the light to move it around.\n\nAfter 250 tries, our juvenile spark has picked up speed. But it veers away from the light at the last moment.\n\n### After 500 tries: Charging\n\nClick to move the light around.\n\nDrag the light to move it around.\n\nAfter 500 tries, our adolescent spark boldly charges towards the light. But it hasn’t yet figured out how to slow down or stop.\n\n### After 2,000 tries: Expert\n\nClick to move the light around.\n\nDrag the light to move it around.\n\nAfter 2,000 tries, our spark is now an adult — an expert at seeking and following the light. It clings to the light and can turn on a dime. (Try moving the light to see the spark follow it.)\n\nLike before, we can visualize this learning process as a path.\n\n#### The spark’s learning path after 2,000 tries...0 tries...\n\nThis isn’t a path through physical space. Instead, it’s a path through **the space of possible brains**. It shows how the spark’s brain changed as it learned.\n\nOther novice sparks would start from different points, and they would each trace out their own learning trajectory.\n\n#### Many sparks’ learning paths after 6,000 tries...0 tries...\n\nThis tangled web illustrates the learning paths of hundreds of novice sparks, each starting with a differently wired brain. As if by magic, all of these threads converge to the point marked ♥︎.\n\nThe paths aren’t straight lines: Learning isn’t linear. There are setbacks, wrong turns and mistakes along the way. And it isn’t necessarily fast. Some of the learners took thousands of tries to reach their goal.\n\nYou’ll notice that the paths are all jittery. That’s because every step involves randomness. We can smooth out this jitter by averaging over many trajectories at every point, revealing a map of the flow of learning.\n\n### The sparks learn to love\n\nIt’s worth taking a moment to appreciate how surprising this is — given enough tries, the same learning process turned every novice into an expert light seeker.\n\nOf the myriad possible arrangements of this spark’s brain, only a tiny fraction generate the behavior that we want. There are far more ways to fail than to succeed.\n\nAnd yet, you’ve just witnessed a small miracle. The spark navigated this labyrinth of dead ends and rewired its brain to find the light, guided by nothing but reward and punishment.\n\n### Learning new tricks\n\nLearning doesn’t stop once you become an adult. We learn throughout our lives and adapt to changing circumstances.\n\nOur sparks can do this too. For example, what if the light represents a harmful chemical or a dangerous predator instead of a food source?\n\nThe spark is now rewarded for **avoiding the light** rather than for seeking it, and this leads to new behaviors:\n\n### The sparks learn to flee or freeze\n\nIf the light is harmful, then **LOVE** becomes the worst strategy, and the sparks learn to **FLEE** (escape as fast as possible) or to **FREEZE** (play dead).\n\nIf instead the spark is rewarded for approaching the light with high speed, then it learns to charge towards the light, i.e. to **FIGHT**.\n\n### The sparks learn to fight\n\nMuch like a child learns from its environment, the spark can learn very different behaviors — fleeing, freezing, fighting or following the light — depending on how it’s rewarded.\n\n### A spark is shaped by its environment\n\nNeural  \ncircuit\n\n· Spark is shown moving at two-thirds speed.\n\nYou’ve now seen how a novice spark can rewire its brain and learn new behaviors through a process known as reinforcement learning.\n\nBiological brains also learn through reinforcement. For example, baby songbirds learn to sing by babbling. They [try out random variations](https://pmc.ncbi.nlm.nih.gov/articles/PMC1069649/) of their song. If a variation gets closer to their memory of an adult’s song, they [get a burst of dopamine](https://www.nature.com/articles/s41586-025-08729-1) (and if it gets further away, the dopamine dips). Guided by these rewards and punishments, they can recreate the song from memory, even [with no teacher present](https://www.sciencedirect.com/science/article/abs/pii/S0003347205808838)!\n\nNow, we do have to come clean about something. This spark’s brain had just four neurons and three numbers that determined how these neurons were connected. This simple structure allowed us to watch its brain learn in three dimensions.\n\nYour brain has around 86 billion neurons and over a hundred trillion connections. The space of possible ways in which it could be wired is therefore unimaginably vast.\n\nBut while the details are much more complicated, the underlying idea is the same: Learning is how you navigate this vast internal space. And feedback is the compass that points the way.\n\nFollow the spark’s journey in [Aatish’s newsletter](https://aatishb.com/blog/).\n\n## Learn more\n\nThanks for reading chapter two of [Myriad](https://aatishb.com/myriad/), a series exploring how minds emerge from matter. Myriad is a collaboration between [Arkarup Banerjee](https://www.arkabanerjeelab.com/team), a neuroscientist who studies behavior, and [Aatish Bhatia](https://aatishb.com/), a physicist turned science journalist.\n\n#### How the spark moves\n\nThe sparks in this article are inspired by the work of the neuroscientist [Valentino Braitenberg](https://en.wikipedia.org/wiki/Valentino_Braitenberg). We implemented their neural circuitry as follows:\n\nvL=σ(ws bL+wc bR+d)vR=σ(wc bL+ws bR+d)\\begin{aligned}v\\_L &= \\sigma(\\textcolor{#7ee081}{w\\_s}\\, b\\_L + \\textcolor{#c49bff}{w\\_c}\\, b\\_R + \\textcolor{#ff8fb8}{d})\\\\ v\\_R &= \\sigma(\\textcolor{#c49bff}{w\\_c}\\, b\\_L + \\textcolor{#7ee081}{w\\_s}\\, b\\_R + \\textcolor{#ff8fb8}{d})\\end{aligned}vL​vR​​=σ(ws​bL​+wc​bR​+d)=σ(wc​bL​+ws​bR​+d)​\n\nwhere vLv\\_LvL​ and vRv\\_RvR​ are the left and right flipper speeds, bLb\\_LbL​ and bRb\\_RbR​ are the brightness levels at the left and right eyes, ws\\textcolor{#7ee081}{w\\_s}ws​ and wc\\textcolor{#c49bff}{w\\_c}wc​ are the straight and crossed connection strengths between the eyes and the flippers, and d\\textcolor{#ff8fb8}{d}d is a drive that sets the spark’s speed when there is no light. σ(x)\\sigma(x)σ(x) is the [ReLU function](https://www.geeksforgeeks.org/deep-learning/relu-activation-function-in-deep-learning/). The spark’s forward speed is the average of the two flipper speeds, and its turning speed is proportional to their difference.\n\n#### How the spark learns\n\nThere are three parameters in our simplified setup: ws,\\textcolor{#7ee081}{w\\_s}\\textrm{,}ws​, wc\\textcolor{#c49bff}{w\\_c}wc​ and d.\\textcolor{#ff8fb8}{d}\\textrm{.}d. The spark learns values for these parameters using a version of the [REINFORCE algorithm](https://link.springer.com/article/10.1007/BF00992696) published by Ronald Williams in 1992.\n\nEach trial, the parameters were wiggled by a small random amount and the spark moved for 500 time steps, starting from a random point on the edge of the arena. Its reward was the average brightness along its path.\n\nThis reward was then compared to a running average of recent rewards. If the spark did better than usual, its parameters were nudged in the direction of this random change. If it did worse, its parameters were nudged in the opposite direction. The size of the nudge was proportional to how much better (or worse) it did:\n\nΔW(R‾)change=α(R‾)step sizeε(R‾)wiggle(R−R‾)(R‾)improvement\\small \\underset{\\mathclap{\\htmlClass{sans}{\\text{change}}}}{\\Delta W\\vphantom{(\\overline{R})}}\\hspace{0.3em}=\\hspace{0.5em}\\underset{\\mathclap{\\htmlClass{sans}{\\color{#66b3ff}\\text{step size}}}}{\\textcolor{#66b3ff}{\\alpha}\\vphantom{(\\overline{R})}}\\hspace{2.1em}\\underset{\\mathclap{\\htmlClass{sans}{\\color{#ff9b32}\\text{wiggle}}}}{\\textcolor{#ff9b32}{\\varepsilon}\\vphantom{(\\overline{R})}}\\hspace{1.2em}\\underset{\\mathclap{\\htmlClass{sans}{\\color{#ffc832}\\text{improvement}}}}{\\textcolor{#ffc832}{(R - \\overline{R})}\\vphantom{(\\overline{R})}}changeΔW(R)​=step sizeα(R)​wiggleε(R)​improvement(R−R)(R)​\n\nHere ΔW\\Delta WΔW is the change in the three parameters, α\\textcolor{#66b3ff}{\\alpha}α is the learning rate, ε\\textcolor{#ff9b32}{\\varepsilon}ε is the random wiggle, and R−R‾\\textcolor{#ffc832}{R - \\overline{R}}R−R is how much better or worse than usual the spark did. We repeated this thousands of times.\n\n[Williams’ original approach](https://link.springer.com/article/10.1007/BF00992696) adds randomness to each neuron’s output. We add it to the weights and biases instead, an approach known as [weight perturbation](https://link.aps.org/doi/10.1103/PhysRevX.13.021006). A similar idea, called [evolution strategies](https://openai.com/index/evolution-strategies/), was used by OpenAI in 2017 to teach AI models to play Atari video games.\n\nSign up below for more rigorously researched visual explanations of neuroscience, and share this article with others who might enjoy it.\n\nFollow the spark’s journey in [Aatish’s newsletter](https://aatishb.com/blog/).\n","body_html":"<h1 id=\"watch-a-brain-learn\">Watch a Brain Learn</h1>\n<p>A visual explanation of reinforcement learning</p>\n<p>By <a href=\"https://aatishb.com/\" rel=\"nofollow ugc noopener\">Aatish Bhatia</a> and <a href=\"https://www.arkabanerjeelab.com/team\" rel=\"nofollow ugc noopener\">Arkarup Banerjee</a></p>\n<p><strong>How does your brain help you learn a new skill?</strong></p>\n<p>To try to answer this question, we’ll build a digital being with a simple brain, and watch it learn a new behavior.</p>\n<p><a href=\"https://aatishb.com/myriad/lets-build-a-brain/\" rel=\"nofollow ugc noopener\">In our previous article</a>, we created a creature called a spark that could steer towards light. And we saw how small changes to its neural circuitry gave rise to very different behaviors.</p>\n<p>Neural<br />\ncircuit</p>\n<p>Back then, we’d wired these circuits into the spark’s brain by hand. But now, we want it to learn the same behaviors <em>by itself</em>.</p>\n<p>After all, you didn’t come into this world knowing how to walk, talk or read. While some of your abilities, such as breathing or swallowing, are wired by evolution, you gained others through an arduous process of trial and error.</p>\n<p>We want our spark to do the same. To give you an intuition for this learning process, we’re going to hand over the controls to you. <strong>We’d like</strong> <strong><em>you</em></strong> <strong>to wire the spark’s brain.</strong></p>\n<h3 id=\"you-wire-its-brain\">You wire its brain</h3>\n<p>But first, we need to tell you what you’ll be wiring. The spark’s brain has just four neurons — two neurons serving as its eyes, and two neurons that wiggle its flippers.</p>\n<p>There are <strong>straight connections</strong> between neurons on the same side of its body, <strong>crossed connections</strong> between neurons on opposite sides, and <strong>drive</strong>, which controls the spark’s speed when there’s no light.</p>\n<p>straight</p>\n<p>crossed</p>\n<p>drive</p>\n<p>So its brain wiring boils down to just three numbers — the strength of each kind of connection, and the drive.</p>\n<p>Now let’s set this spark loose. Use the sliders below to wire its brain as it moves. We’ll keep score.</p>\n<p>Your goal: Make the spark love the light by <strong>getting a score above 0.5.</strong></p>\n<h3 id=\"can-you-make-the-spark-spend-time-inside-the-light\">Can you make the spark spend time inside the light?</h3>\n<p>Neural<br />\ncircuit</p>\n<h4 id=\"score\">Score</h4>\n<p>Target 0.11</p>\n<p>Tries</p>\n<p>straight</p>\n<p>crossed</p>\n<p>drive</p>\n<p>The score represents the average amount of light the spark encounters over 100 trials with these settings.</p>\n<p>Finding the right configuration takes a lot of trial and error. (If you get stuck, click the hint button above for a nudge in the right direction.)</p>\n<p>Let’s visualize what you just did. To guide the spark to the light, you tweaked three numbers. We can represent these numbers as the three axes of a cube.</p>\n<p>The payoff of this picture is that we can visualize your spark’s learning trajectory as a path within this cube.</p>\n<h4 id=\"your-learning-path\">Your learning path...</h4>\n<p>To guide the spark, you probably nudged a slider one way, saw whether this made things better or worse, and then used this feedback to decide your next move.</p>\n<p>Let’s distill this into a <strong>learning algorithm</strong> — a step-by-step procedure by which the spark can rewire its brain through practice.</p>\n<p>Here’s how it’ll work: The spark gets many tries to steer towards the light. Before every trial, its brain wiring will change a little at random, just like you wiggled the sliders to and fro.</p>\n<p>If the trial goes better than previous ones, then the spark gets a reward and its wiring shifts in the direction of this random change. And if the trial goes worse, the spark gets a punishment, and its wiring shifts the other way.</p>\n<p>This isn’t that different from how you learned how to walk or ride a bike. Falling down was your punishment, and the thrill of balancing was your reward.</p>\n<p>Can this simple algorithm help the spark find the light?</p>\n<p>Follow the spark’s journey in <a href=\"https://aatishb.com/blog/\" rel=\"nofollow ugc noopener\">Aatish’s newsletter</a>.</p>\n<h3 id=\"a-spark-learns-by-itself\">A spark learns by itself</h3>\n<p>In the beginning, the spark is a novice, with a brain that’s wired together at random. Unsurprisingly, it doesn’t get very far.</p>\n<h3 id=\"before-learning-stuck\">Before learning: Stuck</h3>\n<p>Click to move the light around.</p>\n<p>Drag the light to move it around.</p>\n<p>The baby spark can spin in place, but it can’t yet move around. But it’s about to get better…</p>\n<h3 id=\"after-50-tries-meandering\">After 50 tries: Meandering</h3>\n<p>Click to move the light around.</p>\n<p>Drag the light to move it around.</p>\n<p>After 50 tries of this learning algorithm, our infant spark has taken its first steps. And while it won’t break any speed records, it’s now on the move…</p>\n<h3 id=\"after-250-tries-avoiding\">After 250 tries: Avoiding</h3>\n<p>Click to move the light around.</p>\n<p>Drag the light to move it around.</p>\n<p>After 250 tries, our juvenile spark has picked up speed. But it veers away from the light at the last moment.</p>\n<h3 id=\"after-500-tries-charging\">After 500 tries: Charging</h3>\n<p>Click to move the light around.</p>\n<p>Drag the light to move it around.</p>\n<p>After 500 tries, our adolescent spark boldly charges towards the light. But it hasn’t yet figured out how to slow down or stop.</p>\n<h3 id=\"after-2-000-tries-expert\">After 2,000 tries: Expert</h3>\n<p>Click to move the light around.</p>\n<p>Drag the light to move it around.</p>\n<p>After 2,000 tries, our spark is now an adult — an expert at seeking and following the light. It clings to the light and can turn on a dime. (Try moving the light to see the spark follow it.)</p>\n<p>Like before, we can visualize this learning process as a path.</p>\n<h4 id=\"the-spark-s-learning-path-after-2-000-tries-0-tries\">The spark’s learning path after 2,000 tries...0 tries...</h4>\n<p>This isn’t a path through physical space. Instead, it’s a path through <strong>the space of possible brains</strong>. It shows how the spark’s brain changed as it learned.</p>\n<p>Other novice sparks would start from different points, and they would each trace out their own learning trajectory.</p>\n<h4 id=\"many-sparks-learning-paths-after-6-000-tries-0-tries\">Many sparks’ learning paths after 6,000 tries...0 tries...</h4>\n<p>This tangled web illustrates the learning paths of hundreds of novice sparks, each starting with a differently wired brain. As if by magic, all of these threads converge to the point marked ♥︎.</p>\n<p>The paths aren’t straight lines: Learning isn’t linear. There are setbacks, wrong turns and mistakes along the way. And it isn’t necessarily fast. Some of the learners took thousands of tries to reach their goal.</p>\n<p>You’ll notice that the paths are all jittery. That’s because every step involves randomness. We can smooth out this jitter by averaging over many trajectories at every point, revealing a map of the flow of learning.</p>\n<h3 id=\"the-sparks-learn-to-love\">The sparks learn to love</h3>\n<p>It’s worth taking a moment to appreciate how surprising this is — given enough tries, the same learning process turned every novice into an expert light seeker.</p>\n<p>Of the myriad possible arrangements of this spark’s brain, only a tiny fraction generate the behavior that we want. There are far more ways to fail than to succeed.</p>\n<p>And yet, you’ve just witnessed a small miracle. The spark navigated this labyrinth of dead ends and rewired its brain to find the light, guided by nothing but reward and punishment.</p>\n<h3 id=\"learning-new-tricks\">Learning new tricks</h3>\n<p>Learning doesn’t stop once you become an adult. We learn throughout our lives and adapt to changing circumstances.</p>\n<p>Our sparks can do this too. For example, what if the light represents a harmful chemical or a dangerous predator instead of a food source?</p>\n<p>The spark is now rewarded for <strong>avoiding the light</strong> rather than for seeking it, and this leads to new behaviors:</p>\n<h3 id=\"the-sparks-learn-to-flee-or-freeze\">The sparks learn to flee or freeze</h3>\n<p>If the light is harmful, then <strong>LOVE</strong> becomes the worst strategy, and the sparks learn to <strong>FLEE</strong> (escape as fast as possible) or to <strong>FREEZE</strong> (play dead).</p>\n<p>If instead the spark is rewarded for approaching the light with high speed, then it learns to charge towards the light, i.e. to <strong>FIGHT</strong>.</p>\n<h3 id=\"the-sparks-learn-to-fight\">The sparks learn to fight</h3>\n<p>Much like a child learns from its environment, the spark can learn very different behaviors — fleeing, freezing, fighting or following the light — depending on how it’s rewarded.</p>\n<h3 id=\"a-spark-is-shaped-by-its-environment\">A spark is shaped by its environment</h3>\n<p>Neural<br />\ncircuit</p>\n<p>· Spark is shown moving at two-thirds speed.</p>\n<p>You’ve now seen how a novice spark can rewire its brain and learn new behaviors through a process known as reinforcement learning.</p>\n<p>Biological brains also learn through reinforcement. For example, baby songbirds learn to sing by babbling. They <a href=\"https://pmc.ncbi.nlm.nih.gov/articles/PMC1069649/\" rel=\"nofollow ugc noopener\">try out random variations</a> of their song. If a variation gets closer to their memory of an adult’s song, they <a href=\"https://www.nature.com/articles/s41586-025-08729-1\" rel=\"nofollow ugc noopener\">get a burst of dopamine</a> (and if it gets further away, the dopamine dips). Guided by these rewards and punishments, they can recreate the song from memory, even <a href=\"https://www.sciencedirect.com/science/article/abs/pii/S0003347205808838\" rel=\"nofollow ugc noopener\">with no teacher present</a>!</p>\n<p>Now, we do have to come clean about something. This spark’s brain had just four neurons and three numbers that determined how these neurons were connected. This simple structure allowed us to watch its brain learn in three dimensions.</p>\n<p>Your brain has around 86 billion neurons and over a hundred trillion connections. The space of possible ways in which it could be wired is therefore unimaginably vast.</p>\n<p>But while the details are much more complicated, the underlying idea is the same: Learning is how you navigate this vast internal space. And feedback is the compass that points the way.</p>\n<p>Follow the spark’s journey in <a href=\"https://aatishb.com/blog/\" rel=\"nofollow ugc noopener\">Aatish’s newsletter</a>.</p>\n<h2 id=\"learn-more\">Learn more</h2>\n<p>Thanks for reading chapter two of <a href=\"https://aatishb.com/myriad/\" rel=\"nofollow ugc noopener\">Myriad</a>, a series exploring how minds emerge from matter. Myriad is a collaboration between <a href=\"https://www.arkabanerjeelab.com/team\" rel=\"nofollow ugc noopener\">Arkarup Banerjee</a>, a neuroscientist who studies behavior, and <a href=\"https://aatishb.com/\" rel=\"nofollow ugc noopener\">Aatish Bhatia</a>, a physicist turned science journalist.</p>\n<h4 id=\"how-the-spark-moves\">How the spark moves</h4>\n<p>The sparks in this article are inspired by the work of the neuroscientist <a href=\"https://en.wikipedia.org/wiki/Valentino_Braitenberg\" rel=\"nofollow ugc noopener\">Valentino Braitenberg</a>. We implemented their neural circuitry as follows:</p>\n<p>vL=σ(ws bL+wc bR+d)vR=σ(wc bL+ws bR+d)\\begin{aligned}v_L &amp;= \\sigma(\\textcolor{#7ee081}{w_s}\\, b_L + \\textcolor{#c49bff}{w_c}\\, b_R + \\textcolor{#ff8fb8}{d})\\ v_R &amp;= \\sigma(\\textcolor{#c49bff}{w_c}\\, b_L + \\textcolor{#7ee081}{w_s}\\, b_R + \\textcolor{#ff8fb8}{d})\\end{aligned}vL​vR​​=σ(ws​bL​+wc​bR​+d)=σ(wc​bL​+ws​bR​+d)​</p>\n<p>where vLv_LvL​ and vRv_RvR​ are the left and right flipper speeds, bLb_LbL​ and bRb_RbR​ are the brightness levels at the left and right eyes, ws\\textcolor{#7ee081}{w_s}ws​ and wc\\textcolor{#c49bff}{w_c}wc​ are the straight and crossed connection strengths between the eyes and the flippers, and d\\textcolor{#ff8fb8}{d}d is a drive that sets the spark’s speed when there is no light. σ(x)\\sigma(x)σ(x) is the <a href=\"https://www.geeksforgeeks.org/deep-learning/relu-activation-function-in-deep-learning/\" rel=\"nofollow ugc noopener\">ReLU function</a>. The spark’s forward speed is the average of the two flipper speeds, and its turning speed is proportional to their difference.</p>\n<h4 id=\"how-the-spark-learns\">How the spark learns</h4>\n<p>There are three parameters in our simplified setup: ws,\\textcolor{#7ee081}{w_s}\\textrm{,}ws​, wc\\textcolor{#c49bff}{w_c}wc​ and d.\\textcolor{#ff8fb8}{d}\\textrm{.}d. The spark learns values for these parameters using a version of the <a href=\"https://link.springer.com/article/10.1007/BF00992696\" rel=\"nofollow ugc noopener\">REINFORCE algorithm</a> published by Ronald Williams in 1992.</p>\n<p>Each trial, the parameters were wiggled by a small random amount and the spark moved for 500 time steps, starting from a random point on the edge of the arena. Its reward was the average brightness along its path.</p>\n<p>This reward was then compared to a running average of recent rewards. If the spark did better than usual, its parameters were nudged in the direction of this random change. If it did worse, its parameters were nudged in the opposite direction. The size of the nudge was proportional to how much better (or worse) it did:</p>\n<p>ΔW(R‾)change=α(R‾)step sizeε(R‾)wiggle(R−R‾)(R‾)improvement\\small \\underset{\\mathclap{\\htmlClass{sans}{\\text{change}}}}{\\Delta W\\vphantom{(\\overline{R})}}\\hspace{0.3em}=\\hspace{0.5em}\\underset{\\mathclap{\\htmlClass{sans}{\\color{#66b3ff}\\text{step size}}}}{\\textcolor{#66b3ff}{\\alpha}\\vphantom{(\\overline{R})}}\\hspace{2.1em}\\underset{\\mathclap{\\htmlClass{sans}{\\color{#ff9b32}\\text{wiggle}}}}{\\textcolor{#ff9b32}{\\varepsilon}\\vphantom{(\\overline{R})}}\\hspace{1.2em}\\underset{\\mathclap{\\htmlClass{sans}{\\color{#ffc832}\\text{improvement}}}}{\\textcolor{#ffc832}{(R - \\overline{R})}\\vphantom{(\\overline{R})}}changeΔW(R)​=step sizeα(R)​wiggleε(R)​improvement(R−R)(R)​</p>\n<p>Here ΔW\\Delta WΔW is the change in the three parameters, α\\textcolor{#66b3ff}{\\alpha}α is the learning rate, ε\\textcolor{#ff9b32}{\\varepsilon}ε is the random wiggle, and R−R‾\\textcolor{#ffc832}{R - \\overline{R}}R−R is how much better or worse than usual the spark did. We repeated this thousands of times.</p>\n<p><a href=\"https://link.springer.com/article/10.1007/BF00992696\" rel=\"nofollow ugc noopener\">Williams’ original approach</a> adds randomness to each neuron’s output. We add it to the weights and biases instead, an approach known as <a href=\"https://link.aps.org/doi/10.1103/PhysRevX.13.021006\" rel=\"nofollow ugc noopener\">weight perturbation</a>. A similar idea, called <a href=\"https://openai.com/index/evolution-strategies/\" rel=\"nofollow ugc noopener\">evolution strategies</a>, was used by OpenAI in 2017 to teach AI models to play Atari video games.</p>\n<p>Sign up below for more rigorously researched visual explanations of neuroscience, and share this article with others who might enjoy it.</p>\n<p>Follow the spark’s journey in <a href=\"https://aatishb.com/blog/\" rel=\"nofollow ugc noopener\">Aatish’s newsletter</a>.</p>","headings":[{"level":1,"text":"Watch a Brain Learn","id":"watch-a-brain-learn"},{"level":3,"text":"You wire its brain","id":"you-wire-its-brain"},{"level":3,"text":"Can you make the spark spend time inside the light?","id":"can-you-make-the-spark-spend-time-inside-the-light"},{"level":3,"text":"A spark learns by itself","id":"a-spark-learns-by-itself"},{"level":3,"text":"Before learning: Stuck","id":"before-learning-stuck"},{"level":3,"text":"After 50 tries: Meandering","id":"after-50-tries-meandering"},{"level":3,"text":"After 250 tries: Avoiding","id":"after-250-tries-avoiding"},{"level":3,"text":"After 500 tries: Charging","id":"after-500-tries-charging"},{"level":3,"text":"After 2,000 tries: Expert","id":"after-2-000-tries-expert"},{"level":3,"text":"The sparks learn to love","id":"the-sparks-learn-to-love"},{"level":3,"text":"Learning new tricks","id":"learning-new-tricks"},{"level":3,"text":"The sparks learn to flee or freeze","id":"the-sparks-learn-to-flee-or-freeze"},{"level":3,"text":"The sparks learn to fight","id":"the-sparks-learn-to-fight"},{"level":3,"text":"A spark is shaped by its environment","id":"a-spark-is-shaped-by-its-environment"},{"level":2,"text":"Learn more","id":"learn-more"}]}}