I listened to the recent Ezra Klein interview with Nvidia CEO Jensen Huang, which seems to be something of a Rorschach test for AI doomerism: if you are inclined to see doom, Jensen sounds dismissive, but if you (like me) do not fear for the future of humanity at the (metaphorical!) hands of a computer program, Jensen’s responses seem very reasonable. (With some exception: does this guy really not know his own zip code?!)
\nBut I also think there was a lost opportunity. While Jensen rightfully indicates that existing regulation (e.g., product liability laws) can act to hold the frontier labs accountable for harms caused by their own products (a point made more thoroughly by former FTC chair Lina Khan), he didn’t challenge Klein enough.
\nIn particular, Klein repeatedly appealed to the authority conferred by AI\nexpertise; how could Jensen disagree with pioneers like Geoffrey Hinton?\nWhile Jensen (rightfully) pointed out Hinton has been wrong in his predictions\n(e.g., his infamously wrong\n2016 prediction that radiology would cease to exist by 2021),\nhe missed an opportunity to explain why Hinton is wrong: it’s not (merely)\nthat Hinton is not seeing the future (or aspects of it) clearly, it’s that\nHinton is making claims that far exceed the range of his domain expertise.
\nTo take the radiologist prediction (conveniently made long enough ago that it\nis now unequivocally wrong), we can fairly say that Hinton was wrong because\nhis prediction did not reflect what a radiologist actually does: had\nHinton bothered to ask the question instead of making an alarmist admonition\n(Hinton explicitly said that no further radiologists should be trained in\n2016!), he would have learned that\npractitioners do much more than interpret diagnostic images!\nMaking image interpretation faster or better does not obviate the radiologist — to the contrary, it allows the radiologist to better use their expertise to\nserve their patients!
\nThis pattern of Fool’s Expertise is repeated over and over again among AI doomers: their experience in one kind of system (e.g., LLMs) lends them unwarranted authority in another — authority that goes largely unchecked by people like Klein who don’t delineate between different kinds of expertise.
\nFool’s Expertise is particularly perilous when discussing catastrophe, which nearly tautologically involves long chains of causation that transcend domains: fully understanding one link does not imply understanding the chain. (This is why when the NTSB investigates an accident, the NTSB Go Team consists of experts across many different domains: they know that a catastrophic accident may involve failures across several different domains, interacting and cascading into broader system failure.)
\nAnd to say it clearly: to the degree that AI-inflicted doom relies on\npathways into the real world, experts on those pathways must be consulted.\nIs AI a cybersecurity threat? Please talk to cybersecurity experts — who\nwill be quick to point out that the ballyhooed OpenAI escape depended on\na pedestrian failure of containment.\nAre we concerned that AI is somehow going to fashion a novel pathogen? Let’s\nplease consult experts on viral synthesis — and listen to why\nthey think that the fears are misplaced.\nOr maybe it’s nukes that we’re afraid that AI will get a hold of? Great news:\nseven decades of living with the possibility of global thermonuclear war gave\nus not just extensive safeguards but also huge wells of expertise to draw\nfrom!
\nThis is not to say that risks in the physical world are zero, but that the risks are much more diffuse and attenuated when one follows the proposed pathways. For a serious, level-headed look at these risks, look at RAND’s On the Extinction Risk from Artificial Intelligence by Michael Vermeer, Emily Lathrop, and Alvin Moon. They trace these pathways by consulting experts in the relevant domains — and intentionally not engaging with AI experts, saying (emphasis mine):
\nWe also spoke with 11 RAND experts over the course of our work: two experts on risk analysis and\ndecisionmaking under uncertainty, two experts on nuclear weapons, six experts on biotechnology, and one\nexpert on climate change. We intentionally did not engage AI experts because we chose to avoid, wherever\npossible, making predictions about how AI capabilities would evolve in the future. We focused instead on\nwhat capabilities AI would require to achieve certain outcomes in each of our scenarios.
\nGiven RAND’s history, the sober analysis of their own domain-specific experts deserves considerably more weight than the apocalyptic forecasts emanating from the frontier labs; may their analysis serve to inoculate us from the contagion of fear!
","headings":[{"level":1,"text":"Fool's Expertise","id":"fool-s-expertise"}]}}