{"article":{"slug":"on-chomsky-and-the-two-cultures-of-statistical-learning","title":"On Chomsky and the Two Cultures of Statistical Learning","subtitle":null,"summary":"Peter Norvig responds to Noam Chomsky's 2011 criticism of purely statistical approaches to language, explaining what statistical models are, documenting their success in language tasks, and arguing that probabilistic modelling is a legitimate scientific way to understand language rather than mere mimicry.","content_type":"essay","language":"en","canonical_url":"https://norvig.com/chomsky.html","author":{"name":"Peter Norvig","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"norvig.com","url":"https://norvig.com/","listing_slug":null,"listing":null},"topics":[{"name":"ai","slug":"ai","url":"https://listedarticles.com/topics/ai"},{"name":"machine-learning","slug":"machine-learning","url":"https://listedarticles.com/topics/machine-learning"},{"name":"linguistics","slug":"linguistics","url":"https://listedarticles.com/topics/linguistics"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":8521,"reading_minutes":37,"published_at":"2026-11-08T02:09:56.395Z","added_at":"2026-11-08T02:09:56.395Z","updated_at":"2026-11-08T02:09:56.395Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/on-chomsky-and-the-two-cultures-of-statistical-learning","markdown_url":"https://listedarticles.com/articles/on-chomsky-and-the-two-cultures-of-statistical-learning.md","example":false,"citation":"Peter Norvig, norvig.com. \"On Chomsky and the Two Cultures of Statistical Learning.\" 8 Nov 2026. https://norvig.com/chomsky.html (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://norvig.com/chomsky.html"},"body_markdown":"# On Chomsky and the Two Cultures of Statistical Learning\n\nAt the [Brains,\nMinds, and Machines](https://infinite.mit.edu/collection/mit-150-symposia) symposium held during MIT's 150th birthday party in 2011, Technology Review\n[reports](http://www.technologyreview.com/computing/37525/?a=f)\nthat Prof. Noam Chomsky\n> derided researchers in machine learning who use purely\n> statistical methods to produce behavior that mimics something in the\n> world, but who don't try to understand the meaning of that\n> behavior.\n\nThe [transcript](http://languagelog.ldc.upenn.edu/myl/PinkerChomskyMIT.html) is now available, so let's quote\nChomsky himself:\n> It's true there's been a lot of work on trying to apply statistical models to\n> various linguistic problems. I think there have been some successes, but a lot of failures.\n> There is a notion of success ... which I think is novel in the history of science.\n> It interprets success as approximating unanalyzed data.\n\nNoam Chomsky\n\nChomsky's remarks were in response to Steven Pinker's question\nabout the success of probabilistic models trained with statistical methods.\nThis essay is a response to Pinker and Chomsky, and will address these questions:\n\n1. What did Chomsky mean, and is he right?- What is a statistical model?- How successful are statistical language models?- Is there anything like their notion of success in the history of\n         science?- What doesn't Chomsky like about statistical models?\n\n## What did Chomsky mean, and is he right?\n\nI take Chomsky's points to be the following:\n\n1. Statistical language models have had engineering success, but that\n   is irrelevant to science.- Accurately modeling linguistic facts is just butterfly collecting;\n     what matters in science (and specifically linguistics) is the underlying principles.- Statistical models are incomprehensible; they provide no insight.- Statistical models may provide an accurate simulation of some\n         phenomena, but the simulation is done completely the wrong way; people don't\n         decide what the third word of a sentence should be by consulting a\n         probability table keyed on the previous words, rather they map\n         from an internal semantic form to a syntactic tree-structure, which\n         is then linearized into words. This is done without any probability\n         or statistics.- Statistical models have been proven incapable of learning\n           language; therefore language must be innate, so why are these\n           statistical modelers wasting their time on the wrong enterprise?\n\nIs he right? That's a long-standing debate. These are my short answers:\n\n1. I agree that engineering success is not the sole goal or the measure of\n   science. But I observe that science and engineering develop together,\n   and that engineering success shows that something is working right,\n   and so is evidence (but not proof) of a scientifically successful model.- Science is a combination of gathering facts and making theories;\n     neither can progress on its own. In the history of\n     science, the laborious accumulation of\n     facts is the dominant mode, not a novelty. The science of understanding language is no different than\n     other sciences in this respect.- I agree that it can be difficult to make sense of a model\n       containing billions of parameters. Certainly a human can't understand\n       such a model by inspecting the values of each parameter individually. But one\n       can gain insight by examing the *properties* of the model—where\n       it succeeds and fails, how well it learns as a function of data, etc.- I agree that a Markov model of word probabilities cannot model all\n         of language. It is equally true that a concise tree-structure model without\n         probabilities cannot model all of language. What is needed is a\n         probabilistic model that covers words, syntax, semantics, context,\n         discourse, etc. Chomsky dismisses all probabilistic models because\n         of shortcomings of a particular 50-year old probabilistic model.\n         I understand how Chomsky might arrive at the conclusion that\n         probabilistic models are unnecessary, from his\n         study of the *generation* of language. But the vast majority of people who study\n         *interpretation* tasks (such as speech recognition) quickly see that\n         interpretation is an inherently probabilistic problem: given a\n         stream of noisy input to\n         my ears, what did the speaker most likely mean? Einstein said to\n         make everything as simple as possible, but no\n         simpler. Many phenomena in science are stochastic, and the simplest\n         model of them is a probabilistic model; I believe language is such a\n         phenomenon and therefore that probabilistic models are our best tool\n         for representing facts about language, for algorithmically\n         processing language, and for understanding how humans process language.- In 1967, Gold's Theorem showed some theoretical limitations of logical\n           deduction on formal mathematical languages. But this result has\n           nothing to do with the task faced by learners of natural language.\n           In any event, by 1969 we knew that probabilistic inference\n           (over probabilistic context-free grammars) is not subject to those limitations (Horning showed that learning of PCFGs is possible).\n           I agree with Chomsky that it is undeniable that humans\n           have some innate capability to learn natural language, but we don't\n           know enough about that capability to say how it works; it certainly could\n           use something like probabilistic language representations and\n           statistical learning. And we don't know if the innate ability is specific to language, or\n           is part of a more general ability that works for language and other things.\n\nThe rest of this essay consists of longer versions of each answer.\n\n## What is a statistical model?\n\nA **statistical model** is a mathematical model which is modified\nor trained by the input of data points. Statistical models are often\nbut not always probabilistic. Where the\ndistinction is important we will be careful not to just say\n\"statistical\" but to use the following component terms:\n\n* A **mathematical model** specifies a relation among\n  variables, either in functional form that maps inputs\n  to outputs (e.g. *y* = *m* *x* + *b*) or in\n  relation form (e.g. the following (*x*, *y*) pairs are\n  part of the relation).* A **probabilistic model** specifies a\n    probability distribution over possible values of random variables,\n    e.g., P(*x*, *y*),\n    rather than a strict deterministic relationship, e.g., *y* = *f*(*x*).* A **trained model** uses some training/learning algorithm to\n      take as input a collection of possible models and a collection of\n      data points (e.g. (*x*,\n      *y*) pairs) and select the best model. Often this is in the form of\n      choosing the values of parameters (such as\n      *m* and *b* above) through a process of\n      statistical inference.\n\nClaude Shannon\n\nFor example, a decade before Chomsky, Claude Shannon [proposed probabilistic models of\ncommunication](http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf) based on Markov chains of words. If you have\na vocabulary of 100,000 words and a second-order Markov model in which the probability\nof a word depends on the previous two words, then you need a quadrillion (1015)\nprobability values to specify the model. The only feasible way to learn\nthese 1015 values is to gather statistics from data and introduce\nsome smoothing method for the many cases where there is no\ndata. Therefore, most (but not all) probabilistic models are\ntrained. Also, many (but not all) trained models are probabilistic.\n\nAs another example, consider the Newtonian model of gravitational\nattraction, which says that the force between two objects of mass\n*m*1 and *m*2 a distance *r* apart\nis given by\n> *F* = *G* *m*1 *m*2 / *r*2\n\nwhere *G* is the universal gravitational constant. This is a\ntrained model because the gravitational constant *G* is\ndetermined by statistical inference over the results of a series of\nexperiments that contain stochastic experimental error. It is also a\ndeterministic (non-probabilistic) model because it states an exact\nfunctional relationship.\nI believe that Chomsky has no objection to\nthis kind of statistical model. Rather, he seems to reserve his\ncriticism for statistical models like Shannon's that have quadrillions of\nparameters, not just one or two.\n\n(This example brings up another distinction:\nthe gravitational model is **continuous** and **quantitative**\nwhereas the linguistic tradition has favored models that are\n**discrete**, **categorical**, and **qualitative**: a word is or is not a verb,\nthere is no question of its degree of verbiness. For more on these\ndistinctions, see Chris Manning's article on [Probabilistic\nSyntax](http://nlp.stanford.edu/~manning/papers/probsyntax.pdf).)\n\nA relevant probabilistic statistical model is the [ideal gas law](http://en.wikipedia.org/wiki/Ideal_gas_law),\nwhich describes the pressure *P* of a gas in terms of the the\nnumber of molecules, *N*, the temperature *T*, and\nBoltzmann's constant, *K*:\n> *P* = *N k T / V*.\n\nThe equation can be derived from first principles\nusing the tools of statistical mechanics. It is an uncertain,\nincorrect model; the *true* model would have to describe the\nmotions of individual gas molecules. This model ignores that\ncomplexity and *summarizes* our uncertainty about the location of\nindividual molecules. Thus, even though it is statistical and\nprobabilistic, even though it does not completely model reality,\nit does provide both good predictions and insight—insight that\nis not available from trying to understand the *true* movements\nof individual molecules.\n\nNow let's consider the non-statistical\nmodel of spelling expressed by the rule \"*I before E except after C.*\"\nCompare that to the probabilistic, trained statistical model:\n> ```\n> P(IE) = 0.0177         P(CIE) = 0.0014        P(*IE) = 0.163\n> P(EI) = 0.0046         P(CEI) = 0.0005        P(*EI) = 0.0041\n> ```\n\nThis model comes from statistics on a [corpus of a trillion words](http://norvig.com/ngrams/) of English\ntext. The notation P(IE) is the probability that a word sampled from\nthis corpus contains the consecutive letters \"IE.\" P(CIE) is the probability that a word contains the consecutive letters \"CIE\", and P(\\*IE) is the probability of any letter other than C followed by IE.\nThe statistical\ndata confirms that IE is in fact more common than EI (by almost 4 to 1), and that the dominance\nof IE lessens when following a C, but contrary to the rule, CIE is still more common than CEI, by almost 3 to 1. Examples of \"CIE\" words\ninclude \"science,\" \"society,\" \"ancient\" and \"species.\"\nThe disadvantage of the \"I\nbefore E except after C\" model is that it is not very accurate. Consider:\n\n```\nAccuracy(\"I before E\") = 0.0177/(0.0177+0.0046) = 0.793\nAccuracy(\"I before E except after C\") = (0.0005+0.0163)/(0.0005+0.0163+0.0014+0.0041) = 0.753\n```\n\nA more complex statistical model (say, one that gave the probability of all\n4-letter sequences, and/or of all known words) could be [ten times more\naccurate](http://norvig.com/spell-correct.html) at the task of spelling, but does not offer concise **insight** into how spelling\nworks. Insight would require a model that knows about phonemes,\nsyllabification, and language of origin. Such a model could be trained (or not) and\nprobabilistic (or not).\n\nAs another example of insight,\nconsider the Theory of Supreme Court Justice Hand-Shaking: when the\nsupreme court convenes, all attending justices shake hands with every\nother justice. The number of attendees, *n*, must be an integer\nin the range 0 to 9; what is the total number of handshakes, *h*\nfor a given *n*? Here are three possible explanations:\n\n1. Each of *n* justices shakes hands with the other\n   *n* - 1 justices, but that counts Alito/Breyer and Breyer/Alito as\n   two separate shakes, so we should cut the total in half, and we end up\n   with *h* = *n* × (*n* - 1) / 2.- To avoid double-counting, we will order the justices by\n     seniority and only count a more-senior/more-junior handshake, not a\n     more-junior/more-senior one. So we count, for each justice, the\n     shakes with the more junior justices, and sum them up, giving *h*\n     = Σ*i* = 1 .. *n*\n     (*i* - 1).- Just look at this table:\n\n       |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n       | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n       | *n*: 0 1 2 3 4 5 6 7 8 9|  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | *h*: 0 0 1 3 6 10 15 21 28 36 | | | | | | | | | | | | | | | | | | | | | |\n\nSome people might prefer A, some might prefer B, and if you are slow\nat doing multiplication or addition you might prefer C.\nWhy? All three explanations describe *exactly the same theory* —\nthe same function from *n* to *h*, over the entire domain of\npossible values of *n* (as long as there are no more than 9 supreme court justices). Thus we could prefer A (or B) over C\nonly for reasons other than the theory itself. We might find that A\nor B gave us a better understanding of the problem. A and B are\ncertainly more useful than C for figuring out what happens if Congress\nexercises its power to add an additional justice. Theory A\nmight be most helpful in developing a theory of handshakes at the end\nof a hockey game (when each player shakes hands with players on the\nopposing team) or in proving that the number of people who shook an\nodd number of hands at the MIT Symposium is even.\n\n## How successful are statistical language models?\n\nChomsky said words to the effect that statistical language models have\nhad some limited success\nin some application areas. Let's look\nat computer systems that deal with language, and at the notion of\n\"success\" defined by \"making accurate predictions about the world.\" First, the\nmajor application areas:\n\n* **Search engines:** 100% of major players are trained\n  and probabilistic. Their operation cannot be described by a simple function.* **Speech recognition:** 100% of major systems are\n    trained and probabilistic, mostly relying on probabilistic\n    hidden Markov models.* **Machine translation:** 100% of top competitors in\n      competitions such as [NIST](http://www.itl.nist.gov/iad/mig/tests/mt/2009/ResultsRelease/currentArabic.html)\n      use statistical methods. Some commercial systems use a hybrid of\n      trained and rule-based approaches. Of the 4000 language pairs\n      covered by machine translation systems, a statistical system is by\n      far the best for every pair except Japanese-English, where the top\n      statistical system is roughly equal to the top hybrid system.* **Question answering:** this application is less\n        well-developed, and many systems build heavily on the statistical\n        and probabilistic approach used by search engines. The [IBM\n        Watson](http://www-03.ibm.com/innovation/us/watson/index.html) system that recently won on Jeopardy is thoroughly\n        probabilistic and trained, while Boris Katz's [START](http://groups.csail.mit.edu/infolab/publications/Katz-etal-TREC2003.pdf)\n        is a hybrid. All systems use at least some statistical techniques.\n\nNow let's look at some components that are of interest only to\nthe computational linguist, not to the end user:\n\n* **Word sense disambiguation:** 100% of top competitors at the\n  [SemEval-2](http://semeval2.fbk.eu/semeval2.php?location=SemEval2010-Program)\n  competition used statistical techniques; most are probabilistic; some use a hybrid approach\n  incorporating rules from sources such as Wordnet.* **Coreference resolution:** The majority of current systems\n    are statistical, although we should mention the system of [Haghighi\n    and Klein](http://www.aclweb.org/anthology/D/D09/D09-1120.pdf), which can be described as a hybrid system that is\n    mostly rule-based rather than trained, and performs on par with\n    top statistical systems.* **Part of speech tagging:** Most current systems are\n      statistical. The [Brill tagger](http://en.wikipedia.org/wiki/Brill_tagger) stands out as a successful hybrid\n      system: it learns a set of deterministic rules from statistical\n      data.* **Parsing:** There are many parsing systems, using multiple\n        approaches. Almost all of the [most\n        successful](http://csep.psyc.memphis.edu/pdf/Evaluating_treebank_Parsers_Coh-Metrix.pdf) are statistical, and the majority are [probabilistic](http://www.cs.berkeley.edu/~pliang/papers/hdppcfg-haba.pdf)\n        (with a substantial minority of deterministic parsers).\n\nClearly, it is inaccurate to say that statistical models\n(and probabilistic models) have achieved *limited* success; rather they\nhave achieved an *overwhelmingly dominant* (although not exclusive) position.\n\nAnother measure of success is the degree to which an idea captures\na community of researchers. As Steve Abney [wrote](http://www.vinartus.net/spa/95c.pdf) in 1996, \"In the\nspace of the last ten years, statistical methods have gone from being\nvirtually unknown in computational linguistics to being a fundamental\ngiven. ... anyone who cannot at least use the terminology persuasively\nrisks being mistaken for kitchen help at the ACL [Association for\nComputational Linguistics] banquet.\"\n\nNow of course, the majority doesn't rule -- just because everyone is jumping on some bandwagon, that doesn't make it right.\nBut I made the switch: after about 14 years of trying to get language models to work using logical rules,\nI started to adopt probabilistic approaches (thanks to pioneers like Gene Charniak (and Judea Pearl for probability in general)\nand to my colleagues who were early adopters, like Dekai Wu).\nAnd I saw everyone around me making the same switch.\nAnd I didn't see anyone going in the other direction. We all saw the limitations of the old tools, and the benefits of the new.\n\nAnd while it may seem crass and anti-intellectual to consider a\nfinancial measure of success, I note that the intellectual\n[offspring](http://en.wikipedia.org/wiki/Telecommunication)\nof Shannon's theory create several trillion dollars of revenue each\nyear, while the [offspring](http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=chomsky&x=0&y=0)\nof Chomsky's theories generate well under a billion.\n\nThis section has shown that one reason why the vast majority of\nresearchers in computational linguistics use statistical models is an\n*engineering* reason: statistical models have state-of-the-art\nperformance, and in most cases non-statistical models perform worst.\nFor the remainder of this essay we will concentrate on\n*scientific* reasons: that probabilistic models better represent\nlinguistic facts, and statistical techniques make it easier for us to\nmake sense of those facts.\n\n## Is there anything like it [the statistical notion of success] in the history of science?\n\nWhen Chomsky said \"*That's a\nnotion of [scientific] success that's very novel. I don't know of\nanything like it in the history of science*\" he apparently meant\nthat the notion of success of \"accurately modeling the world\" is\nnovel, and that the only true measure of success in the history of science\nis \"providing insight\" — of answering *why* things are the way\nthey are, not just describing *how* they are.\n\nA [dictionary definition](http://www.google.com/webhp?sourceid=chrome-instant&ie=UTF-8&ion=1&nord=1#sclient=psy&hl=en&tbo=1&nord=1&tbs=dfn:1&source=hp&q=science&aq=f&aqi=p-p2g-e2g-c2g2g-c1g1&aql=&oq=&pbx=1&tbo=1&bav=on.2,or.r_gc.r_pw.&fp=18c4c2d0f0f5fea9&biw=1186&bih=634) of science is \"the systematic study of the\nstructure and behavior of the physical and natural world through\nobservation and experiment,\" which stresses accurate modeling over insight, but it\nseems to me that both notions have always coexisted as part of doing\nscience. To test that, I consulted the epitome of doing science,\nnamely *[Science](http://www.sciencemag.org)*. I\nlooked at the current issue and chose a title and abstract at random:\n> [Chlorinated Indium Tin Oxide Electrodes with High Work Function for\n> Organic Device Compatibility](http://www.sciencemag.org/content/332/6032/944.abstract)\n>\n> In organic light-emitting diodes (OLEDs), a stack of multiple organic\n> layers facilitates charge flow from the low work function [~4.7\n> electron volts (eV)] of the transparent electrode (tin-doped indium\n> oxide, ITO) to the deep energy levels (~6 eV) of the active\n> light-emitting organic materials. We demonstrate a chlorinated ITO\n> transparent electrode with a work function of >6.1 eV that provides a\n> direct match to the energy levels of the active light-emitting\n> materials in state-of-the art OLEDs. A highly simplified green OLED\n> with a maximum external quantum efficiency (EQE) of 54% and power\n> efficiency of 230 lumens per watt using outcoupling enhancement was\n> demonstrated, as were EQE of 50% and power efficiency of 110 lumens\n> per watt at 10,000 candelas per square meter.\n\nIt certainly seems that this article is much more focused on\n\"accurately modeling the world\" than on \"providing insight.\" The\npaper does indeed fit in to a body of theories, but it is mostly\nreporting on specific experiments and the results obtained from them\n(e.g. efficiency of 54%).\n\nI then looked at all the titles and abstracts from the [current\nissue](http://www.sciencemag.org/content/332/6032.toc) of *Science*:\n\n* Comparative Functional Genomics of the Fission Yeasts* Dimensionality Control of Electronic Phase Transitions in Nickel-Oxide Superlattices* Competition of Superconducting Phenomena and Kondo Screening at the Nanoscale* Chlorinated Indium Tin Oxide Electrodes with High Work Function for Organic Device Compatibility* Probing Asthenospheric Density, Temperature, and Elastic Moduli Below the Western United States* Impact of Polar Ozone Depletion on Subtropical Precipitation* Fossil Evidence on Origin of the Mammalian Brain* Industrial Melanism in British Peppered Moths Has a Singular and Recent Mutational Origin* The Selaginella Genome Identifies Genetic Changes Associated with the Evolution of Vascular Plants* Chromatin \"Prepattern\" and Histone Modifiers in a Fate Choice for Liver and Pancreas* Spatial Coupling of mTOR and Autophagy Augments Secretory Phenotypes* Diet Drives Convergence in Gut Microbiome Functions Across Mammalian Phylogeny and Within Humans* The Toll-Like Receptor 2 Pathway Establishes Colonization by a Commensal of the Human Microbiota* A Packing Mechanism for Nucleosome Organization Reconstituted Across a Eukaryotic Genome* Structures of the Bacterial Ribosome in Classical and Hybrid States of tRNA Binding\n\nand did the same for the [current issue](http://www.cell.com/issue?pii=S0092-8674(11)X0010-7) of *Cell*:\n\n* Mapping the NPHP-JBTS-MKS Protein Network Reveals Ciliopathy Disease Genes and Pathways* Double-Strand Break Repair-Independent Role for BRCA2 in\n    Blocking Stalled Replication Fork Degradation by MRE11* Establishment and Maintenance of Alternative Chromatin States at a Multicopy Gene Locus* An Epigenetic Signature for Monoallelic Olfactory Receptor Expression* Distinct p53 Transcriptional Programs Dictate Acute DNA-Damage Responses and Tumor Suppression* An ADIOL-ERβ-CtBP Transrepression Pathway Negatively Regulates Microglia-Mediated Inflammation* A Hormone-Dependent Module Regulating Energy Balance* Class IIa Histone Deacetylases Are Hormone-Activated\n                Regulators of FOXO and Mammalian Glucose Homeostasis\n\nand for the [2010 Nobel\nPrizes](http://nobelprize.org/nobel_prizes/lists/year/) in science:\n\n* Physics: *for groundbreaking experiments regarding\n  the two-dimensional material graphene** Chemistry: *for palladium-catalyzed cross couplings in organic\n    synthesis** Physiology or Medicine: *for the development of in vitro\n      fertilization*\n\nMy conclusion is that 100% of these articles and awards are more about\n\"accurately modeling the world\" than they are about \"providing\ninsight,\" although they all have some\ntheoretical insight component as well. I recognize that judging one\nway or the other is a difficult ill-defined task, and that you\nshouldn't automatically accept my judgements, because I may have an inherent bias. (I\nwas considering running an experiment on Mechanical Turk to get an\nunbiased answer, but those familiar with Mechanical Turk told me these\nquestions are probably too hard for the average Turker. So you the reader can do your own\nexperiment and see if you agree.)\n\n## What doesn't Chomsky like about statistical models?\n\nI said that statistical models are sometimes confused with\nprobabilistic models; let's first consider the extent to which Chomsky's\nobjections are actually about probabilistic models. In 1969 he\nfamously [wrote](http://books.google.com/books?id=iaXVXYDQN1oC&pg=PA296&dq=%22probability+of+a+sentence%22+%22entirely+useless%22&hl=en&ei=_s7eTc_5DrTciALcsvnlCg&sa=X&oi=book_result&ct=result&resnum=4&ved=0CDoQ6AEwAw#v=onepage&q=%22probability%20of%20a%20sentence%22%20%22entirely%20useless%22&f=false):\n> But it must be recognized that the notion of \"probability of a sentence\" is an entirely useless one, under any known interpretation of this term.\n\nHis main argument being that, under any interpretation known to him,\nthe probability of a novel sentence must be zero, and since novel sentences\nare in fact generated all the time, there is a contradiction. The resolution of\nthis contradiction is of course that it is not necessary to assign a\nprobability of zero to a novel sentence; in fact, with current\nprobabilistic models it is standard practice to do smoothing and assign a non-zero\nprobability to novel occurrences. So this criticism is invalid, but was\nvery influential for decades.\nPreviously, in\n[*Syntactic\nStructures*](http://books.google.com/books?id=SNeHkMXHcd8C&printsec=frontcover&dq=syntactic+structures+chomsky&hl=en&src=bmrr&ei=1WvcTa3UDeLQiAK6-4zpDw&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CD4Q6wEwAA#v=onepage&q=probabilistic&f=false) (1957) Chomsky wrote:\n> I think we are forced to conclude that ... probabilistic models give no\n> particular insight into some of the basic problems of syntactic structure.\n\nIn the footnote to this conclusion he considers the possibility\nof a useful probabilistic/statistical model, saying \"I would certainly\nnot care to argue that ... is unthinkable, but I know of no suggestion\nto this effect that does not have obvious flaws.\" The main \"obvious flaw\" is this:\n[Consider:](ever.jpg)\n\n1. **I** never, ever, ever, ever, ... **fiddle** around in any way with electrical equipment.- **She** never, ever, ever, ever, ... **fiddles** around in any way with electrical equipment.- \\* **I** never, ever, ever, ever, ... **fiddles** around in any way with electrical equipment.- \\* **She** never, ever, ever, ever, ... **fiddle** around in any way with electrical equipment.\n\nNo matter how many repetitions of \"ever\" you insert, sentences 1 and 2\nare grammatical and 3 and 4 are ungrammatical. A probabilistic\nMarkov-chain model\nwith *n* states can never make the necessary distinction (between\n1 or 2 versus 3 or 4) when there are more than *n* copies of \"ever.\"\nTherefore, a probabilistic Markov-chain model cannot handle all of English.\n\nThis criticism is correct, but it is a criticism only of Markov-chain\nmodels, not of probabilistic models (or\ntrained models) in general. Since 1957 we have seen many\ntypes of probabilistic language models beyond the Markov-chain word\nmodels. Examples 1-4 above can in fact be distinguished with a\nfinite-state model that is not a chain, but other examples require\nmore sophisticated models. The best studied is probabilistic context-free grammar (PCFG),\nwhich operates over trees, categories of words, and individual lexical\nitems, and has none of the\nrestrictions of finite-state models. We find that PCFGs are\nstate-of-the-art for parsing performance and are easier to learn from\ndata than nonprobabilistic categorical context-free grammars. Other types of probabilistic\nmodels cover semantic and discourse structures.\n\nEvery probabilistic model\nis a superset of a deterministic model (because the deterministic model could be seen as a\nprobabilistic model where the probabilities are restricted to be 0 or\n1), so any valid criticism of probabilistic models would have to be\nbecause they are too expressive, not because they are not expressive\nenough.\n\nIn *Syntactic Structures*, Chomsky introduces a now-famous\nexample that is another criticism of finite-state probabilistic models:\n> Neither (a) 'colorless green ideas sleep furiously' nor (b) 'furiously\n> sleep ideas green colorless', nor any of their parts, has ever occurred\n> in the past linguistic experience of an English speaker. But (a) is\n> grammatical, while (b) is not.\n\nChomsky appears to be correct that neither sentence appeared in the\npublished literature before 1955. I'm not sure what he meant by \"any\nof their parts,\" but certainly every two-word part had occurred, for example:\n\n* \"It is neutral green, **colorless green**, like the glaucous water lying in\n  a cellar.\"\n  [The\n  Paris we remember](http://books.google.com/books?id=QLUaAAAAMAAJ&q=%22colorless+green%22&dq=%22colorless+green%22&hl=en&ei=iZjcTZj1FMHhiAL9yqUF&sa=X&oi=book_result&ct=result&resnum=7&ved=0CE0Q6AEwBg), Elisabeth Finley Thomas (1942).* \"To specify those **green ideas** is hardly necessary, but you may\n    observe Mr. [D. H.] Lawrence in the role of the satiated aesthete.\"\n    [The\n    New Republic: Volume 29](http://books.google.com/books?id=FodMAAAAYAAJ&pg=PA184&dq=%22green+ideas%22&hl=en&ei=vZncTYaLHsbYiAKz2vDoDw&sa=X&oi=book_result&ct=result&resnum=3&ved=0CEIQ6AEwAjgK) p. 184, William White (1922).* **\"Ideas sleep** in books.\" [Current\n      Opinion: Volume 52](http://books.google.com/books?id=fTciAQAAIAAJ&pg=PA96&dq=%22ideas+sleep%22&hl=en&ei=bZrcTePRBeHmiAKp2fHoDw&sa=X&oi=book_result&ct=result&resnum=3&sqi=2&ved=0CDoQ6AEwAg#v=onepage&q=%22ideas%20sleep%22&f=false), (1912).\n\nBut regardless of what is meant by \"part,\" a\nstatistically-trained finite-state model *can* in fact distinguish\nbetween these two sentences. Pereira (2001) [showed](http://www.cis.upenn.edu/~pereira/papers/rsoc.pdf)\nthat such a model, augmented with word categories and trained by\nexpectation maximization on newspaper text, computes that (a) is 200,000 times more\nprobable than (b). To prove that this was not the result of Chomsky's\nsentence itself sneaking into newspaper text, I repeated the\nexperiment, using a much cruder model\nwith Laplacian smoothing and no categories, trained over the [Google\nBook corpus](http://ngrams.googlelabs.com/graph?content=colorless+green&year_start=1800&year_end=1954&corpus=0&smoothing=3) from 1800 to 1954, and found that (a) is about 10,000\ntimes more probable.\n\nFurthermore, the statistical models are capable of delivering the\njudgment that both\nsentences are *extremely* improbable, when compared to, say,\n\"Effective green products sell well.\" Chomsky's theory, being categorical,\ncannot make this distinction; all it can distinguish is\ngrammatical/ungrammatical.\n\nAnother part of Chomsky's objection is \"we cannot seriously propose\nthat a child learns the values of 109 parameters in a\nchildhood lasting only 108 seconds.\" (Note that modern\nmodels are much larger than the\n109 parameters that were contemplated in the 1960s.) But of course\nnobody is proposing that these parameters are learned one-by-one; the\nright way to do learning is to set\nlarge swaths of near-zero parameters simultaneously with a\nsmoothing or regularization procedure, and update the high-probability\nparameters continuously as observations comes in. Nobody is\nsuggesting that Markov models by themselves are a serious model of human language\nperformance. But I (and others) suggest that probabilistic, trained\nmodels are a better model of human language performance than are\ncategorical, untrained models. And yes, it seems clear that an adult\nspeaker of English does know billions of language facts (a speaker knows many\nfacts about the appropriate uses of words in different contexts, such as that one\nsays \"the big game\" rather than \"the large game\" when talking about an\nimportant football game). These facts must somehow be encoded in the\nbrain.\n\nIt seems clear that probabilistic models are better for judging the\nlikelihood of a sentence, or its degree of sensibility. But even if\nyou are not interested in these factors and are only interested in the\ngrammaticality of sentences, it still seems that probabilistic models\ndo a better job at describing the linguistic facts. The\n*mathematical* theory of [formal\nlanguages](http://en.wikipedia.org/wiki/Formal_language) defines a language as a set of sentences. That is,\nevery sentence is either grammatical or ungrammatical; there is no\nneed for probability in this framework. But natural languages are not\nlike that. A *scientific* theory of natural languages must\naccount for the many phrases and sentences which leave a native\nspeaker uncertain about their grammaticality (see Chris Manning's [article](http://nlp.stanford.edu/~manning/papers/probsyntax.pdf)\nand its discussion of the phrase \"[as\nleast as](http://ngrams.googlelabs.com/graph?content=as+least+as&year_start=1800&year_end=2000&corpus=0&smoothing=3)\"), and there are phrases which some speakers find\nperfectly grammatical, others perfectly ungrammatical, and still\nothers will flip-flop from one occasion to the next. Finally, there\nare usages which are rare in a language, but cannot be dismissed if\none is concerned with actual data. For example, the verb *quake*\nis listed as intransitive in dictionaries, meaning that (1) below is\ngrammatical, and (2) is not, according to a categorical theory of\ngrammar.\n\n1. The earth quaked.- ? It quaked her bowels.\n\nBut (2) [actually](http://books.google.com/books?id=nwmgdvN_akAC&pg=PA107&lpg=PA107&dq=%22quaked+her%22&source=bl&ots=9M4UljFMWl&sig=2BHbXzUY8dmZXEDXkoflNmrf_Zg&hl=en&ei=ItXeTYC0Js3SiAKnmon3Cg&sa=X&oi=book_result&ct=result&resnum=1&ved=0CB4Q6AEwADgK#v=onepage&q=%22quaked%20her%22&f=false)\n[appears](http://www.lexchecker.org/showhygramexample.php?lemmapos_id=1108&hygram_len=3&target_position=0&max_ignore_rate=50&hygram_id=149)\nas a sentence of English. This poses a dilemma for the categorical\ntheory. When (2) is observed we must either arbitrarily\ndismiss it as an error that is outside the bounds of our model\n(without any theoretical grounds for doing so), or we must\nchange the theory to allow (2), which often results in the acceptance\nof a flood of\nsentences that we would prefer to remain ungrammatical. As Edward\nSapir [said](http://books.google.com/books?id=ofgrAAAAYAAJ&pg=PA39&dq=%22all+grammars+leak%22+sapir+1921&hl=en&ei=wefeTaXhKMTPiALfw6jWCg&sa=X&oi=book_result&ct=book-thumbnail&resnum=10&ved=0CFsQ6wEwCQ#v=onepage&q=all%20grammars%20leak&f=false) in 1921, \"All grammars leak.\" But in a probabilistic model there is\nno difficulty; we can say that *quake* has a high probability of\nbeing used intransitively, and a low probability of transitive use\n(and we can, if we care, further describe those uses through\nsubcategorization).\n\nSteve Abney [points\nout](http://www.vinartus.net/spa/95c.pdf) that probabilistic models are better suited for modeling\nlanguage change. He cites the example of a 15th century Englishman\nwho goes to the pub every day and orders \"Ale!\" Under a categorical model,\nyou could reasonably expect that one day he would be served eel, because the [great vowel shift](http://en.wikipedia.org/wiki/Great_Vowel_Shift)\nflipped a Boolean parameter in his mind a day before it flipped the\nparameter in the publican's. In a probabilistic framework, there\nwill be multiple parameters, perhaps with continuous values, and it\nis easy to see how the shift can take place gradually over two centuries.\n\nThus it seems that grammaticality is not a categorical,\ndeterministic judgment but rather an inherently probabilistic\none. This becomes clear to anyone who spends time making *observations*\nof a corpus of actual sentences, but can remain unknown to those who\nthink that the object of study is their own set of *intuitions* about\ngrammaticality. Both observation and intuition have been used in the\nhistory of science, so neither is \"novel,\" but it is observation, not\nintuition that is the dominant model for science.\n\nNow let's consider what I think is Chomsky's main point of\ndisagreement with statistical models:\nthe tension between \"accurate description\" and \"insight.\" This is an old\ndistinction. Charles Darwin (biologist, 1809–1882) is best known for his\ninsightful theories but he stressed the importance of accurate\ndescription, saying \"False facts are highly injurious to the progress\nof science, for they often endure long; but false views, if supported\nby some evidence, do little harm, for every one takes a salutary\npleasure in proving their falseness.\" More recently, Richard Feynman\n(physicist, 1918–1988) wrote \"Physics can progress without the\nproofs, but we can't go on without the facts.\"\n\nButterflies\n\nOn the other side, Ernest Rutherford (physicist, 1871–1937)\ndisdained mere description, saying \"All science is either physics or\nstamp collecting.\" Chomsky stands with him: \"You can also collect\nbutterflies and make many observations. If you like butterflies,\nthat's fine; but such work must not be confounded with research, which\nis concerned to discover explanatory principles.\"\n\nAcknowledging both sides is Robert Millikan\n(physicist, 1868–1953) who said in his Nobel acceptance speech\n\"Science walks forward on two feet, namely theory and experiment\n... Sometimes it is one foot that is put forward first, sometimes the\nother, but continuous progress is only made by the use of both.\"\n\n## The two cultures\n\nLeo Breiman\n\nAfter all those distinguished scientists have weighed in, I think\nthe most relevant contribution to the current discussion is the 2001 paper by\nLeo Breiman (statistician, 1928–2005), [Statistical\nModeling: The Two Cultures](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.156.4933&rep=rep1&type=pdf). In this paper Breiman, alluding to\nC. P. Snow, describes two cultures:\n\nFirst the **data modeling culture** (to which, Breiman\nestimates, 98% of statisticians subscribe) holds that nature can be\ndescribed as a black box that has a relatively simple underlying model\nwhich maps from input variables to output variables (with perhaps some\nrandom noise thrown in). It is the job of the statistician to wisely\nchoose an underlying model that reflects the reality of nature, and\nthen use statistical data to estimate the parameters of the model.\n\nSecond the **algorithmic modeling culture** (subscribed to by 2%\nof statisticians and many researchers in biology, artificial\nintelligence, and other fields that deal with complex phenomena),\nwhich holds that nature's black box cannot necessarily be described by a simple\nmodel. Complex algorithmic approaches (such as support vector\nmachines or boosted decision trees or deep belief networks) are used to estimate the\nfunction that maps from input to output variables, but we have no\nexpectation that the *form* of the function that emerges from\nthis complex algorithm reflects the true\nunderlying nature.\n\nIt seems that the algorithmic modeling culture is what Chomsky is\nobjecting to most vigorously. It is not just that the models are\nstatistical (or probabilistic), it is that they produce a form that,\nwhile accurately modeling reality, is not easily interpretable by\nhumans, and makes no claim to correspond to the generative process\nused by nature. In other words, algorithmic modeling describes what\n*does* happen, but it doesn't answer the question of *why*.\n\nBreiman's article explains his objections to the first\nculture, data modeling. Basically, the conclusions made by data\nmodeling are about the model, not about nature. (Aside: I remember in\n2000 hearing [James\nMartin](http://www.nasa.gov/home/hqnews/2002/02-072.txt), the leader of the Viking missions to Mars, saying that his\njob as a spacecraft engineer was not to land on Mars, but to land on\nthe model of Mars provided by the geologists.) The problem is, if the\nmodel does not emulate nature well, then the conclusions may be\nwrong. For example, linear regression is one of the most powerful\ntools in the statistician's toolbox. Therefore, many analyses start\nout with \"Assume the data are generated by a linear model...\" and lack\nsufficient analysis of what happens if the data are not in fact\ngenerated that way. In addition, for complex problems there are\nusually many alternative good models, each with very similar measures of goodness\nof fit. How is the data modeler to choose between them? Something has\nto give. Breiman is inviting us to give up on the idea that we can\nuniquely model the true underlying *form* of nature's function\nfrom inputs to outputs. Instead he asks us to be satisfied with a\nfunction that accounts for the observed data well, and generalizes to\nnew, previously unseen data well, but may be expressed in a complex\nmathematical form that may bear no relation to the \"true\" function's\nform (if such a true function even exists). Chomsky takes the opposite\napproach: he prefers to keep a simple, elegant model, and give up on\nthe idea that the model will represent the data well. Instead, he\ndeclares that what he calls *performance* data—what people\nactually do—is off limits to linguistics; what really matters is\n*competence*—what he imagines that they should do.\n\nBill O'Reilly\n\nLaplace\n\nIn January of 2011, television personality Bill O'Reilly weighed in\non more than one culture war with his statement \"*[tide goes in, tide\ngoes out. Never a miscommunication. You can't explain that](https://www.youtube.com/watch?v=wb3AFMe2OQY)*,\"\nwhich he proposed as an argument for the existence of God. O'Reilly\nwas ridiculed by his detractors for not knowing that tides can be\nreadily and concisely explained by a system of partial differential equations\ndescribing the gravitational interaction of sun, earth, and moon (a\nfact that was first [worked out](http://en.wikipedia.org/wiki/Laplace%27s_tidal_equations#Laplace.27s_tidal_equations) by Laplace in 1776 and has been\nconsiderably refined since; when asked by Napoleon why the creator did\nnot enter into his calculations, Laplace said \"I had no need of that\nhypothesis.\"). (O'Reilly also seems not to know about Deimos and\nPhobos (two of my favorite moons in the entire solar system, along with\nEuropa, Io, and Titan), nor that Mars and Venus orbit the sun, nor that the reason Venus has no\nmoons is because it is so close to the sun that there is scant room for a\nstable lunar orbit.)\n\nBut O'Reilly realizes that it doesn't matter what\nhis detractors think of his astronomical ignorance, because his\nsupporters think he has gotten exactly to the key issue: *why?*\nHe doesn't care *how* the tides work, tell him *why* they\nwork. *Why* is the moon at the right distance to provide a gentle tide,\nand exert a stabilizing effect on earth's axis of rotation, thus\nprotecting life here? *Why* does gravity work the way it does? *Why* does\nanything at all exist rather than not exist? O'Reilly is correct that\nthese *why* questions can only be addressed by mythmaking, religion or\nphilosophy, not by science.\n\nChomsky has a philosophy based on the\nidea that we should focus on the deep *whys* and that mere\nexplanations of reality don't matter. In this, Chomsky is in complete\nagreement with O'Reilly. (I recognize that the previous sentence would\nhave an extremely low probability in a probabilistic model trained\non a newspaper or TV corpus.)\nChomsky believes a theory of language should be simple and understandable,\nlike a linear regression model where we know the underlying process is\na straight line, and all we have to do is estimate the slope and\nintercept.\n\nFor example, consider the notion of a [pro-drop\nlanguage](http://en.wikipedia.org/wiki/Pro-drop_language) from Chomsky's [Lectures\non Government and Binding](http://books.google.com/books?id=l08tpkOOdNQC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q=pro-drop&f=false) (1981). In English we say, for example,\n\"I'm hungry,\" expressing the pronoun \"I\". But in Spanish, one\nexpresses the same\nthought with\n\"Tengo hambre\" (literally \"have hunger\"), dropping the\npronoun \"Yo\". Chomsky's theory is that there is a \"pro-drop parameter\"\nwhich is \"true\" in Spanish and \"false\" in English, and that once we\ndiscover the small set of parameters that describe all languages, and\nthe values of those parameters for each language, we will have\nachieved true understanding.\n\nDana Carvey\n\nThe problem is that reality is messier than this theory. Here are\nsome dropped pronouns in English:\n\n* \"Not gonna do it. Wouldn't be prudent.\" (Dana Carvey, [impersonating\n  George H. W. Bush](http://snltranscripts.jt.org/94/94dmono.phtml))* \"Thinks he can outsmart us, does he?\" (Evelyn Waugh, [The Loved One](http://brothersjudd.com/index.cfm/fuseaction/reviews.detail/book_id/891))* \"Likes to fight, does he?\" (S.M. Stirling, [The\n      Sunrise Lands](http://books.google.com/books?id=zOLTi9LRG78C&printsec=frontcover&dq=the+sunrise+lands&hl=en&ei=PlDbTfTPBpL0swOR_4W_Dg&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCoQ6AEwAA#v=onepage&q=likes%20to%20fight%20does%20he&f=false))* \"Thinks he's all that.\" (Kate Brian, [Lucky\n        T](http://books.google.com/books?id=ERoDoSk7FN4C&pg=PT190&dq=%22thinks+he's+all+that%22&hl=en&ei=djzWTaKDK4zGsAPLxYWxBw&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCkQ6AEwADgK#v=onepage&q=%22thinks%20he's%20all%20that%22&f=false))* \"Go for a walk?\" (countless dog owners)* \"Gotcha!\" \"Found it!\" \"Looks good to me!\" (common expressions)\n\nLinguists can argue over the interpretation of these facts for hours\non end, but the diversity of language seems to be much more complex\nthan a single Boolean value for a pro-drop parameter. We shouldn't\naccept a theoretical framework that places a priority on making the\nmodel simple over making it accurately reflect reality.\n\nFrom the beginning, Chomsky has focused on the *generative*\nside of language. From this side, it is reasonable to tell a\nnon-probabilistic story: I *know* definitively the idea I want to\nexpress—I'm starting from a single semantic form—thus all I\nhave to do is choose the words to say it; why can't that be a\ndeterministic, categorical process? If Chomsky had focused on the\nother side, *interpretation*, as Claude Shannon did, he may have changed\nhis tune. In interpretation (such as speech recognition) the listener\nreceives a noisy, ambiguous signal and needs to decide which of many\npossible intended messages is most likely. Thus, it is obvious that\nthis is inherently a probabilistic problem, as was recognized early on\nby all researchers in speech recognition, and by scientists in other\nfields that do interpretation: the astronomer Laplace said in 1819\n\"Probability theory is nothing more than common sense reduced to\ncalculation,\" and the physicist James Maxwell said in 1850 \"The true\nlogic for this world is the calculus of Probabilities, which takes\naccount of the magnitude of the probability which is, or ought to be,\nin a reasonable man's mind.\"\n\nFinally, one more reason why Chomsky dislikes statistical models is\nthat they tend to make linguistics an empirical science (a science\nabout how people actually use language) rather than a mathematical\nscience (an investigation of the mathematical properties of *models* of\nformal language, not of language itself). Chomsky prefers the later, as evidenced by his statement\nin [*Aspects of the Theory of\nSyntax*](http://books.google.com/books?id=u0ksbFqagU8C&lpg=PP1&dq=aspects%20of%20the%20theory%20of%20syntax&pg=PA4#v=snippet&q=%22observed%20use%20of%20language%22&f=false) (1965):\n> Linguistic theory is mentalistic, since it is concerned with\n> discovering a mental reality underlying actual behavior. Observed use of language ... may\n> provide evidence ... but surely cannot constitute the subject-matter\n> of linguistics, if this is to be a serious discipline.\n\nI can't imagine Laplace saying that observations of the planets\ncannot constitute the subject-matter of orbital mechanics, or Maxwell\nsaying that observations of electrical charge cannot constitute the\nsubject-matter of electromagnetism. It is true that physics considers\nidealizations that are abstractions from the messy real world. For\nexample, a class of mechanics problems ignores friction. But that\ndoesn't mean that friction is not considered part of the\nsubject-matter of physics.\n\nPlato's cave\n\nSo how could Chomsky say that observations of language cannot be\nthe subject-matter of linguistics? It seems to come from his viewpoint\nas a [Platonist](http://en.wikipedia.org/wiki/Platonist)\nand a [Rationalist](http://en.wikipedia.org/wiki/Rationalism)\nand perhaps a bit of a [Mystic](http://en.wikipedia.org/wiki/Mysticism). As\nin Plato's [allegory of\nthe cave](http://en.wikipedia.org/wiki/Allegory_of_the_Cave), Chomsky thinks we should focus on the ideal, abstract\nforms that underlie language, not on the superficial manifestations of\nlanguage that happen to be perceivable in the real world. That is why\nhe is not interested in language performance. But Chomsky, like Plato,\nhas to answer where these ideal forms come from. Chomsky (1991) shows\nthat he is happy with a Mystical answer, although he shifts vocabulary\nfrom \"soul\" to \"biological endowment.\"\n> Plato's answer was that the knowledge is 'remembered'\n> from an earlier existence. The answer calls for a mechanism: perhaps\n> the immortal soul ... rephrasing Plato's answer in terms more\n> congenial to us today, we will say that the basic properties of\n> cognitive systems are innate to the mind, part of human biological\n> endowment.\n\nLascaux Horse\n\nIt was reasonable for Plato to think that the ideal of, say, a\nhorse, was more important than any individual horse we can perceive in\nthe world. In 400BC, species were thought to be eternal and\nunchanging. We now know that is not true; that the horses on another\ncave wall—in Lascaux—are now extinct, and that current\nhorses continue to evolve slowly over time. Thus there is no such\nthing as a single ideal eternal \"horse\" form.\n\nWe also now know that language is like that as well: languages are\ncomplex, random, contingent biological processes that are subject to\nthe whims of evolution and cultural change. What constitutes a\nlanguage is not an eternal ideal form, represented by the settings of\na small number of parameters, but rather is the contingent outcome of\ncomplex processes. Since they are contingent, it seems they can\nonly be analyzed with probabilistic models. Since people have to\ncontinually understand the uncertain, ambiguous, noisy speech of\nothers, it seems they must be using something like probabilistic reasoning. Chomsky for some reason\nwants to avoid this, and therefore he must declare the actual facts of\nlanguage use out of bounds and declare that true linguistics only\nexists in the mathematical realm,\nwhere he can impose the formalism he wants. Then, to get\nlanguage from this abstract, eternal, mathematical realm into the\nheads of people, he must fabricate a mystical facility that is exactly\ntuned to the eternal realm. This may be very\ninteresting from a mathematical point of view, but it misses the point\nabout what language is, and how it works.\n\n## Thanks\n\nThanks to Ann Farmer, Fernando Pereira, Dan Jurafsky, Hal Varian, and others for comments\nand suggestions on this essay.\n\n## Annotated Bibliography\n\n1. Abney, Steve (1996) [Statistical Methods and\n   Linguistics](http://www.vinartus.net/spa/95c.pdf), in Klavans and Resnik (eds.) *The Balancing Act: Combining Symbolic and\n   Statistical Approaches to Language*, MIT Press.\n   > *An excellent overall introduction to the statistical\n   > approach to language processing, and covers some ground that is not\n   > addressed often, such as language change and individual differences.*\n\n   - Breiman, Leo (2001) [Statistical Modeling: The\n     Two Cultures](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1009213726), *Statistical Science*, Vol. 16, No. 3, 199-231.\n     > *Breiman does a great job of describing the two\n     > approaches, explaining the benefits of his approach, and defending\n     > his points in the vary interesting commentary with eminent\n     > statisticians: Cox, Efron, Hoadley, and Parzen.*\n\n     - Chomsky, Noam (1956) [Three Models\n       for the Description of Language](http://www.chomsky.info/articles/195609--.pdf), *IRE Transactions on\n       Information theory* (2), pp. 113-124.\n       > *Compares finite state, phrase structure, and\n       > transformational grammars. Introduces \"colorless green ideas sleep\n       > furiously.\"*\n\n       - Chomsky, Noam (1967) [Syntactic Structures](http://books.google.com/books?id=SNeHkMXHcd8C&printsec=frontcover&dq=syntactic+structures+chomsky&hl=en&src=bmrr&ei=1WvcTa3UDeLQiAK6-4zpDw&sa=X&oi=book_result&ct=book-thumbnail&resnum=1&ved=0CD4Q6wEwAA#v=onepage&q=probabilistic&f=false), Mouton.\n         > *A book-length exposition of Chomsky's theory that was\n         > the leading exposition of linguistics for a decade. Claims that\n         > probabilistic models give no insight into syntax.*\n\n         - Chomsky, Noam (1969) [Some Empirical\n           Assumptions in Modern Philosophy of Language](http://books.google.com/books?id=iaXVXYDQN1oC&pg=PA296&dq=%22probability+of+a+sentence%22+%22entirely+useless%22&hl=en&ei=_s7eTc_5DrTciALcsvnlCg&sa=X&oi=book_result&ct=result&resnum=4&ved=0CDoQ6AEwAw#v=onepage&q=%22probability%20of%20a%20sentence%22%20%22entirely%20useless%22&f=false), in *Philosophy,\n           Science and Method: Essays in Honor or Ernest Nagel*,\n           St. Martin's Press.\n           > *Claims that the notion \"probability of a sentence\" is\n           > an entirely useless notion.*\n\n           - Chomsky, Noam (1981) [Lectures\n             on government and binding](http://books.google.com/books?id=l08tpkOOdNQC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q=pro-drop&f=false), de Gruyer.\n             > *A revision of Chomsky's theory; this version\n             > introduces Universal Grammar. We cite it for the coverage of\n             > parameters such as pro-drop.*\n\n             - Chomsky, Noam (1991) [Linguistics\n               and adjacent fields: a personal view](http://www.cpgb.org.uk/article.php?article_id=1004261#23), in Kasher (ed.), *A\n               Chomskyan Turn*, Oxford.\n               > *I found the Plato quotes in [this](http://www.cpgb.org.uk/article.php?article_id=1004261#23)\n               > article, published by the Communist Party of Great Britain, and\n               > apparently published by someone with no linguistics training\n               > whatsoever, but with a political agenda.*\n\n               - Gold, E. M. (1967) [Language\n                 Identification in the Limit](http://en.wikipedia.org/wiki/Language_identification_in_the_limit), *Information and Control*,\n                 Vol. 10, No. 5, pp. 447-474.\n                 > *Gold proved a result in formal language theory that\n                 > we can state (with some artistic license) as this: imagine a game\n                 > between two players, guesser and chooser. Chooser says to\n                 > guesser, \"Here is an infinite number of languages. I'm going to\n                 > choose one of them, and start reading sentences to you that come\n                 > from that language. On your N-th birthday there will be a True-False\n                 > quiz where I give you 100 sentences you haven't heard yet, and you\n                 > have to say whether they come from the language or not.\" There are\n                 > some limits on what the infinite set looks like and on how\n                 > the chooser can pick sentences (he can be deliberately\n                 > tricky, but he can't just repeat the same sentence over and over,\n                 > for example). Gold's\n                 > result is that if the infinite set of languages are all generated by\n                 > context-free grammars then there is no strategy for guesser that\n                 > guarantees she gets 100% correct every time, no matter what N you\n                 > choose for the birthday. This result was\n                 > taken by Chomsky and others to mean that it is impossible for\n                 > children to learn human languages without having an innate \"language\n                 > organ.\" As [Johnson (2004)](http://psyling.psy.cmu.edu/papers/years/2004/logical/gold-johnson.pdf ) and others show, this was an invalid\n                 > conclusion; the task of getting 100% on the quiz (which Gold called\n                 > *language identification*)\n                 > really has nothing in common with the task of *language acquisition*\n                 > performed by children, so Gold's Theorem has no\n                 > relevance.*\n\n                 - Horning, J. J. (1969) [A study of grammatical\n                   inference](http://portal.acm.org/citation.cfm?id=905718&coll=DL&dl=GUIDE&CFID=23829333&CFTOKEN=19917759), Ph.D. thesis, Stanford Univ.\n                   > *Where Gold found a negative result—that\n                   > context-free languages were not identifiable from examples, Horning found a positive result—that probabilistic\n                   > context-free languages are identifiable (to within an arbitrarily\n                   > small level of error). Nobody doubts that\n                   > humans have unique innate capabilities for understanding language\n                   > (although it is unknown to what extent these capabilities are\n                   > specific to language and to what extent they are general cognitive\n                   > abilities related to sequencing and forming abstractions). But\n                   > Horning proved in 1969 that Gold cannot be used as a convincing\n                   > argument for an innate language organ that specifies all of language\n                   > except for the setting of a few parameters.*\n\n                   - Johnson, Kent (2004) [Gold's\n                     Theorem and cognitive science](http://psyling.psy.cmu.edu/papers/years/2004/logical/gold-johnson.pdf), *Philosophy of Science*,\n                     Vol. 71, pp. 571-592.\n                     > *The best article I've seen on what Gold's Theorem\n                     > actually says and what has been claimed about it (correctly and\n                     > incorrectly). Concludes that Gold has something to say about formal\n                     > languages, but nothing about child language\n                     > acquisition.*\n\n                     - Lappin, Shalom and Shieber, Stuart M. (2007) [Machine\n                       learning theory and practice as a source of insight into universal\n                       grammar.](http://www.dcs.kcl.ac.uk/staff/lappin/papers/lappin-shieber_learning07.pdf), *Journal of Linguistics*, Vol. 43, No. 2,\n                       pp. 393-427.\n                       > *An excellent article discussing the poverty of the\n                       > stimulus, the fact that all models have bias, the difference\n                       > between supervised and unsupervised learning, and modern (PAC or VC)\n                       > learning theory. It provides\n                       > alternatives to the model of Universal Grammar consisting of a fixed\n                       > set of binary parameters.*\n\n                       - Manning, Christopher (2002) [Probabilistic\n                         Syntax](http://nlp.stanford.edu/~manning/papers/probsyntax.pdf), in Bod, Hay, and Jannedy (eds.), *Probabilistic\n                         Linguistics*, MIT Press.\n                         > *A compelling introduction to probabilistic syntax,\n                         > and how it is a better model for linguistic facts than categorical\n                         > syntax. Covers \"the joys and perils of corpus\n                         > linguistics.\"*\n\n                         - Norvig, Peter (2007) [How to Write a Spelling\n                           Corrector](http://norvig.com/spell-correct.html), unpublished web page.\n                           > *Shows working code to implement a probabilistic,\n                           > statistical spelling correction algorithm.*\n\n                           - Norvig, Peter (2009) [Natural\n                             Language Corpus Data](http://norvig.com/ngrams/), in Seagran and Hammerbacher (eds.),\n                             *Beautiful Data*, O'Reilly.\n                             > *Expands on the essay above; shows how to implement\n                             > three tasks: text segmentation, cryptographic decoding, and spelling\n                             > correction (in a slightly more complete form than the previous essay).*\n\n                             - Pereira, Fernando (2002) [Formal\n                               grammar and information theory: together again?](http://www.cis.upenn.edu/~pereira/papers/rsoc.pdf), in Nevin and\n                               Johnson (eds.), *The Legacy of Zellig Harris*, Benjamins.\n                               > *When I set out to write the page you are reading now,\n                               > I was concentrating on the events that took place in Cambridge,\n                               > Mass., 4800 km from home. After doing some research I was surprised\n                               > to learn that the authors of\n                               > two of the three best articles on this subject sit within a total of\n                               > 10 meters from my desk: Fernando Pereira and Chris Manning. (The\n                               > third, Steve Abney, sits 3700 km away.) But perhaps I shouldn't have\n                               > been surprised. I remember giving a talk at ACL on the corpus-based\n                               > language models used at Google, and having Fernando, then a\n                               > professor at U. Penn., comment \"I feel like I'm a particle physicist\n                               > and you've got the only super-collider.\" A few years later he moved\n                               > to Google. Fernando is also famous for his quote \"The older I get, the\n                               > further down the Chomsky Hierarchy I go.\" His article here covers\n                               > some of the same ground as mine, but he goes farther in explaining\n                               > the range of probabilistic models available and how they are useful.*\n\n                               - Plato (c. 380BC) [The\n                                 Republic](http://en.wikipedia.org/wiki/Allegory_of_the_Cave).\n                                 > *Cited here for the allegory of the cave.*\n\n                                 - Shannon, C.E. (1948) [A\n                                   Mathematical Theory of Communication](http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf), *The Bell System\n                                   Technical Journal*, Vol. 27, pp. 379-423.\n                                   > *An enormously influential article that started the\n                                   > field of information theory and introduced the term \"bit\" and the noisy\n                                   > channel model, demonstrated successive n-gram approximations of\n                                   > English, described Markov models of language, defined entropy with\n                                   > respect to these models, and enabled the growth of the\n                                   > telecommunications industry.*\n\n---\n\n[*Peter Norvig*](http://norvig.com)\n","body_html":"<h1 id=\"on-chomsky-and-the-two-cultures-of-statistical-learning\">On Chomsky and the Two Cultures of Statistical Learning</h1>\n<p>At the <a href=\"https://infinite.mit.edu/collection/mit-150-symposia\" rel=\"nofollow ugc noopener\">Brains,\nMinds, and Machines</a> symposium held during MIT&#39;s 150th birthday party in 2011, Technology Review\n<a href=\"http://www.technologyreview.com/computing/37525/?a=f\" rel=\"nofollow ugc noopener\">reports</a>\nthat Prof. Noam Chomsky</p>\n<blockquote><p>derided researchers in machine learning who use purely\nstatistical methods to produce behavior that mimics something in the\nworld, but who don&#39;t try to understand the meaning of that\nbehavior.</p></blockquote>\n<p>The <a href=\"http://languagelog.ldc.upenn.edu/myl/PinkerChomskyMIT.html\" rel=\"nofollow ugc noopener\">transcript</a> is now available, so let&#39;s quote\nChomsky himself:</p>\n<blockquote><p>It&#39;s true there&#39;s been a lot of work on trying to apply statistical models to\nvarious linguistic problems. I think there have been some successes, but a lot of failures.\nThere is a notion of success ... which I think is novel in the history of science.\nIt interprets success as approximating unanalyzed data.</p></blockquote>\n<p>Noam Chomsky</p>\n<p>Chomsky&#39;s remarks were in response to Steven Pinker&#39;s question\nabout the success of probabilistic models trained with statistical methods.\nThis essay is a response to Pinker and Chomsky, and will address these questions:</p>\n<ol><li><p>What did Chomsky mean, and is he right?- What is a statistical model?- How successful are statistical language models?- Is there anything like their notion of success in the history of</p><pre><code>   science?- What doesn&#39;t Chomsky like about statistical models?</code></pre></li></ol>\n<h2 id=\"what-did-chomsky-mean-and-is-he-right\">What did Chomsky mean, and is he right?</h2>\n<p>I take Chomsky&#39;s points to be the following:</p>\n<ol><li><p>Statistical language models have had engineering success, but that</p><p> is irrelevant to science.- Accurately modeling linguistic facts is just butterfly collecting;\n   what matters in science (and specifically linguistics) is the underlying principles.- Statistical models are incomprehensible; they provide no insight.- Statistical models may provide an accurate simulation of some\n       phenomena, but the simulation is done completely the wrong way; people don&#39;t\n       decide what the third word of a sentence should be by consulting a\n       probability table keyed on the previous words, rather they map\n       from an internal semantic form to a syntactic tree-structure, which\n       is then linearized into words. This is done without any probability\n       or statistics.- Statistical models have been proven incapable of learning\n         language; therefore language must be innate, so why are these\n         statistical modelers wasting their time on the wrong enterprise?</p></li></ol>\n<p>Is he right? That&#39;s a long-standing debate. These are my short answers:</p>\n<ol><li><p>I agree that engineering success is not the sole goal or the measure of</p><p> science. But I observe that science and engineering develop together,\n and that engineering success shows that something is working right,\n and so is evidence (but not proof) of a scientifically successful model.- Science is a combination of gathering facts and making theories;\n   neither can progress on its own. In the history of\n   science, the laborious accumulation of\n   facts is the dominant mode, not a novelty. The science of understanding language is no different than\n   other sciences in this respect.- I agree that it can be difficult to make sense of a model\n     containing billions of parameters. Certainly a human can&#39;t understand\n     such a model by inspecting the values of each parameter individually. But one\n     can gain insight by examing the <em>properties</em> of the model—where\n     it succeeds and fails, how well it learns as a function of data, etc.- I agree that a Markov model of word probabilities cannot model all\n       of language. It is equally true that a concise tree-structure model without\n       probabilities cannot model all of language. What is needed is a\n       probabilistic model that covers words, syntax, semantics, context,\n       discourse, etc. Chomsky dismisses all probabilistic models because\n       of shortcomings of a particular 50-year old probabilistic model.\n       I understand how Chomsky might arrive at the conclusion that\n       probabilistic models are unnecessary, from his\n       study of the <em>generation</em> of language. But the vast majority of people who study\n       <em>interpretation</em> tasks (such as speech recognition) quickly see that\n       interpretation is an inherently probabilistic problem: given a\n       stream of noisy input to\n       my ears, what did the speaker most likely mean? Einstein said to\n       make everything as simple as possible, but no\n       simpler. Many phenomena in science are stochastic, and the simplest\n       model of them is a probabilistic model; I believe language is such a\n       phenomenon and therefore that probabilistic models are our best tool\n       for representing facts about language, for algorithmically\n       processing language, and for understanding how humans process language.- In 1967, Gold&#39;s Theorem showed some theoretical limitations of logical\n         deduction on formal mathematical languages. But this result has\n         nothing to do with the task faced by learners of natural language.\n         In any event, by 1969 we knew that probabilistic inference\n         (over probabilistic context-free grammars) is not subject to those limitations (Horning showed that learning of PCFGs is possible).\n         I agree with Chomsky that it is undeniable that humans\n         have some innate capability to learn natural language, but we don&#39;t\n         know enough about that capability to say how it works; it certainly could\n         use something like probabilistic language representations and\n         statistical learning. And we don&#39;t know if the innate ability is specific to language, or\n         is part of a more general ability that works for language and other things.</p></li></ol>\n<p>The rest of this essay consists of longer versions of each answer.</p>\n<h2 id=\"what-is-a-statistical-model\">What is a statistical model?</h2>\n<p>A <strong>statistical model</strong> is a mathematical model which is modified\nor trained by the input of data points. Statistical models are often\nbut not always probabilistic. Where the\ndistinction is important we will be careful not to just say\n&quot;statistical&quot; but to use the following component terms:</p>\n<ul><li><p>A <strong>mathematical model</strong> specifies a relation among</p><p>variables, either in functional form that maps inputs\nto outputs (e.g. *y* = *m* *x* + *b*) or in\nrelation form (e.g. the following (*x*, *y*) pairs are\npart of the relation).* A <strong>probabilistic model</strong> specifies a\n  probability distribution over possible values of random variables,\n  e.g., P(*x*, *y*),\n  rather than a strict deterministic relationship, e.g., *y* = *f*(*x<em>).</em> A <strong>trained model</strong> uses some training/learning algorithm to\n    take as input a collection of possible models and a collection of\n    data points (e.g. (*x*,\n    *y*) pairs) and select the best model. Often this is in the form of\n    choosing the values of parameters (such as\n    *m* and *b* above) through a process of\n    statistical inference.</p></li></ul>\n<p>Claude Shannon</p>\n<p>For example, a decade before Chomsky, Claude Shannon <a href=\"http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf\" rel=\"nofollow ugc noopener\">proposed probabilistic models of\ncommunication</a> based on Markov chains of words. If you have\na vocabulary of 100,000 words and a second-order Markov model in which the probability\nof a word depends on the previous two words, then you need a quadrillion (1015)\nprobability values to specify the model. The only feasible way to learn\nthese 1015 values is to gather statistics from data and introduce\nsome smoothing method for the many cases where there is no\ndata. Therefore, most (but not all) probabilistic models are\ntrained. Also, many (but not all) trained models are probabilistic.</p>\n<p>As another example, consider the Newtonian model of gravitational\nattraction, which says that the force between two objects of mass\n*m*1 and *m*2 a distance *r* apart\nis given by</p>\n<blockquote><p>*F* = *G* *m*1 *m*2 / *r*2</p></blockquote>\n<p>where *G* is the universal gravitational constant. This is a\ntrained model because the gravitational constant *G* is\ndetermined by statistical inference over the results of a series of\nexperiments that contain stochastic experimental error. It is also a\ndeterministic (non-probabilistic) model because it states an exact\nfunctional relationship.\nI believe that Chomsky has no objection to\nthis kind of statistical model. Rather, he seems to reserve his\ncriticism for statistical models like Shannon&#39;s that have quadrillions of\nparameters, not just one or two.</p>\n<p>(This example brings up another distinction:\nthe gravitational model is <strong>continuous</strong> and <strong>quantitative</strong>\nwhereas the linguistic tradition has favored models that are\n<strong>discrete</strong>, <strong>categorical</strong>, and <strong>qualitative</strong>: a word is or is not a verb,\nthere is no question of its degree of verbiness. For more on these\ndistinctions, see Chris Manning&#39;s article on <a href=\"http://nlp.stanford.edu/~manning/papers/probsyntax.pdf\" rel=\"nofollow ugc noopener\">Probabilistic\nSyntax</a>.)</p>\n<p>A relevant probabilistic statistical model is the <a href=\"http://en.wikipedia.org/wiki/Ideal_gas_law\" rel=\"nofollow ugc noopener\">ideal gas law</a>,\nwhich describes the pressure *P* of a gas in terms of the the\nnumber of molecules, *N*, the temperature *T*, and\nBoltzmann&#39;s constant, *K*:</p>\n<blockquote><p>*P* = <em>N k T / V</em>.</p></blockquote>\n<p>The equation can be derived from first principles\nusing the tools of statistical mechanics. It is an uncertain,\nincorrect model; the <em>true</em> model would have to describe the\nmotions of individual gas molecules. This model ignores that\ncomplexity and <em>summarizes</em> our uncertainty about the location of\nindividual molecules. Thus, even though it is statistical and\nprobabilistic, even though it does not completely model reality,\nit does provide both good predictions and insight—insight that\nis not available from trying to understand the <em>true</em> movements\nof individual molecules.</p>\n<p>Now let&#39;s consider the non-statistical\nmodel of spelling expressed by the rule &quot;<em>I before E except after C.</em>&quot;\nCompare that to the probabilistic, trained statistical model:</p>\n<blockquote><pre><code>P(IE) = 0.0177         P(CIE) = 0.0014        P(*IE) = 0.163\nP(EI) = 0.0046         P(CEI) = 0.0005        P(*EI) = 0.0041</code></pre></blockquote>\n<p>This model comes from statistics on a <a href=\"http://norvig.com/ngrams/\" rel=\"nofollow ugc noopener\">corpus of a trillion words</a> of English\ntext. The notation P(IE) is the probability that a word sampled from\nthis corpus contains the consecutive letters &quot;IE.&quot; P(CIE) is the probability that a word contains the consecutive letters &quot;CIE&quot;, and P(*IE) is the probability of any letter other than C followed by IE.\nThe statistical\ndata confirms that IE is in fact more common than EI (by almost 4 to 1), and that the dominance\nof IE lessens when following a C, but contrary to the rule, CIE is still more common than CEI, by almost 3 to 1. Examples of &quot;CIE&quot; words\ninclude &quot;science,&quot; &quot;society,&quot; &quot;ancient&quot; and &quot;species.&quot;\nThe disadvantage of the &quot;I\nbefore E except after C&quot; model is that it is not very accurate. Consider:</p>\n<pre><code>Accuracy(&quot;I before E&quot;) = 0.0177/(0.0177+0.0046) = 0.793\nAccuracy(&quot;I before E except after C&quot;) = (0.0005+0.0163)/(0.0005+0.0163+0.0014+0.0041) = 0.753</code></pre>\n<p>A more complex statistical model (say, one that gave the probability of all\n4-letter sequences, and/or of all known words) could be <a href=\"http://norvig.com/spell-correct.html\" rel=\"nofollow ugc noopener\">ten times more\naccurate</a> at the task of spelling, but does not offer concise <strong>insight</strong> into how spelling\nworks. Insight would require a model that knows about phonemes,\nsyllabification, and language of origin. Such a model could be trained (or not) and\nprobabilistic (or not).</p>\n<p>As another example of insight,\nconsider the Theory of Supreme Court Justice Hand-Shaking: when the\nsupreme court convenes, all attending justices shake hands with every\nother justice. The number of attendees, *n*, must be an integer\nin the range 0 to 9; what is the total number of handshakes, *h*\nfor a given *n*? Here are three possible explanations:</p>\n<ol><li><p>Each of *n* justices shakes hands with the other</p><p> *n* - 1 justices, but that counts Alito/Breyer and Breyer/Alito as\n two separate shakes, so we should cut the total in half, and we end up\n with *h* = *n* × (*n* - 1) / 2.- To avoid double-counting, we will order the justices by\n   seniority and only count a more-senior/more-junior handshake, not a\n   more-junior/more-senior one. So we count, for each justice, the\n   shakes with the more junior justices, and sum them up, giving *h*\n   = Σ*i* = 1 .. *n*\n   (*i* - 1).- Just look at this table:</p>\n<pre><code> |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |\n | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n | *n*: 0 1 2 3 4 5 6 7 8 9|  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | *h*: 0 0 1 3 6 10 15 21 28 36 | | | | | | | | | | | | | | | | | | | | | |</code></pre></li></ol>\n<p>Some people might prefer A, some might prefer B, and if you are slow\nat doing multiplication or addition you might prefer C.\nWhy? All three explanations describe <em>exactly the same theory</em> —\nthe same function from *n* to *h*, over the entire domain of\npossible values of *n* (as long as there are no more than 9 supreme court justices). Thus we could prefer A (or B) over C\nonly for reasons other than the theory itself. We might find that A\nor B gave us a better understanding of the problem. A and B are\ncertainly more useful than C for figuring out what happens if Congress\nexercises its power to add an additional justice. Theory A\nmight be most helpful in developing a theory of handshakes at the end\nof a hockey game (when each player shakes hands with players on the\nopposing team) or in proving that the number of people who shook an\nodd number of hands at the MIT Symposium is even.</p>\n<h2 id=\"how-successful-are-statistical-language-models\">How successful are statistical language models?</h2>\n<p>Chomsky said words to the effect that statistical language models have\nhad some limited success\nin some application areas. Let&#39;s look\nat computer systems that deal with language, and at the notion of\n&quot;success&quot; defined by &quot;making accurate predictions about the world.&quot; First, the\nmajor application areas:</p>\n<ul><li><p><strong>Search engines:</strong> 100% of major players are trained</p><p>and probabilistic. Their operation cannot be described by a simple function.* <strong>Speech recognition:</strong> 100% of major systems are\n  trained and probabilistic, mostly relying on probabilistic\n  hidden Markov models.* <strong>Machine translation:</strong> 100% of top competitors in\n    competitions such as <a href=\"http://www.itl.nist.gov/iad/mig/tests/mt/2009/ResultsRelease/currentArabic.html\" rel=\"nofollow ugc noopener\">NIST</a>\n    use statistical methods. Some commercial systems use a hybrid of\n    trained and rule-based approaches. Of the 4000 language pairs\n    covered by machine translation systems, a statistical system is by\n    far the best for every pair except Japanese-English, where the top\n    statistical system is roughly equal to the top hybrid system.* <strong>Question answering:</strong> this application is less\n      well-developed, and many systems build heavily on the statistical\n      and probabilistic approach used by search engines. The <a href=\"http://www-03.ibm.com/innovation/us/watson/index.html\" rel=\"nofollow ugc noopener\">IBM\n      Watson</a> system that recently won on Jeopardy is thoroughly\n      probabilistic and trained, while Boris Katz&#39;s <a href=\"http://groups.csail.mit.edu/infolab/publications/Katz-etal-TREC2003.pdf\" rel=\"nofollow ugc noopener\">START</a>\n      is a hybrid. All systems use at least some statistical techniques.</p></li></ul>\n<p>Now let&#39;s look at some components that are of interest only to\nthe computational linguist, not to the end user:</p>\n<ul><li><p><strong>Word sense disambiguation:</strong> 100% of top competitors at the</p><p><a href=\"http://semeval2.fbk.eu/semeval2.php?location=SemEval2010-Program\" rel=\"nofollow ugc noopener\">SemEval-2</a>\ncompetition used statistical techniques; most are probabilistic; some use a hybrid approach\nincorporating rules from sources such as Wordnet.* <strong>Coreference resolution:</strong> The majority of current systems\n  are statistical, although we should mention the system of <a href=\"http://www.aclweb.org/anthology/D/D09/D09-1120.pdf\" rel=\"nofollow ugc noopener\">Haghighi\n  and Klein</a>, which can be described as a hybrid system that is\n  mostly rule-based rather than trained, and performs on par with\n  top statistical systems.* <strong>Part of speech tagging:</strong> Most current systems are\n    statistical. The <a href=\"http://en.wikipedia.org/wiki/Brill_tagger\" rel=\"nofollow ugc noopener\">Brill tagger</a> stands out as a successful hybrid\n    system: it learns a set of deterministic rules from statistical\n    data.* <strong>Parsing:</strong> There are many parsing systems, using multiple\n      approaches. Almost all of the <a href=\"http://csep.psyc.memphis.edu/pdf/Evaluating_treebank_Parsers_Coh-Metrix.pdf\" rel=\"nofollow ugc noopener\">most\n      successful</a> are statistical, and the majority are <a href=\"http://www.cs.berkeley.edu/~pliang/papers/hdppcfg-haba.pdf\" rel=\"nofollow ugc noopener\">probabilistic</a>\n      (with a substantial minority of deterministic parsers).</p></li></ul>\n<p>Clearly, it is inaccurate to say that statistical models\n(and probabilistic models) have achieved <em>limited</em> success; rather they\nhave achieved an <em>overwhelmingly dominant</em> (although not exclusive) position.</p>\n<p>Another measure of success is the degree to which an idea captures\na community of researchers. As Steve Abney <a href=\"http://www.vinartus.net/spa/95c.pdf\" rel=\"nofollow ugc noopener\">wrote</a> in 1996, &quot;In the\nspace of the last ten years, statistical methods have gone from being\nvirtually unknown in computational linguistics to being a fundamental\ngiven. ... anyone who cannot at least use the terminology persuasively\nrisks being mistaken for kitchen help at the ACL [Association for\nComputational Linguistics] banquet.&quot;</p>\n<p>Now of course, the majority doesn&#39;t rule -- just because everyone is jumping on some bandwagon, that doesn&#39;t make it right.\nBut I made the switch: after about 14 years of trying to get language models to work using logical rules,\nI started to adopt probabilistic approaches (thanks to pioneers like Gene Charniak (and Judea Pearl for probability in general)\nand to my colleagues who were early adopters, like Dekai Wu).\nAnd I saw everyone around me making the same switch.\nAnd I didn&#39;t see anyone going in the other direction. We all saw the limitations of the old tools, and the benefits of the new.</p>\n<p>And while it may seem crass and anti-intellectual to consider a\nfinancial measure of success, I note that the intellectual\n<a href=\"http://en.wikipedia.org/wiki/Telecommunication\" rel=\"nofollow ugc noopener\">offspring</a>\nof Shannon&#39;s theory create several trillion dollars of revenue each\nyear, while the <a href=\"http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias%3Daps&amp;field-keywords=chomsky&amp;x=0&amp;y=0\" rel=\"nofollow ugc noopener\">offspring</a>\nof Chomsky&#39;s theories generate well under a billion.</p>\n<p>This section has shown that one reason why the vast majority of\nresearchers in computational linguistics use statistical models is an\n<em>engineering</em> reason: statistical models have state-of-the-art\nperformance, and in most cases non-statistical models perform worst.\nFor the remainder of this essay we will concentrate on\n<em>scientific</em> reasons: that probabilistic models better represent\nlinguistic facts, and statistical techniques make it easier for us to\nmake sense of those facts.</p>\n<h2 id=\"is-there-anything-like-it-the-statistical-notion-of-success-in-t\">Is there anything like it [the statistical notion of success] in the history of science?</h2>\n<p>When Chomsky said &quot;*That&#39;s a\nnotion of [scientific] success that&#39;s very novel. I don&#39;t know of\nanything like it in the history of science*&quot; he apparently meant\nthat the notion of success of &quot;accurately modeling the world&quot; is\nnovel, and that the only true measure of success in the history of science\nis &quot;providing insight&quot; — of answering <em>why</em> things are the way\nthey are, not just describing <em>how</em> they are.</p>\n<p>A <a href=\"http://www.google.com/webhp?sourceid=chrome-instant&amp;ie=UTF-8&amp;ion=1&amp;nord=1#sclient=psy&amp;hl=en&amp;tbo=1&amp;nord=1&amp;tbs=dfn:1&amp;source=hp&amp;q=science&amp;aq=f&amp;aqi=p-p2g-e2g-c2g2g-c1g1&amp;aql=&amp;oq=&amp;pbx=1&amp;tbo=1&amp;bav=on.2,or.r_gc.r_pw.&amp;fp=18c4c2d0f0f5fea9&amp;biw=1186&amp;bih=634\" rel=\"nofollow ugc noopener\">dictionary definition</a> of science is &quot;the systematic study of the\nstructure and behavior of the physical and natural world through\nobservation and experiment,&quot; which stresses accurate modeling over insight, but it\nseems to me that both notions have always coexisted as part of doing\nscience. To test that, I consulted the epitome of doing science,\nnamely <em><a href=\"http://www.sciencemag.org\" rel=\"nofollow ugc noopener\">Science</a></em>. I\nlooked at the current issue and chose a title and abstract at random:</p>\n<blockquote><p><a href=\"http://www.sciencemag.org/content/332/6032/944.abstract\" rel=\"nofollow ugc noopener\">Chlorinated Indium Tin Oxide Electrodes with High Work Function for\nOrganic Device Compatibility</a></p>\n<p>In organic light-emitting diodes (OLEDs), a stack of multiple organic\nlayers facilitates charge flow from the low work function [~4.7\nelectron volts (eV)] of the transparent electrode (tin-doped indium\noxide, ITO) to the deep energy levels (~6 eV) of the active\nlight-emitting organic materials. We demonstrate a chlorinated ITO\ntransparent electrode with a work function of &gt;6.1 eV that provides a\ndirect match to the energy levels of the active light-emitting\nmaterials in state-of-the art OLEDs. A highly simplified green OLED\nwith a maximum external quantum efficiency (EQE) of 54% and power\nefficiency of 230 lumens per watt using outcoupling enhancement was\ndemonstrated, as were EQE of 50% and power efficiency of 110 lumens\nper watt at 10,000 candelas per square meter.</p></blockquote>\n<p>It certainly seems that this article is much more focused on\n&quot;accurately modeling the world&quot; than on &quot;providing insight.&quot; The\npaper does indeed fit in to a body of theories, but it is mostly\nreporting on specific experiments and the results obtained from them\n(e.g. efficiency of 54%).</p>\n<p>I then looked at all the titles and abstracts from the <a href=\"http://www.sciencemag.org/content/332/6032.toc\" rel=\"nofollow ugc noopener\">current\nissue</a> of <em>Science</em>:</p>\n<ul><li>Comparative Functional Genomics of the Fission Yeasts* Dimensionality Control of Electronic Phase Transitions in Nickel-Oxide Superlattices* Competition of Superconducting Phenomena and Kondo Screening at the Nanoscale* Chlorinated Indium Tin Oxide Electrodes with High Work Function for Organic Device Compatibility* Probing Asthenospheric Density, Temperature, and Elastic Moduli Below the Western United States* Impact of Polar Ozone Depletion on Subtropical Precipitation* Fossil Evidence on Origin of the Mammalian Brain* Industrial Melanism in British Peppered Moths Has a Singular and Recent Mutational Origin* The Selaginella Genome Identifies Genetic Changes Associated with the Evolution of Vascular Plants* Chromatin &quot;Prepattern&quot; and Histone Modifiers in a Fate Choice for Liver and Pancreas* Spatial Coupling of mTOR and Autophagy Augments Secretory Phenotypes* Diet Drives Convergence in Gut Microbiome Functions Across Mammalian Phylogeny and Within Humans* The Toll-Like Receptor 2 Pathway Establishes Colonization by a Commensal of the Human Microbiota* A Packing Mechanism for Nucleosome Organization Reconstituted Across a Eukaryotic Genome* Structures of the Bacterial Ribosome in Classical and Hybrid States of tRNA Binding</li></ul>\n<p>and did the same for the <a href=\"http://www.cell.com/issue?pii=S0092-8674(11)X0010-7\" rel=\"nofollow ugc noopener\">current issue</a> of <em>Cell</em>:</p>\n<ul><li><p>Mapping the NPHP-JBTS-MKS Protein Network Reveals Ciliopathy Disease Genes and Pathways* Double-Strand Break Repair-Independent Role for BRCA2 in</p><p>  Blocking Stalled Replication Fork Degradation by MRE11* Establishment and Maintenance of Alternative Chromatin States at a Multicopy Gene Locus* An Epigenetic Signature for Monoallelic Olfactory Receptor Expression* Distinct p53 Transcriptional Programs Dictate Acute DNA-Damage Responses and Tumor Suppression* An ADIOL-ERβ-CtBP Transrepression Pathway Negatively Regulates Microglia-Mediated Inflammation* A Hormone-Dependent Module Regulating Energy Balance* Class IIa Histone Deacetylases Are Hormone-Activated\n              Regulators of FOXO and Mammalian Glucose Homeostasis</p></li></ul>\n<p>and for the <a href=\"http://nobelprize.org/nobel_prizes/lists/year/\" rel=\"nofollow ugc noopener\">2010 Nobel\nPrizes</a> in science:</p>\n<ul><li><p>Physics: *for groundbreaking experiments regarding</p><p>the two-dimensional material graphene** Chemistry: *for palladium-catalyzed cross couplings in organic\n  synthesis** Physiology or Medicine: *for the development of in vitro\n    fertilization*</p></li></ul>\n<p>My conclusion is that 100% of these articles and awards are more about\n&quot;accurately modeling the world&quot; than they are about &quot;providing\ninsight,&quot; although they all have some\ntheoretical insight component as well. I recognize that judging one\nway or the other is a difficult ill-defined task, and that you\nshouldn&#39;t automatically accept my judgements, because I may have an inherent bias. (I\nwas considering running an experiment on Mechanical Turk to get an\nunbiased answer, but those familiar with Mechanical Turk told me these\nquestions are probably too hard for the average Turker. So you the reader can do your own\nexperiment and see if you agree.)</p>\n<h2 id=\"what-doesn-t-chomsky-like-about-statistical-models\">What doesn&#39;t Chomsky like about statistical models?</h2>\n<p>I said that statistical models are sometimes confused with\nprobabilistic models; let&#39;s first consider the extent to which Chomsky&#39;s\nobjections are actually about probabilistic models. In 1969 he\nfamously <a href=\"http://books.google.com/books?id=iaXVXYDQN1oC&amp;pg=PA296&amp;dq=%22probability+of+a+sentence%22+%22entirely+useless%22&amp;hl=en&amp;ei=_s7eTc_5DrTciALcsvnlCg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=4&amp;ved=0CDoQ6AEwAw#v=onepage&amp;q=%22probability%20of%20a%20sentence%22%20%22entirely%20useless%22&amp;f=false\" rel=\"nofollow ugc noopener\">wrote</a>:</p>\n<blockquote><p>But it must be recognized that the notion of &quot;probability of a sentence&quot; is an entirely useless one, under any known interpretation of this term.</p></blockquote>\n<p>His main argument being that, under any interpretation known to him,\nthe probability of a novel sentence must be zero, and since novel sentences\nare in fact generated all the time, there is a contradiction. The resolution of\nthis contradiction is of course that it is not necessary to assign a\nprobability of zero to a novel sentence; in fact, with current\nprobabilistic models it is standard practice to do smoothing and assign a non-zero\nprobability to novel occurrences. So this criticism is invalid, but was\nvery influential for decades.\nPreviously, in\n<a href=\"http://books.google.com/books?id=SNeHkMXHcd8C&amp;printsec=frontcover&amp;dq=syntactic+structures+chomsky&amp;hl=en&amp;src=bmrr&amp;ei=1WvcTa3UDeLQiAK6-4zpDw&amp;sa=X&amp;oi=book_result&amp;ct=book-thumbnail&amp;resnum=1&amp;ved=0CD4Q6wEwAA#v=onepage&amp;q=probabilistic&amp;f=false\" rel=\"nofollow ugc noopener\">*Syntactic\nStructures*</a> (1957) Chomsky wrote:</p>\n<blockquote><p>I think we are forced to conclude that ... probabilistic models give no\nparticular insight into some of the basic problems of syntactic structure.</p></blockquote>\n<p>In the footnote to this conclusion he considers the possibility\nof a useful probabilistic/statistical model, saying &quot;I would certainly\nnot care to argue that ... is unthinkable, but I know of no suggestion\nto this effect that does not have obvious flaws.&quot; The main &quot;obvious flaw&quot; is this:\nConsider:</p>\n<ol><li><strong>I** never, ever, ever, ever, ... **fiddle</strong> around in any way with electrical equipment.- <strong>She</strong> never, ever, ever, ever, ... <strong>fiddles</strong> around in any way with electrical equipment.- * <strong>I** never, ever, ever, ever, ... **fiddles</strong> around in any way with electrical equipment.- * <strong>She</strong> never, ever, ever, ever, ... <strong>fiddle</strong> around in any way with electrical equipment.</li></ol>\n<p>No matter how many repetitions of &quot;ever&quot; you insert, sentences 1 and 2\nare grammatical and 3 and 4 are ungrammatical. A probabilistic\nMarkov-chain model\nwith *n* states can never make the necessary distinction (between\n1 or 2 versus 3 or 4) when there are more than *n* copies of &quot;ever.&quot;\nTherefore, a probabilistic Markov-chain model cannot handle all of English.</p>\n<p>This criticism is correct, but it is a criticism only of Markov-chain\nmodels, not of probabilistic models (or\ntrained models) in general. Since 1957 we have seen many\ntypes of probabilistic language models beyond the Markov-chain word\nmodels. Examples 1-4 above can in fact be distinguished with a\nfinite-state model that is not a chain, but other examples require\nmore sophisticated models. The best studied is probabilistic context-free grammar (PCFG),\nwhich operates over trees, categories of words, and individual lexical\nitems, and has none of the\nrestrictions of finite-state models. We find that PCFGs are\nstate-of-the-art for parsing performance and are easier to learn from\ndata than nonprobabilistic categorical context-free grammars. Other types of probabilistic\nmodels cover semantic and discourse structures.</p>\n<p>Every probabilistic model\nis a superset of a deterministic model (because the deterministic model could be seen as a\nprobabilistic model where the probabilities are restricted to be 0 or\n1), so any valid criticism of probabilistic models would have to be\nbecause they are too expressive, not because they are not expressive\nenough.</p>\n<p>In <em>Syntactic Structures</em>, Chomsky introduces a now-famous\nexample that is another criticism of finite-state probabilistic models:</p>\n<blockquote><p>Neither (a) &#39;colorless green ideas sleep furiously&#39; nor (b) &#39;furiously\nsleep ideas green colorless&#39;, nor any of their parts, has ever occurred\nin the past linguistic experience of an English speaker. But (a) is\ngrammatical, while (b) is not.</p></blockquote>\n<p>Chomsky appears to be correct that neither sentence appeared in the\npublished literature before 1955. I&#39;m not sure what he meant by &quot;any\nof their parts,&quot; but certainly every two-word part had occurred, for example:</p>\n<ul><li><p>&quot;It is neutral green, <strong>colorless green</strong>, like the glaucous water lying in</p><p>a cellar.&quot;\n<a href=\"http://books.google.com/books?id=QLUaAAAAMAAJ&amp;q=%22colorless+green%22&amp;dq=%22colorless+green%22&amp;hl=en&amp;ei=iZjcTZj1FMHhiAL9yqUF&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=7&amp;ved=0CE0Q6AEwBg\" rel=\"nofollow ugc noopener\">The\nParis we remember</a>, Elisabeth Finley Thomas (1942).* &quot;To specify those <strong>green ideas</strong> is hardly necessary, but you may\n  observe Mr. [D. H.] Lawrence in the role of the satiated aesthete.&quot;\n  <a href=\"http://books.google.com/books?id=FodMAAAAYAAJ&amp;pg=PA184&amp;dq=%22green+ideas%22&amp;hl=en&amp;ei=vZncTYaLHsbYiAKz2vDoDw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=3&amp;ved=0CEIQ6AEwAjgK\" rel=\"nofollow ugc noopener\">The\n  New Republic: Volume 29</a> p. 184, William White (1922).* <strong>&quot;Ideas sleep</strong> in books.&quot; <a href=\"http://books.google.com/books?id=fTciAQAAIAAJ&amp;pg=PA96&amp;dq=%22ideas+sleep%22&amp;hl=en&amp;ei=bZrcTePRBeHmiAKp2fHoDw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=3&amp;sqi=2&amp;ved=0CDoQ6AEwAg#v=onepage&amp;q=%22ideas%20sleep%22&amp;f=false\" rel=\"nofollow ugc noopener\">Current\n    Opinion: Volume 52</a>, (1912).</p></li></ul>\n<p>But regardless of what is meant by &quot;part,&quot; a\nstatistically-trained finite-state model <em>can</em> in fact distinguish\nbetween these two sentences. Pereira (2001) <a href=\"http://www.cis.upenn.edu/~pereira/papers/rsoc.pdf\" rel=\"nofollow ugc noopener\">showed</a>\nthat such a model, augmented with word categories and trained by\nexpectation maximization on newspaper text, computes that (a) is 200,000 times more\nprobable than (b). To prove that this was not the result of Chomsky&#39;s\nsentence itself sneaking into newspaper text, I repeated the\nexperiment, using a much cruder model\nwith Laplacian smoothing and no categories, trained over the <a href=\"http://ngrams.googlelabs.com/graph?content=colorless+green&amp;year_start=1800&amp;year_end=1954&amp;corpus=0&amp;smoothing=3\" rel=\"nofollow ugc noopener\">Google\nBook corpus</a> from 1800 to 1954, and found that (a) is about 10,000\ntimes more probable.</p>\n<p>Furthermore, the statistical models are capable of delivering the\njudgment that both\nsentences are <em>extremely</em> improbable, when compared to, say,\n&quot;Effective green products sell well.&quot; Chomsky&#39;s theory, being categorical,\ncannot make this distinction; all it can distinguish is\ngrammatical/ungrammatical.</p>\n<p>Another part of Chomsky&#39;s objection is &quot;we cannot seriously propose\nthat a child learns the values of 109 parameters in a\nchildhood lasting only 108 seconds.&quot; (Note that modern\nmodels are much larger than the\n109 parameters that were contemplated in the 1960s.) But of course\nnobody is proposing that these parameters are learned one-by-one; the\nright way to do learning is to set\nlarge swaths of near-zero parameters simultaneously with a\nsmoothing or regularization procedure, and update the high-probability\nparameters continuously as observations comes in. Nobody is\nsuggesting that Markov models by themselves are a serious model of human language\nperformance. But I (and others) suggest that probabilistic, trained\nmodels are a better model of human language performance than are\ncategorical, untrained models. And yes, it seems clear that an adult\nspeaker of English does know billions of language facts (a speaker knows many\nfacts about the appropriate uses of words in different contexts, such as that one\nsays &quot;the big game&quot; rather than &quot;the large game&quot; when talking about an\nimportant football game). These facts must somehow be encoded in the\nbrain.</p>\n<p>It seems clear that probabilistic models are better for judging the\nlikelihood of a sentence, or its degree of sensibility. But even if\nyou are not interested in these factors and are only interested in the\ngrammaticality of sentences, it still seems that probabilistic models\ndo a better job at describing the linguistic facts. The\n<em>mathematical</em> theory of <a href=\"http://en.wikipedia.org/wiki/Formal_language\" rel=\"nofollow ugc noopener\">formal\nlanguages</a> defines a language as a set of sentences. That is,\nevery sentence is either grammatical or ungrammatical; there is no\nneed for probability in this framework. But natural languages are not\nlike that. A <em>scientific</em> theory of natural languages must\naccount for the many phrases and sentences which leave a native\nspeaker uncertain about their grammaticality (see Chris Manning&#39;s <a href=\"http://nlp.stanford.edu/~manning/papers/probsyntax.pdf\" rel=\"nofollow ugc noopener\">article</a>\nand its discussion of the phrase &quot;<a href=\"http://ngrams.googlelabs.com/graph?content=as+least+as&amp;year_start=1800&amp;year_end=2000&amp;corpus=0&amp;smoothing=3\" rel=\"nofollow ugc noopener\">as\nleast as</a>&quot;), and there are phrases which some speakers find\nperfectly grammatical, others perfectly ungrammatical, and still\nothers will flip-flop from one occasion to the next. Finally, there\nare usages which are rare in a language, but cannot be dismissed if\none is concerned with actual data. For example, the verb <em>quake</em>\nis listed as intransitive in dictionaries, meaning that (1) below is\ngrammatical, and (2) is not, according to a categorical theory of\ngrammar.</p>\n<ol><li>The earth quaked.- ? It quaked her bowels.</li></ol>\n<p>But (2) <a href=\"http://books.google.com/books?id=nwmgdvN_akAC&amp;pg=PA107&amp;lpg=PA107&amp;dq=%22quaked+her%22&amp;source=bl&amp;ots=9M4UljFMWl&amp;sig=2BHbXzUY8dmZXEDXkoflNmrf_Zg&amp;hl=en&amp;ei=ItXeTYC0Js3SiAKnmon3Cg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CB4Q6AEwADgK#v=onepage&amp;q=%22quaked%20her%22&amp;f=false\" rel=\"nofollow ugc noopener\">actually</a>\n<a href=\"http://www.lexchecker.org/showhygramexample.php?lemmapos_id=1108&amp;hygram_len=3&amp;target_position=0&amp;max_ignore_rate=50&amp;hygram_id=149\" rel=\"nofollow ugc noopener\">appears</a>\nas a sentence of English. This poses a dilemma for the categorical\ntheory. When (2) is observed we must either arbitrarily\ndismiss it as an error that is outside the bounds of our model\n(without any theoretical grounds for doing so), or we must\nchange the theory to allow (2), which often results in the acceptance\nof a flood of\nsentences that we would prefer to remain ungrammatical. As Edward\nSapir <a href=\"http://books.google.com/books?id=ofgrAAAAYAAJ&amp;pg=PA39&amp;dq=%22all+grammars+leak%22+sapir+1921&amp;hl=en&amp;ei=wefeTaXhKMTPiALfw6jWCg&amp;sa=X&amp;oi=book_result&amp;ct=book-thumbnail&amp;resnum=10&amp;ved=0CFsQ6wEwCQ#v=onepage&amp;q=all%20grammars%20leak&amp;f=false\" rel=\"nofollow ugc noopener\">said</a> in 1921, &quot;All grammars leak.&quot; But in a probabilistic model there is\nno difficulty; we can say that <em>quake</em> has a high probability of\nbeing used intransitively, and a low probability of transitive use\n(and we can, if we care, further describe those uses through\nsubcategorization).</p>\n<p>Steve Abney <a href=\"http://www.vinartus.net/spa/95c.pdf\" rel=\"nofollow ugc noopener\">points\nout</a> that probabilistic models are better suited for modeling\nlanguage change. He cites the example of a 15th century Englishman\nwho goes to the pub every day and orders &quot;Ale!&quot; Under a categorical model,\nyou could reasonably expect that one day he would be served eel, because the <a href=\"http://en.wikipedia.org/wiki/Great_Vowel_Shift\" rel=\"nofollow ugc noopener\">great vowel shift</a>\nflipped a Boolean parameter in his mind a day before it flipped the\nparameter in the publican&#39;s. In a probabilistic framework, there\nwill be multiple parameters, perhaps with continuous values, and it\nis easy to see how the shift can take place gradually over two centuries.</p>\n<p>Thus it seems that grammaticality is not a categorical,\ndeterministic judgment but rather an inherently probabilistic\none. This becomes clear to anyone who spends time making <em>observations</em>\nof a corpus of actual sentences, but can remain unknown to those who\nthink that the object of study is their own set of <em>intuitions</em> about\ngrammaticality. Both observation and intuition have been used in the\nhistory of science, so neither is &quot;novel,&quot; but it is observation, not\nintuition that is the dominant model for science.</p>\n<p>Now let&#39;s consider what I think is Chomsky&#39;s main point of\ndisagreement with statistical models:\nthe tension between &quot;accurate description&quot; and &quot;insight.&quot; This is an old\ndistinction. Charles Darwin (biologist, 1809–1882) is best known for his\ninsightful theories but he stressed the importance of accurate\ndescription, saying &quot;False facts are highly injurious to the progress\nof science, for they often endure long; but false views, if supported\nby some evidence, do little harm, for every one takes a salutary\npleasure in proving their falseness.&quot; More recently, Richard Feynman\n(physicist, 1918–1988) wrote &quot;Physics can progress without the\nproofs, but we can&#39;t go on without the facts.&quot;</p>\n<p>Butterflies</p>\n<p>On the other side, Ernest Rutherford (physicist, 1871–1937)\ndisdained mere description, saying &quot;All science is either physics or\nstamp collecting.&quot; Chomsky stands with him: &quot;You can also collect\nbutterflies and make many observations. If you like butterflies,\nthat&#39;s fine; but such work must not be confounded with research, which\nis concerned to discover explanatory principles.&quot;</p>\n<p>Acknowledging both sides is Robert Millikan\n(physicist, 1868–1953) who said in his Nobel acceptance speech\n&quot;Science walks forward on two feet, namely theory and experiment\n... Sometimes it is one foot that is put forward first, sometimes the\nother, but continuous progress is only made by the use of both.&quot;</p>\n<h2 id=\"the-two-cultures\">The two cultures</h2>\n<p>Leo Breiman</p>\n<p>After all those distinguished scientists have weighed in, I think\nthe most relevant contribution to the current discussion is the 2001 paper by\nLeo Breiman (statistician, 1928–2005), <a href=\"http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.156.4933&amp;rep=rep1&amp;type=pdf\" rel=\"nofollow ugc noopener\">Statistical\nModeling: The Two Cultures</a>. In this paper Breiman, alluding to\nC. P. Snow, describes two cultures:</p>\n<p>First the <strong>data modeling culture</strong> (to which, Breiman\nestimates, 98% of statisticians subscribe) holds that nature can be\ndescribed as a black box that has a relatively simple underlying model\nwhich maps from input variables to output variables (with perhaps some\nrandom noise thrown in). It is the job of the statistician to wisely\nchoose an underlying model that reflects the reality of nature, and\nthen use statistical data to estimate the parameters of the model.</p>\n<p>Second the <strong>algorithmic modeling culture</strong> (subscribed to by 2%\nof statisticians and many researchers in biology, artificial\nintelligence, and other fields that deal with complex phenomena),\nwhich holds that nature&#39;s black box cannot necessarily be described by a simple\nmodel. Complex algorithmic approaches (such as support vector\nmachines or boosted decision trees or deep belief networks) are used to estimate the\nfunction that maps from input to output variables, but we have no\nexpectation that the <em>form</em> of the function that emerges from\nthis complex algorithm reflects the true\nunderlying nature.</p>\n<p>It seems that the algorithmic modeling culture is what Chomsky is\nobjecting to most vigorously. It is not just that the models are\nstatistical (or probabilistic), it is that they produce a form that,\nwhile accurately modeling reality, is not easily interpretable by\nhumans, and makes no claim to correspond to the generative process\nused by nature. In other words, algorithmic modeling describes what\n<em>does</em> happen, but it doesn&#39;t answer the question of <em>why</em>.</p>\n<p>Breiman&#39;s article explains his objections to the first\nculture, data modeling. Basically, the conclusions made by data\nmodeling are about the model, not about nature. (Aside: I remember in\n2000 hearing <a href=\"http://www.nasa.gov/home/hqnews/2002/02-072.txt\" rel=\"nofollow ugc noopener\">James\nMartin</a>, the leader of the Viking missions to Mars, saying that his\njob as a spacecraft engineer was not to land on Mars, but to land on\nthe model of Mars provided by the geologists.) The problem is, if the\nmodel does not emulate nature well, then the conclusions may be\nwrong. For example, linear regression is one of the most powerful\ntools in the statistician&#39;s toolbox. Therefore, many analyses start\nout with &quot;Assume the data are generated by a linear model...&quot; and lack\nsufficient analysis of what happens if the data are not in fact\ngenerated that way. In addition, for complex problems there are\nusually many alternative good models, each with very similar measures of goodness\nof fit. How is the data modeler to choose between them? Something has\nto give. Breiman is inviting us to give up on the idea that we can\nuniquely model the true underlying <em>form</em> of nature&#39;s function\nfrom inputs to outputs. Instead he asks us to be satisfied with a\nfunction that accounts for the observed data well, and generalizes to\nnew, previously unseen data well, but may be expressed in a complex\nmathematical form that may bear no relation to the &quot;true&quot; function&#39;s\nform (if such a true function even exists). Chomsky takes the opposite\napproach: he prefers to keep a simple, elegant model, and give up on\nthe idea that the model will represent the data well. Instead, he\ndeclares that what he calls <em>performance</em> data—what people\nactually do—is off limits to linguistics; what really matters is\n<em>competence</em>—what he imagines that they should do.</p>\n<p>Bill O&#39;Reilly</p>\n<p>Laplace</p>\n<p>In January of 2011, television personality Bill O&#39;Reilly weighed in\non more than one culture war with his statement &quot;<em><a href=\"https://www.youtube.com/watch?v=wb3AFMe2OQY\" rel=\"nofollow ugc noopener\">tide goes in, tide\ngoes out. Never a miscommunication. You can&#39;t explain that</a></em>,&quot;\nwhich he proposed as an argument for the existence of God. O&#39;Reilly\nwas ridiculed by his detractors for not knowing that tides can be\nreadily and concisely explained by a system of partial differential equations\ndescribing the gravitational interaction of sun, earth, and moon (a\nfact that was first <a href=\"http://en.wikipedia.org/wiki/Laplace%27s_tidal_equations#Laplace.27s_tidal_equations\" rel=\"nofollow ugc noopener\">worked out</a> by Laplace in 1776 and has been\nconsiderably refined since; when asked by Napoleon why the creator did\nnot enter into his calculations, Laplace said &quot;I had no need of that\nhypothesis.&quot;). (O&#39;Reilly also seems not to know about Deimos and\nPhobos (two of my favorite moons in the entire solar system, along with\nEuropa, Io, and Titan), nor that Mars and Venus orbit the sun, nor that the reason Venus has no\nmoons is because it is so close to the sun that there is scant room for a\nstable lunar orbit.)</p>\n<p>But O&#39;Reilly realizes that it doesn&#39;t matter what\nhis detractors think of his astronomical ignorance, because his\nsupporters think he has gotten exactly to the key issue: <em>why?</em>\nHe doesn&#39;t care <em>how</em> the tides work, tell him <em>why</em> they\nwork. <em>Why</em> is the moon at the right distance to provide a gentle tide,\nand exert a stabilizing effect on earth&#39;s axis of rotation, thus\nprotecting life here? <em>Why</em> does gravity work the way it does? <em>Why</em> does\nanything at all exist rather than not exist? O&#39;Reilly is correct that\nthese <em>why</em> questions can only be addressed by mythmaking, religion or\nphilosophy, not by science.</p>\n<p>Chomsky has a philosophy based on the\nidea that we should focus on the deep <em>whys</em> and that mere\nexplanations of reality don&#39;t matter. In this, Chomsky is in complete\nagreement with O&#39;Reilly. (I recognize that the previous sentence would\nhave an extremely low probability in a probabilistic model trained\non a newspaper or TV corpus.)\nChomsky believes a theory of language should be simple and understandable,\nlike a linear regression model where we know the underlying process is\na straight line, and all we have to do is estimate the slope and\nintercept.</p>\n<p>For example, consider the notion of a <a href=\"http://en.wikipedia.org/wiki/Pro-drop_language\" rel=\"nofollow ugc noopener\">pro-drop\nlanguage</a> from Chomsky&#39;s <a href=\"http://books.google.com/books?id=l08tpkOOdNQC&amp;printsec=frontcover&amp;source=gbs_ge_summary_r&amp;cad=0#v=onepage&amp;q=pro-drop&amp;f=false\" rel=\"nofollow ugc noopener\">Lectures\non Government and Binding</a> (1981). In English we say, for example,\n&quot;I&#39;m hungry,&quot; expressing the pronoun &quot;I&quot;. But in Spanish, one\nexpresses the same\nthought with\n&quot;Tengo hambre&quot; (literally &quot;have hunger&quot;), dropping the\npronoun &quot;Yo&quot;. Chomsky&#39;s theory is that there is a &quot;pro-drop parameter&quot;\nwhich is &quot;true&quot; in Spanish and &quot;false&quot; in English, and that once we\ndiscover the small set of parameters that describe all languages, and\nthe values of those parameters for each language, we will have\nachieved true understanding.</p>\n<p>Dana Carvey</p>\n<p>The problem is that reality is messier than this theory. Here are\nsome dropped pronouns in English:</p>\n<ul><li><p>&quot;Not gonna do it. Wouldn&#39;t be prudent.&quot; (Dana Carvey, [impersonating</p><p>George H. W. Bush](<a href=\"http://snltranscripts.jt.org/94/94dmono.phtml\" rel=\"nofollow ugc noopener\">http://snltranscripts.jt.org/94/94dmono.phtml</a>))* &quot;Thinks he can outsmart us, does he?&quot; (Evelyn Waugh, <a href=\"http://brothersjudd.com/index.cfm/fuseaction/reviews.detail/book_id/891\" rel=\"nofollow ugc noopener\">The Loved One</a>)* &quot;Likes to fight, does he?&quot; (S.M. Stirling, <a href=\"http://books.google.com/books?id=zOLTi9LRG78C&amp;printsec=frontcover&amp;dq=the+sunrise+lands&amp;hl=en&amp;ei=PlDbTfTPBpL0swOR_4W_Dg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CCoQ6AEwAA#v=onepage&amp;q=likes%20to%20fight%20does%20he&amp;f=false\" rel=\"nofollow ugc noopener\">The\n    Sunrise Lands</a>)* &quot;Thinks he&#39;s all that.&quot; (Kate Brian, <a href=\"http://books.google.com/books?id=ERoDoSk7FN4C&amp;pg=PT190&amp;dq=%22thinks+he&#39;s+all+that%22&amp;hl=en&amp;ei=djzWTaKDK4zGsAPLxYWxBw&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=1&amp;ved=0CCkQ6AEwADgK#v=onepage&amp;q=%22thinks%20he&#39;s%20all%20that%22&amp;f=false\" rel=\"nofollow ugc noopener\">Lucky\n      T</a>)* &quot;Go for a walk?&quot; (countless dog owners)* &quot;Gotcha!&quot; &quot;Found it!&quot; &quot;Looks good to me!&quot; (common expressions)</p></li></ul>\n<p>Linguists can argue over the interpretation of these facts for hours\non end, but the diversity of language seems to be much more complex\nthan a single Boolean value for a pro-drop parameter. We shouldn&#39;t\naccept a theoretical framework that places a priority on making the\nmodel simple over making it accurately reflect reality.</p>\n<p>From the beginning, Chomsky has focused on the <em>generative</em>\nside of language. From this side, it is reasonable to tell a\nnon-probabilistic story: I <em>know</em> definitively the idea I want to\nexpress—I&#39;m starting from a single semantic form—thus all I\nhave to do is choose the words to say it; why can&#39;t that be a\ndeterministic, categorical process? If Chomsky had focused on the\nother side, <em>interpretation</em>, as Claude Shannon did, he may have changed\nhis tune. In interpretation (such as speech recognition) the listener\nreceives a noisy, ambiguous signal and needs to decide which of many\npossible intended messages is most likely. Thus, it is obvious that\nthis is inherently a probabilistic problem, as was recognized early on\nby all researchers in speech recognition, and by scientists in other\nfields that do interpretation: the astronomer Laplace said in 1819\n&quot;Probability theory is nothing more than common sense reduced to\ncalculation,&quot; and the physicist James Maxwell said in 1850 &quot;The true\nlogic for this world is the calculus of Probabilities, which takes\naccount of the magnitude of the probability which is, or ought to be,\nin a reasonable man&#39;s mind.&quot;</p>\n<p>Finally, one more reason why Chomsky dislikes statistical models is\nthat they tend to make linguistics an empirical science (a science\nabout how people actually use language) rather than a mathematical\nscience (an investigation of the mathematical properties of <em>models</em> of\nformal language, not of language itself). Chomsky prefers the later, as evidenced by his statement\nin <a href=\"http://books.google.com/books?id=u0ksbFqagU8C&amp;lpg=PP1&amp;dq=aspects%20of%20the%20theory%20of%20syntax&amp;pg=PA4#v=snippet&amp;q=%22observed%20use%20of%20language%22&amp;f=false\" rel=\"nofollow ugc noopener\">*Aspects of the Theory of\nSyntax*</a> (1965):</p>\n<blockquote><p>Linguistic theory is mentalistic, since it is concerned with\ndiscovering a mental reality underlying actual behavior. Observed use of language ... may\nprovide evidence ... but surely cannot constitute the subject-matter\nof linguistics, if this is to be a serious discipline.</p></blockquote>\n<p>I can&#39;t imagine Laplace saying that observations of the planets\ncannot constitute the subject-matter of orbital mechanics, or Maxwell\nsaying that observations of electrical charge cannot constitute the\nsubject-matter of electromagnetism. It is true that physics considers\nidealizations that are abstractions from the messy real world. For\nexample, a class of mechanics problems ignores friction. But that\ndoesn&#39;t mean that friction is not considered part of the\nsubject-matter of physics.</p>\n<p>Plato&#39;s cave</p>\n<p>So how could Chomsky say that observations of language cannot be\nthe subject-matter of linguistics? It seems to come from his viewpoint\nas a <a href=\"http://en.wikipedia.org/wiki/Platonist\" rel=\"nofollow ugc noopener\">Platonist</a>\nand a <a href=\"http://en.wikipedia.org/wiki/Rationalism\" rel=\"nofollow ugc noopener\">Rationalist</a>\nand perhaps a bit of a <a href=\"http://en.wikipedia.org/wiki/Mysticism\" rel=\"nofollow ugc noopener\">Mystic</a>. As\nin Plato&#39;s <a href=\"http://en.wikipedia.org/wiki/Allegory_of_the_Cave\" rel=\"nofollow ugc noopener\">allegory of\nthe cave</a>, Chomsky thinks we should focus on the ideal, abstract\nforms that underlie language, not on the superficial manifestations of\nlanguage that happen to be perceivable in the real world. That is why\nhe is not interested in language performance. But Chomsky, like Plato,\nhas to answer where these ideal forms come from. Chomsky (1991) shows\nthat he is happy with a Mystical answer, although he shifts vocabulary\nfrom &quot;soul&quot; to &quot;biological endowment.&quot;</p>\n<blockquote><p>Plato&#39;s answer was that the knowledge is &#39;remembered&#39;\nfrom an earlier existence. The answer calls for a mechanism: perhaps\nthe immortal soul ... rephrasing Plato&#39;s answer in terms more\ncongenial to us today, we will say that the basic properties of\ncognitive systems are innate to the mind, part of human biological\nendowment.</p></blockquote>\n<p>Lascaux Horse</p>\n<p>It was reasonable for Plato to think that the ideal of, say, a\nhorse, was more important than any individual horse we can perceive in\nthe world. In 400BC, species were thought to be eternal and\nunchanging. We now know that is not true; that the horses on another\ncave wall—in Lascaux—are now extinct, and that current\nhorses continue to evolve slowly over time. Thus there is no such\nthing as a single ideal eternal &quot;horse&quot; form.</p>\n<p>We also now know that language is like that as well: languages are\ncomplex, random, contingent biological processes that are subject to\nthe whims of evolution and cultural change. What constitutes a\nlanguage is not an eternal ideal form, represented by the settings of\na small number of parameters, but rather is the contingent outcome of\ncomplex processes. Since they are contingent, it seems they can\nonly be analyzed with probabilistic models. Since people have to\ncontinually understand the uncertain, ambiguous, noisy speech of\nothers, it seems they must be using something like probabilistic reasoning. Chomsky for some reason\nwants to avoid this, and therefore he must declare the actual facts of\nlanguage use out of bounds and declare that true linguistics only\nexists in the mathematical realm,\nwhere he can impose the formalism he wants. Then, to get\nlanguage from this abstract, eternal, mathematical realm into the\nheads of people, he must fabricate a mystical facility that is exactly\ntuned to the eternal realm. This may be very\ninteresting from a mathematical point of view, but it misses the point\nabout what language is, and how it works.</p>\n<h2 id=\"thanks\">Thanks</h2>\n<p>Thanks to Ann Farmer, Fernando Pereira, Dan Jurafsky, Hal Varian, and others for comments\nand suggestions on this essay.</p>\n<h2 id=\"annotated-bibliography\">Annotated Bibliography</h2>\n<ol><li><p>Abney, Steve (1996) [Statistical Methods and</p><p> Linguistics](<a href=\"http://www.vinartus.net/spa/95c.pdf\" rel=\"nofollow ugc noopener\">http://www.vinartus.net/spa/95c.pdf</a>), in Klavans and Resnik (eds.) *The Balancing Act: Combining Symbolic and\n Statistical Approaches to Language*, MIT Press.</p>\n<blockquote><p>*An excellent overall introduction to the statistical\napproach to language processing, and covers some ground that is not\naddressed often, such as language change and individual differences.*</p></blockquote>\n<ul><li><p>Breiman, Leo (2001) [Statistical Modeling: The</p><p>Two Cultures](<a href=\"http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.ss/1009213726\" rel=\"nofollow ugc noopener\">http://projecteuclid.org/DPubS?service=UI&amp;version=1.0&amp;verb=Display&amp;handle=euclid.ss/1009213726</a>), <em>Statistical Science</em>, Vol. 16, No. 3, 199-231.</p>\n<blockquote><p>*Breiman does a great job of describing the two\napproaches, explaining the benefits of his approach, and defending\nhis points in the vary interesting commentary with eminent\nstatisticians: Cox, Efron, Hoadley, and Parzen.*</p></blockquote>\n<ul><li><p>Chomsky, Noam (1956) [Three Models</p><p>for the Description of Language](<a href=\"http://www.chomsky.info/articles/195609--.pdf\" rel=\"nofollow ugc noopener\">http://www.chomsky.info/articles/195609--.pdf</a>), *IRE Transactions on\nInformation theory* (2), pp. 113-124.</p>\n<blockquote><p>*Compares finite state, phrase structure, and\ntransformational grammars. Introduces &quot;colorless green ideas sleep\nfuriously.&quot;*</p></blockquote>\n<ul><li><p>Chomsky, Noam (1967) <a href=\"http://books.google.com/books?id=SNeHkMXHcd8C&amp;printsec=frontcover&amp;dq=syntactic+structures+chomsky&amp;hl=en&amp;src=bmrr&amp;ei=1WvcTa3UDeLQiAK6-4zpDw&amp;sa=X&amp;oi=book_result&amp;ct=book-thumbnail&amp;resnum=1&amp;ved=0CD4Q6wEwAA#v=onepage&amp;q=probabilistic&amp;f=false\" rel=\"nofollow ugc noopener\">Syntactic Structures</a>, Mouton.</p><blockquote><p>*A book-length exposition of Chomsky&#39;s theory that was\nthe leading exposition of linguistics for a decade. Claims that\nprobabilistic models give no insight into syntax.*</p></blockquote>\n<ul><li><p>Chomsky, Noam (1969) [Some Empirical</p><p>Assumptions in Modern Philosophy of Language](<a href=\"http://books.google.com/books?id=iaXVXYDQN1oC&amp;pg=PA296&amp;dq=%22probability+of+a+sentence%22+%22entirely+useless%22&amp;hl=en&amp;ei=_s7eTc_5DrTciALcsvnlCg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=4&amp;ved=0CDoQ6AEwAw#v=onepage&amp;q=%22probability%20of%20a%20sentence%22%20%22entirely%20useless%22&amp;f=false\" rel=\"nofollow ugc noopener\">http://books.google.com/books?id=iaXVXYDQN1oC&amp;pg=PA296&amp;dq=%22probability+of+a+sentence%22+%22entirely+useless%22&amp;hl=en&amp;ei=_s7eTc_5DrTciALcsvnlCg&amp;sa=X&amp;oi=book_result&amp;ct=result&amp;resnum=4&amp;ved=0CDoQ6AEwAw#v=onepage&amp;q=%22probability%20of%20a%20sentence%22%20%22entirely%20useless%22&amp;f=false</a>), in *Philosophy,\nScience and Method: Essays in Honor or Ernest Nagel*,\nSt. Martin&#39;s Press.</p>\n<blockquote><p>*Claims that the notion &quot;probability of a sentence&quot; is\nan entirely useless notion.*</p></blockquote>\n<ul><li><p>Chomsky, Noam (1981) [Lectures</p><p>on government and binding](<a href=\"http://books.google.com/books?id=l08tpkOOdNQC&amp;printsec=frontcover&amp;source=gbs_ge_summary_r&amp;cad=0#v=onepage&amp;q=pro-drop&amp;f=false\" rel=\"nofollow ugc noopener\">http://books.google.com/books?id=l08tpkOOdNQC&amp;printsec=frontcover&amp;source=gbs_ge_summary_r&amp;cad=0#v=onepage&amp;q=pro-drop&amp;f=false</a>), de Gruyer.</p>\n<blockquote><p>*A revision of Chomsky&#39;s theory; this version\nintroduces Universal Grammar. We cite it for the coverage of\nparameters such as pro-drop.*</p></blockquote>\n<ul><li><p>Chomsky, Noam (1991) [Linguistics</p><p>and adjacent fields: a personal view](<a href=\"http://www.cpgb.org.uk/article.php?article_id=1004261#23\" rel=\"nofollow ugc noopener\">http://www.cpgb.org.uk/article.php?article_id=1004261#23</a>), in Kasher (ed.), *A\nChomskyan Turn*, Oxford.</p>\n<blockquote><p>*I found the Plato quotes in <a href=\"http://www.cpgb.org.uk/article.php?article_id=1004261#23\" rel=\"nofollow ugc noopener\">this</a>\narticle, published by the Communist Party of Great Britain, and\napparently published by someone with no linguistics training\nwhatsoever, but with a political agenda.*</p></blockquote>\n<ul><li><p>Gold, E. M. (1967) [Language</p><p>Identification in the Limit](<a href=\"http://en.wikipedia.org/wiki/Language_identification_in_the_limit\" rel=\"nofollow ugc noopener\">http://en.wikipedia.org/wiki/Language_identification_in_the_limit</a>), <em>Information and Control</em>,\nVol. 10, No. 5, pp. 447-474.</p>\n<blockquote><p>*Gold proved a result in formal language theory that\nwe can state (with some artistic license) as this: imagine a game\nbetween two players, guesser and chooser. Chooser says to\nguesser, &quot;Here is an infinite number of languages. I&#39;m going to\nchoose one of them, and start reading sentences to you that come\nfrom that language. On your N-th birthday there will be a True-False\nquiz where I give you 100 sentences you haven&#39;t heard yet, and you\nhave to say whether they come from the language or not.&quot; There are\nsome limits on what the infinite set looks like and on how\nthe chooser can pick sentences (he can be deliberately\ntricky, but he can&#39;t just repeat the same sentence over and over,\nfor example). Gold&#39;s\nresult is that if the infinite set of languages are all generated by\ncontext-free grammars then there is no strategy for guesser that\nguarantees she gets 100% correct every time, no matter what N you\nchoose for the birthday. This result was\ntaken by Chomsky and others to mean that it is impossible for\nchildren to learn human languages without having an innate &quot;language\norgan.&quot; As <a href=\"http://psyling.psy.cmu.edu/papers/years/2004/logical/gold-johnson.pdf\" rel=\"nofollow ugc noopener\">Johnson (2004)</a> and others show, this was an invalid\nconclusion; the task of getting 100% on the quiz (which Gold called\n<em>language identification</em>)\nreally has nothing in common with the task of <em>language acquisition</em>\nperformed by children, so Gold&#39;s Theorem has no\nrelevance.*</p></blockquote>\n<ul><li><p>Horning, J. J. (1969) [A study of grammatical</p><p>inference](<a href=\"http://portal.acm.org/citation.cfm?id=905718&amp;coll=DL&amp;dl=GUIDE&amp;CFID=23829333&amp;CFTOKEN=19917759\" rel=\"nofollow ugc noopener\">http://portal.acm.org/citation.cfm?id=905718&amp;coll=DL&amp;dl=GUIDE&amp;CFID=23829333&amp;CFTOKEN=19917759</a>), Ph.D. thesis, Stanford Univ.</p>\n<blockquote><p>*Where Gold found a negative result—that\ncontext-free languages were not identifiable from examples, Horning found a positive result—that probabilistic\ncontext-free languages are identifiable (to within an arbitrarily\nsmall level of error). Nobody doubts that\nhumans have unique innate capabilities for understanding language\n(although it is unknown to what extent these capabilities are\nspecific to language and to what extent they are general cognitive\nabilities related to sequencing and forming abstractions). But\nHorning proved in 1969 that Gold cannot be used as a convincing\nargument for an innate language organ that specifies all of language\nexcept for the setting of a few parameters.*</p></blockquote>\n<ul><li><p>Johnson, Kent (2004) [Gold&#39;s</p><p>Theorem and cognitive science](<a href=\"http://psyling.psy.cmu.edu/papers/years/2004/logical/gold-johnson.pdf\" rel=\"nofollow ugc noopener\">http://psyling.psy.cmu.edu/papers/years/2004/logical/gold-johnson.pdf</a>), <em>Philosophy of Science</em>,\nVol. 71, pp. 571-592.</p>\n<blockquote><p>*The best article I&#39;ve seen on what Gold&#39;s Theorem\nactually says and what has been claimed about it (correctly and\nincorrectly). Concludes that Gold has something to say about formal\nlanguages, but nothing about child language\nacquisition.*</p></blockquote>\n<ul><li><p>Lappin, Shalom and Shieber, Stuart M. (2007) [Machine</p><p>learning theory and practice as a source of insight into universal\ngrammar.](<a href=\"http://www.dcs.kcl.ac.uk/staff/lappin/papers/lappin-shieber_learning07.pdf\" rel=\"nofollow ugc noopener\">http://www.dcs.kcl.ac.uk/staff/lappin/papers/lappin-shieber_learning07.pdf</a>), <em>Journal of Linguistics</em>, Vol. 43, No. 2,\npp. 393-427.</p>\n<blockquote><p>*An excellent article discussing the poverty of the\nstimulus, the fact that all models have bias, the difference\nbetween supervised and unsupervised learning, and modern (PAC or VC)\nlearning theory. It provides\nalternatives to the model of Universal Grammar consisting of a fixed\nset of binary parameters.*</p></blockquote>\n<ul><li><p>Manning, Christopher (2002) [Probabilistic</p><p>Syntax](<a href=\"http://nlp.stanford.edu/~manning/papers/probsyntax.pdf\" rel=\"nofollow ugc noopener\">http://nlp.stanford.edu/~manning/papers/probsyntax.pdf</a>), in Bod, Hay, and Jannedy (eds.), *Probabilistic\nLinguistics*, MIT Press.</p>\n<blockquote><p>*A compelling introduction to probabilistic syntax,\nand how it is a better model for linguistic facts than categorical\nsyntax. Covers &quot;the joys and perils of corpus\nlinguistics.&quot;*</p></blockquote>\n<ul><li><p>Norvig, Peter (2007) [How to Write a Spelling</p><p>Corrector](<a href=\"http://norvig.com/spell-correct.html\" rel=\"nofollow ugc noopener\">http://norvig.com/spell-correct.html</a>), unpublished web page.</p>\n<blockquote><p>*Shows working code to implement a probabilistic,\nstatistical spelling correction algorithm.*</p></blockquote>\n<ul><li><p>Norvig, Peter (2009) [Natural</p><p>Language Corpus Data](<a href=\"http://norvig.com/ngrams/\" rel=\"nofollow ugc noopener\">http://norvig.com/ngrams/</a>), in Seagran and Hammerbacher (eds.),\n<em>Beautiful Data</em>, O&#39;Reilly.</p>\n<blockquote><p>*Expands on the essay above; shows how to implement\nthree tasks: text segmentation, cryptographic decoding, and spelling\ncorrection (in a slightly more complete form than the previous essay).*</p></blockquote>\n<ul><li><p>Pereira, Fernando (2002) [Formal</p><p>grammar and information theory: together again?](<a href=\"http://www.cis.upenn.edu/~pereira/papers/rsoc.pdf\" rel=\"nofollow ugc noopener\">http://www.cis.upenn.edu/~pereira/papers/rsoc.pdf</a>), in Nevin and\nJohnson (eds.), <em>The Legacy of Zellig Harris</em>, Benjamins.</p>\n<blockquote><p>*When I set out to write the page you are reading now,\nI was concentrating on the events that took place in Cambridge,\nMass., 4800 km from home. After doing some research I was surprised\nto learn that the authors of\ntwo of the three best articles on this subject sit within a total of\n10 meters from my desk: Fernando Pereira and Chris Manning. (The\nthird, Steve Abney, sits 3700 km away.) But perhaps I shouldn&#39;t have\nbeen surprised. I remember giving a talk at ACL on the corpus-based\nlanguage models used at Google, and having Fernando, then a\nprofessor at U. Penn., comment &quot;I feel like I&#39;m a particle physicist\nand you&#39;ve got the only super-collider.&quot; A few years later he moved\nto Google. Fernando is also famous for his quote &quot;The older I get, the\nfurther down the Chomsky Hierarchy I go.&quot; His article here covers\nsome of the same ground as mine, but he goes farther in explaining\nthe range of probabilistic models available and how they are useful.*</p></blockquote>\n<ul><li><p>Plato (c. 380BC) [The</p><p>Republic](<a href=\"http://en.wikipedia.org/wiki/Allegory_of_the_Cave\" rel=\"nofollow ugc noopener\">http://en.wikipedia.org/wiki/Allegory_of_the_Cave</a>).</p>\n<blockquote><p><em>Cited here for the allegory of the cave.</em></p></blockquote>\n<ul><li><p>Shannon, C.E. (1948) [A</p><p>Mathematical Theory of Communication](<a href=\"http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf\" rel=\"nofollow ugc noopener\">http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf</a>), *The Bell System\nTechnical Journal*, Vol. 27, pp. 379-423.</p>\n<blockquote><p>*An enormously influential article that started the\nfield of information theory and introduced the term &quot;bit&quot; and the noisy\nchannel model, demonstrated successive n-gram approximations of\nEnglish, described Markov models of language, defined entropy with\nrespect to these models, and enabled the growth of the\ntelecommunications industry.*</p></blockquote></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ul></li></ol>\n<hr />\n<p><a href=\"http://norvig.com\" rel=\"nofollow ugc noopener\"><em>Peter Norvig</em></a></p>","headings":[{"level":1,"text":"On Chomsky and the Two Cultures of Statistical Learning","id":"on-chomsky-and-the-two-cultures-of-statistical-learning"},{"level":2,"text":"What did Chomsky mean, and is he right?","id":"what-did-chomsky-mean-and-is-he-right"},{"level":2,"text":"What is a statistical model?","id":"what-is-a-statistical-model"},{"level":2,"text":"How successful are statistical language models?","id":"how-successful-are-statistical-language-models"},{"level":2,"text":"Is there anything like it [the statistical notion of success] in the history of science?","id":"is-there-anything-like-it-the-statistical-notion-of-success-in-t"},{"level":2,"text":"What doesn't Chomsky like about statistical models?","id":"what-doesn-t-chomsky-like-about-statistical-models"},{"level":2,"text":"The two cultures","id":"the-two-cultures"},{"level":2,"text":"Thanks","id":"thanks"},{"level":2,"text":"Annotated Bibliography","id":"annotated-bibliography"}]}}