0.LM. Data in Language Models - JulTob/Data GitHub Wiki

🧠 Data in Language Models

Statistical Paradigm

Language models

Here, data is the core input.

These models learn patterns, like probabilities of word sequences, directly from large text corpora.

Data defines the model’s behavior.

Claude Shannon’s information theory underpins this tradition.

  • A model might predict “the” is likely after “in”, purely from co-occurrence.
  • No need to posit syntax trees, Universal Grammar, or innate structures.

🧠 Chomsky's Critique of Big Data Models

Chomsky has criticized deep learning based models as:

  • “Superficial curve-fitting”
  • Failing to explain why or how language works
  • Not distinguishing between data driven mimicry and principled understanding.

“There’s a notion of science. Science is about explanation. Deep learning is not science.”
— Noam Chomsky

📘 To Explore Further:

  • Chomsky (1965) – Aspects of the Theory of Syntax
  • Chomsky (2000) – New Horizons in the Study of Language and Mind
  • Chris Manning (Stanford NLP) – critiques and contrasts between symbolic and statistical NLP
  • Shannon (1948) – statistical foundation of language models
  • Floridi – Philosophy of Information (for meta-concept of data and models)
⚠️ **GitHub.com Fallback** ⚠️