The Minimum Description Length Principle (Adaptive Computation and Machine Learning)

738 Pages·2007·3.02 MB·English

by  Peter D. Grünwald

the
computer science/statistics t
h
e
The Minimum Description Length Principle
M
Peter D. Grünwald
i
foreword by Jorma Rissanen n
i
m
The minimum description length (MDL) principle is a powerful method of inductive
u
inference, the basis of statistical modeling, pattern recognition, and machine m Minimum
learning. It holds that the best explanation, given a limited set of observed data,
D
is the one that permits the greatest compression of the data. MDL methods are
e
particularly well suited for dealing with model selection, prediction, and estimation s
c
problems in situations where the models under consideration can be arbitrarily
r
i
complex, and overfitting the data is a serious concern. p
This extensive, step-by-step introduction to the MDL principle provides a t
i PETER D. GRÜNWALD
comprehensive reference (with an emphasis on conceptual issues) that is accessible o
n
to graduate students and researchers in statistics, pattern classification, machine
L Description
learning, and data mining, to philosophers interested in the foundations of
e
statistics, and to researchers in other applied sciences that involve model selection, n
including biology, econometrics, and experimental psychology. Part I provides a basic g
t
introduction to MDL and an overview of the concepts in statistics and information h
theory needed to understand MDL. Part II treats universal coding, the information-
p
theoretic notion on which MDL is built, and part III gives a formal treatment of MDL r
i
theory as a theory of inductive inference based on universal coding. Part IV provides n
c
a comprehensive overview of the statistical theory of exponential families with an
i
p
emphasis on their information-theoretic properties. The text includes a number of Length
l
e
summaries, paragraphs offering the reader a “fast track” through the material, and

boxes highlighting the most important concepts.
Peter D. Grünwald is Senior Researcher and Project Leader at CWI, the National
G
Research Institute for Mathematics and Computer Science in Amsterdam, and is R
Ü
affiliated with EURANDOM at Eindhoven University of Technology, the Netherlands. N
W
He is the coeditor of Advances in Minimum Description Length: Theory and A
L
Applications (MIT Press, 2005). D
principle
Adaptive Computation and Machine Learning series
0-262-07281-5
The MIT Press 978-0-262-07281-6
Massachusetts Institute of Technology
Cambridge, Massachusetts 02142
http://mitpress.mit.edu
foreword by Jorma Rissanen

The
Minimum
Description
Length
Principle

The
Minimum
Description
Length
Principle
PeterD.Grünwald
TheMITPress
Cambridge,Massachusetts
London,England
©2007MassachusettsInstituteofTechnology
All rights reserved. No part of this book may be reproducedin any form by any
electronicormechanicalmeans(includingphotocopying,recording,orinformation
storageandretrieval)withoutpermissioninwritingfromthepublisher.
TypesetinPalatinobytheauthorusingLATEX2εwithC.Manning’sfbook.clsand
statnlpbook.stymacros.
PrintedandboundintheUnitedStatesofAmerica.
LibraryofCongressCataloging-in-PublicationInformation
Grünwald,PeterD.
Theminimumdescriptionlengthprinciple/PeterD.Grünwald.
p. cm.—(Adaptivecomputationandmachinelearning)
Includesbibliographicalreferencesandindex.
ISBN-13:978-0-262-07281-6(alk.paper)
1.Minimumdescriptionlength(Informationtheory)I.Title
QA276.9G782007
003’.54—dc22
2006046646
10987654321
Tomyfather

Brief Contents
I IntroductoryMaterial 1
1 Learning,Regularity,andCompression 3
2 ProbabilisticandStatisticalPreliminaries 41
3 Information-TheoreticPreliminaries 79
4 Information-TheoreticPropertiesofStatisticalModels 109
5 CrudeTwo-PartCodeMDL 131
II UniversalCoding 165
6 UniversalCodingwithCountableModels 171
7 ParametricModels:NormalizedMaximumLikelihood 207
8 ParametricModels:Bayes 231
9 ParametricModels:PrequentialPlug-in 257
10 ParametricModels:Two-Part 271
11 NMLWithInfiniteComplexity 295
12 LinearRegression 335
13 BeyondParametrics 369
III RefinedMDL 403
14 MDLModelSelection 409
15 MDLPredictionandEstimation 459
16 MDLConsistencyandConvergence 501
17 MDLinContext 523
viii BriefContents
IV AdditionalBackground 597
18 TheExponentialor“MaximumEntropy”Families 599
19 Information-TheoreticPropertiesofExponentialFamilies 623