Introduction to statistical signal processing

460 Pages·2000·2.25 MB·english

by  Gray R.M.

An Introduction to
Statistical Signal Processing
f¡1(F)
f
F
Pr(f 2F)=P(f! :! 2Fg)=P(f¡1(F))
-
August 24, 2002
ii
An Introduction to
Statistical Signal Processing
Robert M. Gray
and
Lee D. Davisson
Information Systems Laboratory
Department of Electrical Engineering
Stanford University
and
Department of Electrical Engineering and Computer Science
University of Maryland
iv
(cid:176)c1999{2002 by the authors.
v
to our Families
vi
Contents
Preface xi
Glossary xv
1 Introduction 1
2 Probability 11
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Spinning Pointers and Flipping Coins . . . . . . . . . . . . 15
2.3 Probability Spaces . . . . . . . . . . . . . . . . . . . . . . . 23
2.3.1 Sample Spaces . . . . . . . . . . . . . . . . . . . . . 27
2.3.2 Event Spaces . . . . . . . . . . . . . . . . . . . . . . 31
2.3.3 Probability Measures. . . . . . . . . . . . . . . . . . 42
2.4 Discrete Probability Spaces . . . . . . . . . . . . . . . . . . 45
2.5 Continuous Probability Spaces . . . . . . . . . . . . . . . . 55
2.6 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . 69
2.7 Elementary Conditional Probability . . . . . . . . . . . . . 70
2.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3 Random Objects 85
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
3.1.1 Random Variables . . . . . . . . . . . . . . . . . . . 85
3.1.2 Random Vectors . . . . . . . . . . . . . . . . . . . . 89
3.1.3 Random Processes . . . . . . . . . . . . . . . . . . . 93
3.2 Random Variables . . . . . . . . . . . . . . . . . . . . . . . 95
3.3 Distributions of Random Variables . . . . . . . . . . . . . . 104
3.3.1 Distributions . . . . . . . . . . . . . . . . . . . . . . 104
3.3.2 Mixture Distributions . . . . . . . . . . . . . . . . . 108
3.3.3 Derived Distributions . . . . . . . . . . . . . . . . . 111
3.4 Random Vectors and Random Processes . . . . . . . . . . . 115
3.5 Distributions of Random Vectors . . . . . . . . . . . . . . . 117
vii
viii CONTENTS
3.5.1 ?Multidimensional Events . . . . . . . . . . . . . . . 118
3.5.2 Multidimensional Probability Functions . . . . . . . 119
3.5.3 Consistency of Joint and Marginal Distributions . . 120
3.6 Independent Random Variables . . . . . . . . . . . . . . . . 127
3.6.1 IID Random Vectors . . . . . . . . . . . . . . . . . . 128
3.7 Conditional Distributions . . . . . . . . . . . . . . . . . . . 129
3.7.1 Discrete Conditional Distributions . . . . . . . . . . 129
3.7.2 Continuous Conditional Distributions . . . . . . . . 131
3.8 Statistical Detection and Classiflcation . . . . . . . . . . . . 134
3.9 Additive Noise . . . . . . . . . . . . . . . . . . . . . . . . . 137
3.10 Binary Detection in Gaussian Noise . . . . . . . . . . . . . 144
3.11 Statistical Estimation . . . . . . . . . . . . . . . . . . . . . 146
3.12 Characteristic Functions . . . . . . . . . . . . . . . . . . . . 147
3.13 Gaussian Random Vectors . . . . . . . . . . . . . . . . . . . 152
3.14 Examples: Simple Random Processes . . . . . . . . . . . . . 153
3.15 Directly Given Random Processes . . . . . . . . . . . . . . 157
3.15.1 The Kolmogorov Extension Theorem . . . . . . . . . 157
3.15.2 IID Random Processes . . . . . . . . . . . . . . . . . 157
3.15.3 Gaussian Random Processes. . . . . . . . . . . . . . 158
3.16 Discrete Time Markov Processes . . . . . . . . . . . . . . . 159
3.16.1 A Binary Markov Process . . . . . . . . . . . . . . . 159
3.16.2 The Binomial Counting Process. . . . . . . . . . . . 162
3.16.3 ?Discrete Random Walk . . . . . . . . . . . . . . . . 165
3.16.4 The Discrete Time Wiener Process . . . . . . . . . . 165
3.16.5 Hidden Markov Models . . . . . . . . . . . . . . . . 167
3.17 ?Nonelementary Conditional Probability . . . . . . . . . . . 167
3.18 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
4 Expectation and Averages 185
4.1 Averages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
4.2 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
4.2.1 Examples: Expectation . . . . . . . . . . . . . . . . 190
4.3 Functions of Several Random Variables. . . . . . . . . . . . 198
4.4 Properties of Expectation . . . . . . . . . . . . . . . . . . . 198
4.5 Examples: Functions of Random Variables . . . . . . . . . . 201
4.5.1 Correlation . . . . . . . . . . . . . . . . . . . . . . . 201
4.5.2 Covariance . . . . . . . . . . . . . . . . . . . . . . . 203
4.5.3 Covariance Matrices . . . . . . . . . . . . . . . . . . 204
4.5.4 Linear Operations on Gaussian Vectors . . . . . . . 205
4.5.5 Example: Difierential Entropy of a Gaussian Vector 208
4.6 Conditional Expectation . . . . . . . . . . . . . . . . . . . . 209
4.7 ?Jointly Gaussian Vectors . . . . . . . . . . . . . . . . . . . 212
CONTENTS ix
4.8 Expectation as Estimation . . . . . . . . . . . . . . . . . . . 215
4.9 ? Implications for Linear Estimation . . . . . . . . . . . . . 221
4.10 Correlation and Linear Estimation . . . . . . . . . . . . . . 223
4.11 Correlation and Covariance Functions . . . . . . . . . . . . 230
4.12 ?The Central Limit Theorem . . . . . . . . . . . . . . . . . 233
4.13 Sample Averages . . . . . . . . . . . . . . . . . . . . . . . . 236
4.14 Convergence of Random Variables . . . . . . . . . . . . . . 238
4.15 Weak Law of Large Numbers . . . . . . . . . . . . . . . . . 243
4.16 ?Strong Law of Large Numbers . . . . . . . . . . . . . . . . 245
4.17 Stationarity . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
4.18 Asymptotically Uncorrelated Processes . . . . . . . . . . . . 255
4.19 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
5 Second-Order Moments 279
5.1 Linear Filtering of Random Processes . . . . . . . . . . . . 280
5.2 Second-Order Linear Systems I/O Relations . . . . . . . . . 282
5.3 Power Spectral Densities . . . . . . . . . . . . . . . . . . . . 287
5.4 Linearly Filtered Uncorrelated Processes . . . . . . . . . . . 290
5.5 Linear Modulation . . . . . . . . . . . . . . . . . . . . . . . 296
5.6 White Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
5.7 ?Time-Averages . . . . . . . . . . . . . . . . . . . . . . . . . 303
5.8 ?Difierentiating Random Processes . . . . . . . . . . . . . . 307
5.9 ?Linear Estimation and Filtering . . . . . . . . . . . . . . . 310
5.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
6 A Menagerie of Processes 341
6.1 Discrete Time Linear Models . . . . . . . . . . . . . . . . . 342
6.2 Sums of IID Random Variables . . . . . . . . . . . . . . . . 346
6.3 Independent Stationary Increments . . . . . . . . . . . . . . 348
6.4 ?Second-Order Moments of ISI Processes . . . . . . . . . . 351
6.5 Speciflcation of Continuous Time ISI Processes . . . . . . . 353
6.6 Moving-Average and Autoregressive Processes . . . . . . . . 356
6.7 The Discrete Time Gauss-Markov Process . . . . . . . . . . 357
6.8 Gaussian Random Processes . . . . . . . . . . . . . . . . . . 358
6.9 ?The Poisson Counting Process . . . . . . . . . . . . . . . . 359
6.10 Compound Processes . . . . . . . . . . . . . . . . . . . . . . 362
6.11 ?Exponential Modulation . . . . . . . . . . . . . . . . . . . 364
6.12 ?Thermal Noise . . . . . . . . . . . . . . . . . . . . . . . . . 369
6.13 Ergodicity and Strong Laws of Large Numbers . . . . . . . 371
6.14 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
x CONTENTS
A Preliminaries 387
A.1 Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
A.2 Examples of Proofs . . . . . . . . . . . . . . . . . . . . . . . 395
A.3 Mappings and Functions . . . . . . . . . . . . . . . . . . . . 399
A.4 Linear Algebra . . . . . . . . . . . . . . . . . . . . . . . . . 400
A.5 Linear System Fundamentals . . . . . . . . . . . . . . . . . 403
A.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
B Sums and Integrals 415
B.1 Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
B.2 ?Double Sums. . . . . . . . . . . . . . . . . . . . . . . . . . 418
B.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
B.4 ?The Lebesgue Integral . . . . . . . . . . . . . . . . . . . . 421
C Common Univariate Distributions 425
D Supplementary Reading 427
Bibliography 432
Index 436