<<< Previous 10 Lectures <<< >>> Next 10 Lectures >>>

EE 503 Lectures (Fall 2020/21)
Lec. #11

00:00 - "similarity" of two r.v.'s
00:45 - Inner product as an indicator of "similarity" for N-dim vectors (reminder)
04:14 - Correlation coefficient (definition for zero mean r.v.'s)
08:28 - Properties of correlation coefficient
09:31 - |r_{xy}| \leq 1 (proof)
13:35 - |r_{xy}| = 1 implies Y = aX
16:41 - Y = aX imples |r_{xy}| = 1
20:03 - r_{xy} as angle between two r.v.'s
22:01 - Orthgonal r.v.'s (definition)
23:05 - Example: r_{xy} calculation for Y = aX + N
29:52 - SNR definition (example continues)
32:55 - Interpretation of results (example continues)
37:40 - Correlation (definition)
38:22 - Correlation coefficient (definition)
42:25 - Cov(X,Y) (definition)
45:20 - Uncorrelated r.v.'s
47:35 - Example: Cov(X,Y) for r.v.'s X and Y which are indicator functions events A and B
55:03 - Interpretation of results (example continues)
57:51 - Properties of Cov(X,Y)
1:00:58 - Example: Var ( \sum_{i=1}^N x_i )
1:06:20 - Independence implies uncorrelatedness (proof)
Lec. #12a

A Matlab illustration on the correlation coefficient

.pdf document

Lec. #12b

00:00 - Random vectors
03:04 - Correlation matrix (definition)
03:31 - Covariance matrix (definition)
11:10 - Properties of covariance matrix
12:14 - Hermitian symmetry (properties continue)
15:18 - Positive semi-definiteness (properties continue)
20:47 - Gaussian Distribution
21:16 - 1D Gaussian r.v.
28:10 - N-dimensional Gaussian vectors
33:08 - 2-dimensional Gaussian vectors
40:15 - Level curves (2D Gaussian vectors)
45:15 - Level curves (2D Gaussian vector, Cx : diagonal)
49:10 - Level curves (2D Gaussian vector, Cx \propto I )
50:30 - Facts on Gaussian vectors
50:45 - Marginalization (facts continue)
52:48 - Example on marginalization of Gaussian vectors
55:30 - Linear processing of Gaussian vectors (facts continue)
56:15 - Example: Ry matrix in terms of Rx for y = Mx
1:02:13 - Example: Var ( \sum_{i=1}^N x_i ) (redo earlier example with vector operations)


Corrections:
36:20 - 2D Gaussian case: pdf (2nd line on the left side) should have Cx^{-1} not Cx (Cagatay C.)
Lec. #13

0:00 - Linear processing of random vectors (summary)
3:40 - Decorrelation of random vectors
5:45 - Case 1: Diagonalization by eigendecomposition
19:15 - Case 2: Diagonalization by unitary transformation and scaling
23:35 - Case 3: Diagonalization by LU decomposition
29:29 - Unit lower triangular matrix (definition)
36:27 - Causal decorrelation operation by LU decomposition

Lec. #14

0:00 - Diagonalization of Covariance Matrices (review)
3:50 - Diagonalization by eigendecomposition (review)
06:01 - Diagonalization by unitary Transformation and followed by scaling (review)
08:35 - Diagonalization by LU decomposition (review)
11:40 - Joint Diagonalization of two covariance matrices
15:13 - Joint Diagonalization of two covariance matrices (Step 1)
18:14 - Joint Diagonalization of two covariance matrices (Step 2)
26:35 - Joint Diagonalization of two covariance matrices (with a single step)
40:50 - Random processes

Lec. #15

0:00 - Descriptions for Random Processes
0:35 - Joint pdf description
6:42 - Example: joint pdf description of x(t) = A cos(2\pi f t + \theta), \theta: r.v. ~ Unif. [0,2\pi)
12:10 - 1st order pdf description (example)
13:22 - One function of one r.v. discussion
26:47 - 1st order pdf description (example, continued)
37:12 - 1st order pdf description (comments)
41:51 - 2nd order pdf description (example)
58:51 - 3rd order pdf description (example)

Lec. #16

0:00 - Random Process Descriptions (cont'd)
0:58 - Gaussian Processes
2:20 - Joint pdf of samples (Gaussian processes)
6:00 - Example: Marginal pdf's are Gaussian, while joint pdf is not
14:50 - Example: x(t) = a t+ b (a,b: r.v.'s)
19:13 - 1st order pdf description (example)
27:05 - 2nd order pdf description (example)
37:40 - 3rd order pdf description (example)
42:27 - Discussion of degenerate cov. mat. (example)

Lec. #17

0:15 - Moments Description
2:35 - 1st Order Moment Description: mean function
3:19 - 2nd Order Moment Description: Autocorrelation/Covariance functions
5:08 - Comments: on the Moments Descriptions
9:18 - Example: x(t) a r.p. with mu_x(t)=3 and R_x(t1,t2)=9+4*exp(-0.2|t1-t2|), z=x(5), w=x(8), Find E{z}, E{w}, E{z^2}, E{w^2} and E{zw}
14:50 - Example: z = x(t1) + x(t2), Find E{z^2}
17:12 - Example: s = integral_{from a to b}(x(t)dt), a) Find E{s}, b) Find E{s^2}
21:33 - Example: x(t)=Acos(wt+Q), A,Q are independent r.v.'s, Q~uniform[0,2*pi), Find mu_x(t) and R_x(t1,t2)
34:14 - Notes: about complex valued processes
37:13 - Example: x(t)=A*exp(wt+Q), A,Q are independent r.v.'s, Q~uniform[0,2*pi), Find mu_x(t) and R_x(t1,t2)

Lec. #18

0:00 - White noise process
8:10 - Example: x_1[k] = w and x_2[k] = w_k
14:17 - Joint pdf description (example)
21:15 - Moment description (example)
30:05 - Example: w_1[n] \in {1,-1} iid; w_2[n] ~ N(0,1), iid
33:50 - Joint pdf description (example)
38:30 - Moment description (example)
space
Lec. #19

0:00 - Linear systems with stochastic inputs
0:33 - Linear systems - continuous time (review)
2:16 - LTI systems - continuous time (review)
5:06 - Linear systems - discrete time (review)
7:40 - LTI systems - discrete time (review)
8:58 - LTI systems - discrete time, convolution matrix (review)
11:11 - Moment descriptions for linear systems with stochastic inputs
11:30 - Linear processing output mean function calculation
14:02 - Basic Assumption: Commutation of Expectation operation and Linear operations
16:29 - Linear processing output auto-correlation function calculation
17:01 - Linear processing output auto-correlation function calculation (step 1: cross correlation)
25:10 - Linear processing output auto-correlation function calculation (step 2: auto-correlation)
32:04 - Example: y(t) = L{x(t)} = d/dt x(t), Find output mean and autocorrelation functions
41:00 - Connections with earlier finite dimensional results
47:12 - Brief discussion on LTI and WSS input case
53:15 - Stationary Random Processes
58:03 - Stationarity in joint pdf description
1:02:07 - Stationarity in moment description

Lec. #20

0:00 - Stationarity in pdf/moment descriptions (review)
10:40 - SSS implies WSS
11:48 - Gaussian process and WSS is equivalent to SSS
13:41 - Example: x(t) = a cos(wt) + b sin(wt), find conditions on a,b r.v.'s for x(t) be WSS
14:30 - Stationarity in the mean (example, cont'd)
19:51 - Stationarity in the autocorrelation (example, cont'd)
32:38 - Example: x[n] r.p with inpendent sample with x[2n] ~ Unif(-\sqrt(3),\sqrt(3) and x[2n+1] ~ N(0,1). Is x[n] WSS/SSS?
42:13 - Jointly WSS random processes
45:10 - LTI processing of WSS processes
47:05 - Mean function calculation (LTI processing of WSS processes)
space

Corrections:
39:38- left board, very bottom line should be \delta[k] instead of \delta[n-k] (Thanks to Gulin T.)
<<< Previous 10 Lectures <<< >>> Next 10 Lectures >>>

 

 

 

 

 

 

EE 503 Statistical Signal Processing and Modeling
(Fall 2019– 2020)

Short Description:

This course is the first course on statistical signal processing in the graduate curriculum of Department of Electrical and Electronics Engineering, Middle East Technical University (METU). Topics covered in this course are random vectors, random processes, stationary random processes, wide sense stationary processes and their processing with LTI systems with applications in optimal filtering, smoothing and prediction. A major goal is to introduce the concept of mean square error (MSE) optimal processing of random signals by LTI systems.

For the processing of the random signals, it is assumed that some statistical information about the signal of interest and distortion is known. By utilizing this information, MSE optimal LTI filters (Wiener filters) are designed. This forms the processing part of the course. The estimation of the statistical information to construct Wiener filters forms the modeling part of the course. In the modeling part, we examine AR, MA, ARMA models for random signals and give a brief discussion of Pade, Prony methods for the deterministic modeling. Among other topics of importance are decorrelating transforms (whitening), spectral factorization, Karhunen-Loeve transform

This course is a natural pre-requisite (not a formal one) to EE5506 Advanced Statistical Signal Processing. The estimation theory topics in EE 503 is mostly limited to the moment description of random processes which forms a special, but the most important, case of EE 5506.

Outline of Topics:

  1. Review
    1. Basics of Mathematical Deduction
      1. Necessary, Sufficient Conditions
      2. Proofs via contradiction, contraposition
    2. Basics of Linear Algebra
      1. Linear independence of vectors (points in linear space)
      2. Range and Null space of the combination process
      3. Projection to Range/Null Space (orthogonality principle)
      4. Positive Definite Matrices
    3. Basics of Probability
      1. Probability as a mapping, axioms, conditional probability
      2. Expectation, law of large numbers
      3. Moments, moment generating function

  2. Random Processes
    1. Random variables, random vectors (or a sequence of random variables), moment descriptions (mean, variance, correlation), decorrelating transforms
    2. Random processes, stationarity, wide Sense Stationarity (WSS), power spectral density, spectral factorization, linear time invariant processing of WSS random processes, ergodicity

    Ref: Therrien, Hayes, Papoulis, Ross
     
  3. Signal Modeling
    1. LS methods, Pade, Prony (Deterministic methods)
    2. AR, MA, ARMA Processes (Stochastic approach), Yule-Walker Equations, Non-linear set of equations for MA system fit
    3. Harmonic Processes

    Ref: Hayes, Papoulis
     
  4. Estimation Theory Topics
    1. Random parameter estimation
      1. Cost function, loss function, square error, absolute error
      2. Conditional mean (regression line) as the minimum mean square error (MSE) estimator, orthogonality properties
      3. Linear minimum mean square error (LMMSE) estimators, orthogonality principle
      4. Regression line, orthogonality
      5. FIR, IIR, Causal–IIR Wiener filters
      6. Linear Prediction, backward prediction
      7. Random vector LMMSE estimation (multiple parameter)
    2. Non-random parameter estimation
      1. Maximum likelihood method
      2. Best Linear Unbiased Estimator (BLUE)
      3. Discussion of linear estimators for the linear observation model y=Ax+n
    3. Karhunen – Loeve Transform

    Ref: Therrien, Hayes
     
References:

[Hayes]: M. H. Hayes, Statistical Signal Processing and Modeling, Wiley, New York, NY, 1996.

[Therrien]: C. W. Therrien, Discrete random signals and statistical signal processing, Prentice Hall, c1992.

[Papoulis]: A. Papoulis, Probability, Random Variables, and Stochastic Processes, 3rd edition, McGraw Hill, 1991.

[Ross]: S. M. Ross, Introduction to probability models, 7th ed. Harcourt Academic Press, 2000.