Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 726

Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 729

Deprecated: Function get_magic_quotes_gpc() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 421

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/recipecheck.php on line 80

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/recipecheck.php on line 94

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Warning: Use of undefined constant MathJaxInlineCallback - assumed 'MathJaxInlineCallback' (this will throw an Error in a future version of PHP) in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/MathJax.php on line 84

Warning: Use of undefined constant MathJaxEquationCallback - assumed 'MathJaxEquationCallback' (this will throw an Error in a future version of PHP) in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/MathJax.php on line 88

Warning: Use of undefined constant MathJaxLatexeqrefCallback - assumed 'MathJaxLatexeqrefCallback' (this will throw an Error in a future version of PHP) in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/MathJax.php on line 94

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/wikipublisher/wikipublisher.php on line 1020

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/wikipublisher/wikipublisher.php on line 1056

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/wikipublisher/wikipublisher.php on line 1074

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/cookbook/analyze.php on line 89

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/scripts/skins.php on line 118

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Function create_function() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 484

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/scripts/pagelist.php on line 436

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/scripts/pagelist.php on line 486

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/scripts/pagelist.php on line 488

Deprecated: Function get_magic_quotes_gpc() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 421

Deprecated: Function get_magic_quotes_gpc() is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 421

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/scripts/upload.php on line 345

Deprecated: Array and string offset access syntax with curly braces is deprecated in /cgihome/cis520/html/dynamic/2019/wiki/scripts/upload.php on line 384

Warning: Cannot modify header information - headers already sent by (output started at /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php:484) in /cgihome/cis520/html/dynamic/2019/wiki/pmwiki.php on line 1304
CIS520 Machine Learning | Lectures / PCR

PCA review and extensions

Given the SVD {$X = UDV^T$} {$ Z = UD$} are the principal components (PCs) or “component scores” or sometmes “factor scores”, and the columns of {$V$} are the corresponding loadings of the principal components. (Last time instead of {$D$} for the matrix with singular values on the diagonal, I used {$\Lambda$}.)

We write each vector {$x_i$} as {$x_i = \sum_k \alpha_{ik} v_k$}.

What is {$\alpha_{ik}$}?

As described last time, PCA can be viewed as an {$L_2$} optimization, minimizing distortion, the reconstruction error {$||X -\hat{X}||_F = ||X -ZV^T||_F$} where we use the thin SVD, keeping only the {$k$} largest singular vectors.

This can be generalized to other loss functions, giving sparse PCA.

The idea behind sparse PCA is simple:

{$argmin_{Z,V} ||X - Z V^T||^2$}

with constraints {$||v_i||_1 < c_1$} for {$i = 1...k$} {$||z_i||_1 < c_2$} for {$i = 1...k$}

You can also view this as a sort of penalized regression PCA.

How do the constraints end up in the main loss function?

PCR

We can also use PCA as a way of generating features for use in a supervised setting. The simplest approach is to just project the feature vectors {$x_i$} onto the largest principal components {$v_j$},

The approximation {$x_i = \sum_j z_{ij} v_j$} or {$X= ZV^T$}

getting a new set of features {$z_i = x_i V^T$} (or {$Z = XV^T$}) , and then run the regression with {$z_i$} (or {$Z$}) as the features.

Keeping only the largest principal components is a form of regularization. (why?)

PCR and Ridge

Recall that OLS regression estimates {$\hat{w} = (X^TX)^{-1} X^T y$} or

{$\hat{y}^{OLS} = X(X^TX)^{-1} X^T y$}

but in a singular value decomposition with {$X = UDV^T$} , we have

covariance matrix {$X^TX = (UDV^T)^T(UDV^T) = VD^2V^T$}

and hat matrix

{$X(X^TX)^{-1}X^T = UDV^T (VD^{-2}V^T) VDU^T = UU^T$}

Aside: {$U^TU$} is the k*k identity matrix; {$UU^T$} in the n*n hat matrix

so

{$\hat{y}^{OLS} = UU^T y$}

Ridge regression estimates {$\hat{w} = (X^TX + \gamma I)^{-1} X^T y$} or

{$\hat{y}^{ridge} = X(X^TX + \gamma I)^{-1} X^T y$}

{$X(X^TX + \gamma I)^{-1}X^T = UDV^T V(D^2 + \gamma I)^{-1}V^T VDU^T$} = = {$UD(D^2 + \gamma I)^{-1}DU^T$} = {$\sum_j u_j (\lambda_j^2 / (\lambda_j^2 + \gamma) u_j^T$}

or, equivalently {$X(X^TX + \lambda I)^{-1} X = V \Gamma V^T$} where {$\Gamma$} is a diagonal matrix with diagonal elements {$\lambda_j^2/(\lambda_j^2 + \gamma)$}

so Ridge does shrinkage in the singular value space, effectively shrinking the small singular values more than the big ones. Note that PCR does a “feature selection” in the singular value space, just zeroing out all the small singular values.

PLS (Partial Least Squares)

Often we have multiple {$y$}’s for each observation, so that we might want to predict a vector {$y_i$} for each observation {$x_i$}. This can be done using standard (OLS) regression, or principal components regression (PCR), but can also be done in other ways.

For example, we can try to simultaneously minimize the criteria for linear regression {$||y - \hat{y}||_2$} or, equivalently {$||y - Zw||_2$} and for PCA {$||X - \hat{X}||_2$} or equivalently {$||X - ZV||_2$} PLS can be viewed as doing something like this. (Note that there is a whole class of methods that range from OLS to PCA, depending on how much they weight the two components.

CCA

As we’ve seen, PCA can be viewed as trying to find the directions that capture the maximum variance in the {$X$} space. This is done by finding the eigenvectors of {$X^TX$} (Assume through-out this that we have mean centered {$X$} and {$Y$})

If one has two sets of observations {$X$} and {$Y$}, one can instead try to maximize the covariance between them. This is done by finding the singular vectors of {$X^TY$}. This method is called canonical covariance.

Having two sets of observations allows an important extension. Instead of maximizing the covariance, one can maximize the correlation. This leads to ‘ ’canonical correlation analysis” or CCA, which finds the left and right singular vectors of

{$(X^TX)^{-1/2} X^TY (Y^TY)^{-1/2} $}

The largest left and right singular vectors {$u_1$} and {$v_1$} are the projections that maximize the correlation between {$X u$} and {$Y v$}. One can view the pre- and post-multiplication of the covariance {$X^TY$} as whitening it, both rescaling and removing redundancy.

Before, in PCA, we found a way to map from {$X$} to a (in the thin SVD case) reduced dimension coordinate system, finding new coordinates for each observation {$x_i$} CCA does the same, but now finds new coordinates for both {$X$} and {$Y$}, in a way that maximizes the correlation between them.

Note that because it uses correlation, CCA (unlike PCA or canonical covariates) is ‘ ’scale invariant”.

Extra questions

  • Show that PCA can be kernelized
  • Explain Rayleigh quotients as an alternate formulation for CCA??

Back to Lectures