Principal Component Analysis (PCA)
A statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components.
PCA is widely used for dimensionality reduction, feature extraction, and data visualization. It works by identifying the directions (principal components) along which the variance in the data is maximal.
The first principal component accounts for the largest possible variance in the data, and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components.
Covariance Matrix
Where ฮฃ is the covariance matrix, xแตข is an observation, and xฬ is the mean of the observations.
Eigenvectors and Eigenvalues
Here, v represents the eigenvectors (principal components) and ฮป represents the eigenvalues (variance explained).
Interactive PCA Visualization
Interactive Plot:
๐ก Try: hover over elements, zoom with mouse wheel, pan by dragging, use toolbar buttons
Advantages of PCA
- โ Reduces dimensionality, combating the curse of dimensionality.
- โ Improves model performance by removing noise and redundant features.
- โ Facilitates data visualization in lower dimensions.
- โ Can speed up machine learning algorithms.
Disadvantages of PCA
- โ Loss of interpretability: Principal components are linear combinations of original features.
- โ Information loss: Some variance is discarded, potentially losing important details.
- โ Sensitive to scaling: PCA is affected by the scale of features.
- โ Assumes linearity: PCA works best when relationships between variables are linear.
Key Takeaways
- โ PCA transforms data into a new coordinate system based on variance.
- โ Principal components are orthogonal and ordered by explained variance.
- โ Useful for dimensionality reduction, visualization, and noise reduction.
- โ Requires feature scaling and can lead to loss of interpretability.