**Table Of Contents :-**

· PCA :-

· Why we use PCA?

∘ Curse of dimensionality :-

∘ Why Dimensionality Reduction?

∘ Feature Extraction :-

· PCA Geometric Intuition :-

· PCA mathematical Intuition :-

∘ How to decide this line is best principal component or not?

· Eigen decomposition :-

∘ Implementation :-

PCA (Principal Component Analysis) is a popular technique used for dimensionality reduction.

To understand this we have to understand curse of dimensionality

**Curse of dimensionality :-**

the curse of dimensionality refers to the sparsity of data in high-dimensional spaces, which can lead to overfitting, increased computational complexity, and reduced model performance. PCA helps combat this issue by reducing the number of dimensions while preserving the most important information in the data.

There are two ways to remove curse of dimensionality :-

- 1) Feature selection :- select most important features and then we train our model.
- 2) Dimensionality reduction(PCA) :- Feature Extraction

**Why Dimensionality Reduction?**

- 1) To prevent curse of dimensionality
- 2) To improve the performance of model.
- 3) To visualize the data.

**Feature Extraction :-**

Suppose we have a housing dataset in which room size and no. of rooms are two independent features and price is a output feature.

Then we apply some transformation to extract new feature from independent feature. Suppose we extract house size as a new feature.

- PCA is used for dimensionality reduction.
- Suppose we have housing dataset with below plot

- Let’s consider with the help of PCA we want to reduce 2-dimensional features into 1-dimension.
- We apply the transformation on the data. This transformation is called eigen decomposition on matrix.
- With the help of this transformation you get new axis which will look like →
- We will project all the information on the axis.
- you will get another axis for the no. of rooms

- Due to this maximum variance is get captured. Soo, we are able to convert 2-D → 3-D with ensure that more information is not loosed.
- The final goal of PCA is to find the best principal component in such a way that maximum variance is captured.

## How to decide this line is best principal component or not?

There are two important things :-1) Projections 2) cost function that is related to variance.

- Lets take one point for now,

- formula for point 1 :-
- Projection(p1)u = p1*u/||u|| →Projection of specific vector on unit vector.
- p1′ = p1*u(dot product) ……………….||u|| = 1.
- where we get p1, as a scalar value.
- likewise we get p1′, p2′, p3′ p4′ , ………………, pn’.
- This points are taking distance from the origin.
- lets consider p1′, p2′, p3′ p4′ , ………………, pn’ ~~ x1′, x2′, x3′, x4′, ……, xn’.
- Variance = Σ(xi-x̄)²/n — — → Goal is to find the best unit vector which captures the max variance.
- To find best unit vector we have a technique called eigen decomposition in which we calculate eigne vectors and the eigen values.

- 1) we will first find the covariance matrix between the features.
- 2) After that we calculate eigen vectors and eigen values using the covariance matrix.
- 3) Whichever the eigen vector is largest one(Eigen value is high for that eigen vector[Magnitude of Eigen Vector]) that captures the maximum variance.

Note :- Eigen vectors and values can be found out by the equation

A*v = λ *v ……(Linear Transformation of the matrix)