Principal Component Analysis

In this page, we will learn What is Principal Component Analysis (PCA)?, Some common terms used in PCA algoritm, Principal Components in PCA, Steps for PCA algorithm, Applications of Principal Component Analysis.

---

What is Principal Component Analysis (PCA)?

Principal Component Analysis is an unsupervised learning algorithm used in machine learning to reduce dimensionality. With the help of orthogonal transformation, it is a statistical technique that turns observations of correlated features into a set of linearly uncorrelated data. The Principal Components are the newly altered features. It's one of the most widely used programs for exploratory data analysis and predictive modeling. It's a method for extracting strong patterns from a dataset by lowering variances.

PCA seeks out the lowest-dimensional surface on which to project the high-dimensional data.

The variance of each characteristic is taken into account by PCA since the high attribute indicates a good separation between the classes and so minimizes dimensionality. Image processing, movie recommendation systems, and optimizing power allocation in multiple communication channels are some of the real-world uses of PCA. Because it is a feature extraction technique, it keeps the significant variables while discarding the less important ones.

The PCA algorithm is based on the following mathematical concepts:

• Variance and Covariance
• Eigenvalues and Eigen factors

Some common terms used in PCA algoritm:

Dimensionality: The number of features or variables in a dataset is referred to as its dimensionality. It's the amount of columns in the dataset, to put it simply.

Correlation: It is a term that describes how closely two variables are related to one another. For example, if one variable changes, the other variable changes as well. The correlation value can be anywhere between -1 and +1. If the variables are inversely proportional to each other, the result is -1, and if the variables are directly proportional to each other, the result is +1.

Orthogonal: It denotes that the variables are unrelated to one another, and so the correlation between them is zero.

Eigenvectors: If you have a square matrix M and a non-zero vector v, you have eigenvectors. If Av is the scalar multiple of v, then v is an eigenvector.

Covariance Matrix: The Covariance Matrix is a matrix that contains the covariance between two variables.

Principal Components in PCA

The Principal Components are the converted new features or the result of PCA, as stated above. The number of PCs in this dataset is either equal to or less than the number of original features in the dataset. The following are some of the properties of these main components:

• The linear combination of the original features must be the main component.
• These components are orthogonal, which means there is no link between two variables.
• When going from 1 to n, the importance of each component declines, indicating that the 1 PC is the most important and the n PC is the least important.

Steps for PCA algorithm

Getting the dataset
To begin, we must divide the input dataset into two halves, X and Y, with X being the training set and Y being the validation set.

Representing data into a structure
Now we'll use a structure to represent our data. As an example, the two-dimensional matrix of independent variable X will be represented. Each column correlates to the Features, and each row corresponds to the data elements. The dataset's dimensions are determined by the number of columns.

Standardizing the data
We'll normalize our data in this step. In a given column, for example, features with a large variance are more essential than features with a lower variance.

If the importance of features is unaffected by the variance of the feature, we shall divide each data item in a column by the column's standard deviation. The matrix will be referred to as Z in this case.

Calculating the Covariance of Z
We will take the matrix Z and transpose it to get the covariance of Z. We'll multiply it by Z after it's been transposed. The Covariance matrix of Z will be the output matrix.

Calculating the Eigen Values and Eigen Vectors
The eigenvalues and eigenvectors for the resulting covariance matrix Z must now be calculated. The directions of the axes with high information are called eigenvectors or the covariance matrix. The eigenvalues are defined as the coefficients of these eigenvectors.

Sorting the Eigen Vectors
We'll take all of the eigenvalues and arrange them in decreasing order, from largest to smallest, in this phase. In the eigenvalues matrix P, sort the eigenvectors in the same order. P* will be the name of the resulting matrix.

Calculating the new features Or Principal Components
We'll calculate the new features here. We'll do this by multiplying the P* matrix by the Z. Each observation in the resulting matrix Z* is a linear combination of the original features. The Z* matrix's columns are all independent of one another.

Remove less or unimportant features from the new dataset.
We'll determine what to keep and what to discard now that the new feature set has arrived. It indicates that only relevant or crucial features will be kept in the new dataset, while unimportant ones will be deleted.

Applications of Principal Component Analysis

• PCA is mostly utilized as a dimensionality reduction technique in AI applications like computer vision and picture compression.
• If the data includes a lot of dimensions, it can also be utilized to find hidden patterns. Finance, data mining, psychology, and other fields employ PCA.