Multi-Dimensional Exploration Is Possible!
Exploring multi-dimensional worlds is a common theme in sci-fi stories and movies. While traveling through such worlds remains a fantasy, math can help us approach even an idea as outrageous as this one. If you have ever thought, ‘Another day without calculating the determinant of a matrix,' then hold on. Linear algebra might be closer than you realize! Transitioning from complex to simpler data structures happens every day on your phone, computer, and streaming apps. These operations are real, mathematical multi-dimensional portals. This article explains Principal Component Analysis – what it is, why it matters, and how it works.
Carl Sagan explains…
There is a video of Carl Sagan on YouTube in which he explains how a visitor from a higher-dimensional world would look like to their lower-dimensional counterparts. This beautiful lesson is drawn from Sagan's renowned series, Cosmos. As with many other excerpts from his show, Sagan masterfully explains how some 2-dimensional individuals in "flatland" would experience a visit from a 3-dimensional apple. In reality, the Mathematics behind moving from higher to lower dimensional spaces does not go beyond the contents of a linear algebra course. As Sagan puts it towards the end of the video, ‘While we cannot imagine the world of four dimensions we can perfectly well think about it'.
An apple projected in a 2D space might not look exactly like an apple but it can give us an idea of its shape and size. The picture below shows an apple and a carrot projected onto 2-dimensional surfaces. We might recognize the apple only by looking at any of the projections. On the other hand, it might be difficult to realize that the lower projection corresponds to a carrot. This means that we lose some information every time we project high-dimensional objects onto low-dimensional spaces. However, these projections help us think about how a high-dimensional object would look in our world.
Depending on which plane we use to project the objects, we will get a different idea of their shape and characteristics. So, while it is true that projections can help us to think about a higher dimension, the way in which we project is crucial for our understanding. Take another look at the lower projection of the carrot. Is there a way of knowing that this is a carrot just judging by this projection? Probably not. It would be better to choose any of the other two projections. But, how can we choose them? If we are trying to think about an object in 10 dimensions, how can we choose the best three dimensions that our brain will be able to process?
Going beyond apples and carrots
To understand this, we first need to realize that multi-dimensional spaces are not only related to the worlds of carrots and apples. Each dimension can represent a particular parameter or characteristic in a set of data points. If we take a group of people and measure five of their characteristics we can plot each one of these individuals in a 5-dimensional space. Once again, we cannot imagine this space but we can think about it if we project each one of these 5-dimensional points onto a 3-dimensional or 2-dimensional space that we can visualize. In the example below, someone living in a 2-dimensional world can only see that there are three spheres if they are projected onto the lower (xy) plane. Otherwise, it will only see two which is not a good way of thinking about this group of 3-dimensional spheres after all.
The answer to which are the best dimensions to project data onto is solved with Principal Component Analysis (PCA). The name might seem unrelated to what we have been discussing so far but it originates from the fact that in this analysis we are finding which are the principal components of a dataset. In this case, components mean dimensions or planes where we can project the original data. How does PCA find the best components or dimensions? It looks at the variability of each pair of parameters. The bigger their differences, the easier is to have a good projection of their true behaviour. The reason why we cannot recognize a carrot if we look at its projection on the lower plane is because we fail to see the main characteristics of that carrot. Looking at that projection we cannot see it is long, thinner at the bottom and with some green leaves at the top. We only see a roundish blurb that could represent anything else. PCA helps us to look at an object or group of data points in such a way we can see their main characteristics.
Projections into lower-dimensional spaces
A matrix multiplication is a way of projecting points on a lower-dimensional space. This means, mathematically going from 3 to 2 dimensions or from 10 to 2 dimensions. The following picture shows a group of points in 2 dimensions that were projected onto one-dimensional lines. This picture is taken from a Python notebook that contains additional details about PCA. The lines in the plot could have been defined anywhere. Multiplying a matrix with the original points with a matrix that contains unit vectors that define each line will give us the projection of the original points onto the lines. So the red and green crosses represent how the blue points coming from a 2D world would look like in 1D. Note how the variation in the red crosses is bigger than in the green crosses. This means that in the green crosses world, the original points will not look as different as they look in the red crosses world. The main idea to take here is that even if we have a way to move through different dimensions (at least mathematically!), how we project the original data will impact the way we see and the way we think about higher-dimensional objects.
In an example similar to what we saw in the picture with the carrot and the apple, we can imagine a group of points in 3 dimensions and their projections in different planes. Once again, depending on the plane that we choose to project the points onto we will get different results. This is when PCA comes into play. With PCA we can define the best combination of projections so that we will end up looking at the biggest variation. So, PCA helps us to have a projection that will be more similar to the red crosses than to the green crosses which, in turn, will represent a better idea of a group of points in a higher dimension.
PCA relies on many of the things you might have studied in a linear algebra course. The steps to apply PCA can be summarized as follows:
Covariance matrix
Generate a covariance matrix: the covariance measures how strong the relationship between two variables is. If we have a group of points that belong to a 10-dimensional space, then we will have 45 different combinations of each pair of dimensions. If we calculate the variance for each of these combinations then we can have an idea about the direction and the strength of the relationship between two variables. The result of this calculation is written in a covariance matrix such as the one shown in the picture. Note how each value is repeated twice since each pair of variables can be found twice inside the matrix. Take into account that before calculating the covariances, the data is generally standardized since each dimension might contain values with different scales. Variables with larger scales will dominate the principal components. Standardizing the data ensures that all variables have the same scale, allowing PCA to give equal weight to each variable.
Eigenvectors and eigenvalues
Calculate the eigenvectors and eigenvalues of the covariance matrix: yes! You might have heard this name before in a Linear Algebra lecture! And yes! It might sound really strange! However, the key thing to understand here is that eigenvectors and eigenvalues tell us something special about a particular transformation. Remember how matrix multiplication is essentially a transformation in which we take some points and change them through sums and multiplications. An eigenvector represents a particular direction that does not change after performing a transformation. On the other hand, an eigenvalue represents how much the eigenvector is stretched or squished without changing its direction. So eigenvectors and eigenvalues are really important because they tell us which are the directions that remained unchanged after a transformation.
The following picture shows three different linear transformations applied to the same group of data. This linear transformation is a horizontal shear. For all the cases shown in the picture, the eigenvector is always the same: (1, 0). This means that any point that lies on top of the unit vector (1,0) will remain unchanged after the transformation. Note how the points (1,0) and (-1,0) are not changed by the transformation and neither is any point that lies on top of the eigenvector (1,0) such as (0.25, 0), (-0.75, 0) and so on.
Transformation
Use the eigenvectors to transform the original data into a lower-dimensional space. The eigenvalues will tell you which are the most important eigenvectors to project onto. So in a plot like the one shown in Figure 4, the eigenvector corresponding to the red crosses has a higher eigenvalue than the eigenvector with the green crosses. This is because the variation in the red crosses is bigger than in the green crosses. If we want to have a good idea about the points in 2D from a 1D point of view, it is better to transform the points using the first vector than the second one. You can see how an array of (nx2) corresponding to two dimensions is converted into one dimension if it is multiplied by an eigenvector of shape (2×1). We can follow the same process if we want to convert an original dataset of 10 dimensions into 3 or 2 dimensions.
Projection
Generally, it is useful to project the converted points onto the original dimensions. For this example, it means taking the transformed 1D points and plotting them back in a 2D space which is what Figure 4 is showing. To project the points onto the original dimensions you can multiply the transformed array by the transpose of the eigenvectors array. In this case, this means multiplying (nx1) times (1×2). This will result in the (nx2) array that is shown in Figure 7. In this plot, Eigenvector 1 and Eigenvector 2 have eigenvalues of 39.2 and 5.3 respectively. This means that Eigenvector 1 represents a better projection than Eigenvector 2 which can be seen in the plot as the spread of the red crosses is bigger than the spread in the green crosses.
A numerical carrot
Let's apply the previous steps to a problem similar to what we presented at the beginning of this article. The following picture shows a 3D distribution of points that look like a carrot (see this Python notebook for more information about how to build this carrot). It also shows the projections on each plane. We can see that someone who observes the projection on the XY plane will find it difficult to realize that the points actually represent a carrot, whereas projections ZX and ZY have a better representation. So, in this case, it is clear that if we want to visualize this 3D object in 2D, we better project it onto planes ZX or YZ.
What if the best projection plane is not that obvious? Let's say that now the group of points we want to project looks like the following picture. Note that is the same carrot as before but rotated. If we project these points onto the three surfaces we have seen before, we will end up with a rotated version of the carrot. The best projection is a plane that goes through the middle of the carrot and this is precisely the projection that we can get if we apply PCA to the original points. By finding the direction in which the variation between the points is bigger, PCA can help us visualize data points in high-dimensional spaces. The 2D picture of a carrot is a good representation of a 3D carrot although it is not exactly the same. However, it is better than projecting in any other plane. For more information about how to apply PCA in Python using scikit-learn, review this Python notebook.
Applications of PCA
One of the most important applications of PCA is related to image processing. This means taking an image and analyzing its features to classify it or recognize that an image belongs to a certain category. Image recognition algorithms are widely used nowadays in our cell phones and cameras. To understand how PCA is applied in image processing, we should first take a look at how images are stored and handled. This is a very simple example that can help us to understand the application of PCA in image processing.
Let's say we have a greyscale image that we want to process. One of the multiple ways of doing this is to convert this image into an array that has several rows and columns that correspond to the number of pixels in the image (see this Python notebook to learn how to load images and apply PCA to them). So at each pixel, we will have a number that typically ranges from 0 to 255 and that represents the intensity value. Zero represents black, 255 represents white, and any intermediate value is a different shade of grey. This means that the following 50×50 pixel greyscale picture can be represented as a 50×50 array.
A 50×50 array can also be understood as 50 points plotted in a space of 50 dimensions. While we cannot imagine a space such as this one, we can think about it! Each of the 50 points has 50 different characteristics and these characteristics end up creating the picture we see in Figure 11. Similar to what we did before, we can try to find which are the principal components of this data. This can help us to plot the points in a lower-dimensional space. The following picture shows how the projection would look like if we use different numbers of principal components. Note how the fewer the components the blurrier the image is. Also, note how we can get an image that looks really similar with 25 principal components.
What we are doing with this picture is not different from what we did previously with the points in space and the plane. In the case of the picture that is converted into 25 principal components, what we are doing is:
- Calculating the principal components of the original data set. This means that we built a 50×50 covariance matrix and calculated the eigenvectors and eigenvalues from that matrix.
- We extracted the 25 eigenvectors that had the biggest eigenvalues. Then we multiplied the original dataset of (nx50) by the array of eigenvectors (50×25). This transformed the original data into a 25-dimensional space. In our previous example, this is like going from the 3D carrot to the 2D carrot.
- Since we are interested in comparing how much information we lost by doing this dimensionality reduction, we can project the new dataset into a 50-dimensional space. This would be as taking the points in 2D and projecting them in the original 3D space.
- Figure 13 shows the comparison between the original image and the projected image using 25 principal components. It also shows the transformed image if we use a resolution of 50×25 instead of 50×50.
Now that we have an image in reduced dimensions we can think about how useful this is. Instead of storing and analyzing an array of 50×50=2500 numbers, we are actually using an array of 50×25=1250 numbers. This can help us to speed up multiple processes and to better understand our data.
The following picture shows a dimensionality reduction performed on a portrait with a high level of detail. A smaller number of principal components makes the image blurrier and more difficult to see. However, we can see that even when using half of the original dimensions we can get an image that is still acceptable for many applications.
Conclusion
Though our everyday experience is rooted in three dimensions, we have seen that it is possible to think about data structures in higher dimensions. As outrageous as it sounds, math is the portal that connects us to worlds of different dimensions. However, more important than this, is that going from lower to higher dimensions is not something only related to sci-fi and blockbuster movies. Transforming and projecting data in multiple dimensions occur seamlessly in our daily interactions with cell phones, apps, and computers. Processing and analyzing data in high-dimensional spaces allow us to understand patterns and reach conclusions that are part of our daily lives. Even though the process explained in this article is undoubtedly a simplification of a more convoluted process, the main idea still holds. So, next time you watch a movie in which the main character goes through a magic portal into a different dimension, remember that at the very least we have the numerical foundation to contemplate such journeys. As Carl Sagan said: ‘While we cannot imagine the world of four dimensions we can perfectly well think about it'.
