Principal Components Analysis

• Creating a single index

Combine the correlated variables to be one variable

Since population and advantage spending are in correlation, we can create a linear combination to capture two values into one variable to explain

• Understanding relationships between variables

• Seeing how variables are associated with observations on a single biplot.

• Visualising high-dimensional time series.

Assumption

• Features must be in normal distribution.

• Linearity: The data set to be linear combinations of the variables.

• PC2 has to be uncorrelated to PC1.

Some terms

• In multivariate (multiple variable) space, the correlation between the component and the original variables is called the component loadings.

• Think of it as correlation coefficients, squaring them give the amount of explained variation.
• Therefore the component loadings tell us how much of the variation in a variable is explained by the component

Weighted linear combination

Maximise variance

Second Principal Component

Example : Correlation between a variable and the corresponding PC.

A high (low) weight indicates a strong positive (negative) association between a variable and the corresponding PC.

 

Biplot

• A plot to indicate the weight vectors on the scatterplot
Why?
We want Biplot to :
• See how the observation relate to one another
• See how the variables relate to one another
• See how the observations relate to the variables
How ?
---
There are two ways to draw Biplots:
Distance Biplot
• See how the observations relate to one another
(By viewing the distances of the observation)


• See how the variables relate to one another
• (By viewing the angle of the variables )
• See how the observations relate to the variables
• (Draw an extended line for the variable and view how closed the observation is to the variable)
 
• Correlation Biplot
Distance Biplot 這個主要看 observation
---
• The distance between observations implies similarity between observations
Example : US

Louisiana (LA) and South Carolina (SC) are close therefore are similar.
Arkansas (AR) and California (CA) are far apart and therefore different.
• If the variables are ignored this is identical to a scatter plot of principal components.
 
Correlation Biplot 這個主要看 variable 標
---
• The angles between variables tell us something about correlation (approximately).
Example : US

Income and HSGrad are highly positively correlated.
• The angle between them is close to zero 銳角.
LifeExp and Income are close to uncorrelated.
• The angle between them is close 90 degrees 直角.
Murder and LifeExp are highly negatively correlated.
• The angle between them is close 180 degrees 鈍角.
 
 

Third PC

• uncorrelated with PC1 and PC2.

• has the highest variance

• cannot visualise this with a biplot

In which you cannot look at 109 time series plots to visualise general macroeconomic conditions. However, one can look at time series plots of the PC of these variables.

Implementation of PCA

Standardisation

Before conducting PCA, it is important to scale the data if it is of different scale. Otherwise the weights can be influenced by the units of measurement. (Sensitivity towards the data)

See that the difference can generate as high as 99.9% in the LifeExp, if you do not scale.

Do PCA in R

Use prcomp ()

• The output of the prcomp function is a prcomp object.

• It is a list that contains a lot of information. Of most interest are ($them) :
• The principal components : x
• The weights : rotation

• What is the Cumulative Proportion?
• It means the Proportion of the ability to interpret the data. (Higher Better)
• So, with PC1, PC2, and PC3 together, they can explain 90.7% of the data.

Screeplot : It is like an elbow chart

• Along the horizontal axis is the Principal Component.

• The Scree plot indicates how much each PC explains the total variance of the data.

• Look for a part where the plot flattens out also called the elbow of the Scree Plot.

• Along the vertical axis is the variance corresponding to each Principal Component.

• screeplot(pca,type="lines")

Significant of loadings (ETC3250)

```{r eval=FALSE}
# devtools::install_github("jimmyday12/fitzRoy")
library(fitzRoy)
aflw <- fetch_player_stats(2020, comp = "AFLW")
save(aflw, file="aflw.rda")
```

```{r}
library(boot) 
# The first variable, goals, can be used as the 
# indicator of sign because it has a large coefficient
compute_PC2 <- function(data, index) {
  pc2 <- prcomp(data[index,], 
                center=TRUE, 
                scale=TRUE)$rotation[,2]
  # Coordinate signs
  if (sign(pc2[1]) < 0) 
    pc2 <- -pc2 
  return(pc2)
}

# Make sure sign of first PC element is positive
PC2_boot <- boot(data=aflw_av[,4:36], compute_PC2, R=1000)
colnames(PC2_boot$t) <- colnames(aflw_av[,4:36])
PC2_boot_ci <- as_tibble(PC2_boot$t) %>%
  gather(var, coef) %>% 
  mutate(var = factor(var, 
                      levels=colnames(aflw_av[,4:36]))) %>%
  group_by(var) %>%
  summarise(q2.5 = quantile(coef, 0.025), 
            q50 = median(coef),
            q97.5 = quantile(coef, 0.975)) %>%
  mutate(t0 = PC2_boot$t0) 
```

```{r fig.height=4}
PC2_boot_ci %>% ggplot() +
  geom_point(aes(x=var, y=t0)) +
  geom_errorbar(aes(x=var, ymin=q2.5, ymax=q97.5),
                width=0.2) + 
  geom_hline(yintercept=c(-1/sqrt(nrow(PC2_boot_ci)),
                          1/sqrt(nrow(PC2_boot_ci))),
             colour="red") +
  theme(axis.text.x = element_text(angle = 90, 
                                   vjust = 0.5, 
                                   hjust=1)) +
  xlab("Predictors") + ylab("Coefficient") +
  geom_hline(yintercept = 0, color = "gray") 
```

Interpretation

To judge whether the variable is significant, it depends on 2 things, any violation means the variable is insignificant :

1) If the interval or point does not touch the 0 line, the variable is significant.

2) If the interval or point goes beyond the red line, the variable is significant

So, in this case, On PC2 m100 and m200 contrast m1500 and m3000 (and possibly marathon). These are significantly different from 0.

 

 

Math

• Sum of loading = 1 if standardized variables.

Drawing Scree plot

FAQ

PCA vs LDA

Component axes here mean direction.

In practice, often a PCA is done followed by an LDA for dimensionality reduction.

• Very similar; only differ that LDA does not have class-specific covariance matrices, but one shared covariance matrix among the classes. [TDS]

Reference

Extra resource