Machine Learning Analysis of Automation-Related Job Displacement

Machine Perception & Cognition | November 2025

Project Overview

This project used machine learning and statistical methods to examine potential automation-related job displacement across 30 occupations from 10 major occupational groups. A custom dataset was developed using 12 occupational and labor-market features, including automation exposure, probability of computerization, AI vulnerability, education requirements, wages, employment trends, and projected job growth.

Three approaches were used to classify occupations into low, medium, and high automation-risk groups: Hidden Markov Models (HMM), Principal Component Analysis (PCA) with K-Means clustering, and Linear Discriminant Analysis (LDA). MATLAB was used for data processing, model development, visualization, and evaluation.

Rather than treating automation exposure as the only indicator of displacement, the analysis examined how multiple occupational characteristics interact to influence the resulting risk classifications.

Objectives

• Develop a dataset combining automation, occupational, and labor-market characteristics.

• Identify characteristics associated with automation-related job displacement.

• Standardize and analyze the dataset using MATLAB.

• Apply HMM, PCA with K-Means, and LDA to classify occupations by risk.

• Compare the classifications produced by the different methods.

• Examine which features contribute most strongly to the resulting classifications.

Constraints and Challenges 

The analysis was developed around several practical considerations:

• Limited dataset: The study included 30 occupations, which limited the amount of data available for more complex models.

• No predefined risk labels: The dataset did not contain established low, medium, and high displacement categories, so the classifications had to be inferred from the available features.

• Different feature scales: The dataset combined percentages, wages, employment values, scores, and other measurements that required standardization before analysis.

• Interpretability: The methods needed to provide insight into why occupations were grouped together rather than simply producing classifications without an explanation.

Design Methodology 

The analysis combined data preparation, machine learning, and statistical methods to identify patterns in automation-related job displacement and understand the factors influencing the resulting classifications.

Dataset Development and Preprocessing

The analysis began with a custom dataset containing three occupations from each of ten major Standard Occupational Classification (SOC) groups. The 30 occupations represented a broad range of fields, including engineering, education, healthcare, business, manufacturing, and transportation.

Twelve features were used to characterize each occupation. These included the Automation Exposure Index, probability of computerization, AI vulnerability, Risk of Automation Score, minimum education level, employment, projected job openings, historical and projected growth, and wage-related measures.

Before applying the machine learning methods, the numerical features were standardized using z-score normalization. This placed the variables on a common scale so that features with larger numerical values would not disproportionately influence the analysis.

Hidden Markov Model

The first approach used a Hidden Markov Model because the desired risk category was not directly observable in the dataset. Instead, low, medium, and high automation risk were treated as hidden states inferred from observable occupational characteristics such as automation exposure, computerization probability, education, wages, and projected growth.

Because the MATLAB HMM implementation required discrete observations, K-Means clustering was first used to group the standardized feature vectors into discrete patterns. These cluster assignments were then used as the observable inputs to the HMM.

The model was trained using the Expectation-Maximization algorithm, and the Viterbi algorithm was used to determine the most likely hidden state for each occupation. The resulting states were interpreted as low, medium, or high risk based on their average automation-related characteristics.

Principal Component Analysis and K-Means 

PCA was used to reduce the original 12 features into a smaller number of dimensions while preserving the major patterns within the dataset. This made the relationships between occupations easier to visualize without having to analyze all 12 features simultaneously.

K-Means was then used to identify three groups within the reduced feature space. The clusters were interpreted as low, medium, and high automation-risk groups based on their feature characteristics.

Figure 2 shows the occupations positioned according to their PC1 and PC2 scores (the first two principal components), with the K-Means classifications represented by the three groups. This is a great visual representation of how the algorithm grouped occupations with similar combinations of characteristics together. 

The scree plot shows the percentage of the dataset's variance explained by each principal component. PC1 accounts for approximately 37%, PC2 for approximately 24%, and PC3 for approximately 20%. Since these first three components capture more than 80% of the total variance, they provide a compact representation of the original 12-feature dataset.

The PCA biplot combines the occupation scores with the loading directions of the original features. The length and direction of each feature vector show how strongly that characteristic contributes to the principal-component space. Automation Exposure Index, Probability of Computerization, and Risk of Automation Score load strongly toward the higher-risk region, while education, wages, and projected growth contribute toward the opposite direction. This plot basically explains which characteristics are responsible for the positions of the occupations.

Linear Discriminant Analysis and K-Means 

LDA was used as a complementary classification method to determine which combinations of features provided the strongest separation between the three risk groups. K-Means established the initial groups, while LDA identified linear combinations of the original features that best distinguished those groups.

The data were standardized before LDA was applied, and five-fold cross-validation was used to evaluate the resulting classification.

The LDA projection shows the occupations after they have been transformed into two discriminant dimensions, LD1 and LD2. The three risk groups form distinct regions in this space. LD1 provides the strongest separation between the low- and high-risk occupations, while LD2 helps distinguish the medium-risk group. Essentially, combinations of the original 12 features can separate the occupations more clearly than looking at individual features alone.

The LDA weight plot shows how strongly each original feature contributes to the two discriminant dimensions. Automation exposure and probability of computerization have strong contributions to LD1, while education and wages contribute in the opposite direction. LD2 is influenced by characteristics including education, wages, and job growth.

Results and Evaluation 

The three methods provided complementary perspectives on the same dataset. The HMM inferred hidden automation-risk states, PCA and K-Means revealed the underlying grouping structure, and LDA showed which combinations of features most strongly separated the resulting groups.

The three approaches were evaluated separately, with each providing an independent perspective on the dataset. The methods did not use the results of the HMM or PCA analysis as inputs to one another. Within the LDA approach, K-Means clustering was used to establish the initial groups before applying LDA.

The HMM classified occupations into low, medium, and high-risk categories based on their observable characteristics. Several occupations consistently appeared toward the lower-risk end, including Electrical Engineers, Robotics Engineers, Mechanical Engineers, Writers, Teachers, Dentists, and Pharmacists. Occupations such as Sewers, Welders, Butchers, Medical Secretaries, and Public Safety Telecommunicators appeared among the higher-risk classifications.

The PCA analysis showed that the first three components retained more than 80% of the original dataset's variance, indicating that much of the structure within the 12 features could be represented using a smaller number of dimensions.

LDA provided a quantitative evaluation of the classification and achieved 96.67% accuracy using five-fold cross-validation. The results showed that the selected occupational features could effectively distinguish between the three risk categories.

One of the main findings was that automation exposure alone did not determine displacement risk. The classifications also reflected characteristics such as education, wages, job growth, complexity, creativity, and human interaction. This was particularly visible in the PCA biplot and LDA weight plot, where these characteristics contributed to the separation between occupation groups.

Conclusions and Future Work 

The analysis demonstrated how multiple machine learning techniques can be used to examine automation-related job displacement from different perspectives. Rather than relying on a single automation metric, the results showed the value of combining automation indicators with occupational and labor-market characteristics.

The project also provided practical experience with data preprocessing, feature standardization, dimensionality reduction, unsupervised clustering, hidden-state modeling, supervised classification, visualization, and cross-validation in MATLAB.

Future work could expand the dataset to include a larger number of occupations and additional occupational characteristics. A larger dataset could also make more complex approaches, including neural networks, more practical while providing additional data for model training and evaluation.