Machine Learning | Model Drift | Mlops
Taming the Tides of Drift in Machine Learning
Types of model drift in machine learning and ways to measure and monitor them.
In this article we will look at the concepts of model drift in machine learning, why it occurs, how to monitor them, and how to fix them. Let’s start with the definition. A model deployed in production is considered drifting when its predictive accuracy reduces over time.
Let us first understand what drift means. Suppose, we have three models deployed in production. Model A, Model B, and Model C. The chart above shows the accuracy of models across 5 months. Model A starts with an accuracy of 90%, and over the next four months, it fluctuates between 90% and 95%. It stays within the band, and hence considered not drifting. Model B starts with an accuracy of 90%, but after the third month, its performance starts degrading and falls to 83% at the end of month five. This is considered a sudden drift. On the contrary, Model C starts at 90% accuracy, and its accuracy goes down slowly over time to 68%. This is called slow drift. If the models accuracy fluctuates between the band, it is okay and not considered as a drift. It is important to monitor the model performance after deployment and look out for these drifts.
Next, let us look at different types of drifts that can occur when the model is deployed in production.
Data drift
Data drift refers to the phenomenon where the statistical properties (like the distribution) of the data used for training a machine learning model changes over time.
To illustrate this concept, let’s consider an example. Suppose, you have a large dataset of emails, labeled as “spam” or “not spam,” and you use this data to train a machine learning model. During training, the model learns patterns and features in the emails that distinguish spam from non-spam emails. Next, you deploy this model to start filtering incoming emails in real-time. Over time, the email patterns, certain words or phrases that were once indicative of spam may change as spammers find new ways to bypass those spam filter models. This shift in the email data distribution is an example of data drift. As a result of this the accuracy of your spam detection model may start dropping.
Feature Drift
Feature drift occurs when the characteristics of the input variables in the deployed environment differ from those of the training environment. Feature drift occurs due to reasons like changes in source data (new classes get added), data generation or data processing error (like missing data, erroneous data, etc.)
Let’s consider a model that predicts whether a person is eligible for loan. Say we have a feature ‘age of the applicant’. Let’s look at the distribution of the classes from the variable over a few months.
In the first three months, the ratio of applicants with age > 45 to age > 20 is approximately 2. In months four and five, the ratio drops to approximately 1. It is also important to note, the change in ratio was primarily due to change in the percentage of applicants with age > 45. The percentage of applicants with age > 20 did not change much. This type of change in the distribution of classes is called feature drift.
Concept Drift
Concept drift is another important concept in machine learning, and it is closely related to both data drift and feature drift. Feature drift specifically refers to the situation where the input features themselves change over time. Concept drift occurs when the underlying concept being learned by the ml model changes over time.
Suppose you build a model to predict customer churn for Netflix subscriptions. During model training, you use historical data that includes features like customer demographics, usage patterns, and engagement metrics to predict whether a customer is likely to churn or not. Over time, Netflix introduces some changes to its pricing plans or content offerings (for e.g. password sharing). As a result, the factors influencing customer churn may evolve, leading to concept drift. Customers who were previously satisfied might now be unhappy and want to leave. The model’s assumptions about the relationship between the input features and churn no longer hold true, resulting in drop in accuracy.
How to measure and manage feature drift and concept drift:
- Continuously monitor the model’s performance metrics.
- Develop methods to adapt the model to the changing concept. This can involve techniques such as online learning, where the model is updated incrementally as new data arrives.
- Retrain and update your model to account for the changes in the underlying concept.
- Set up prediction accuracy thresholds.
- Combining predictions (ensemble methods) from multiple models trained on different time periods or subsets of data can help mitigate the impact of concept drift.
Thank you for reading. Get in touch via LinkedIn if you have further questions.
