Where I work, we are obsessed with what happens to a model's performance after it has been deployed. We call this post-deployment data science.
text
Where I work, we are obsessed with what happens to a model's performance after it has been deployed. We call this post-deployment data science.
new_line
new_line
Let me tell you about a post-deployment data science algorithm that we recently developed to measure the impact of Concept Drift on a model's performance.
text
Let me tell you about a post-deployment data science algorithm that we recently developed to measure the impact of Concept Drift on a model's performance.
new_line
new_line
How can we detect Concept Drift? 🤔
text
How can we detect Concept Drift? 🤔
new_line
new_line
All ML models are designed to do one thing: learning a probability distribution in the form of P(y|X). In other words, they try to learn how to model an outcome 'y' given the input variables 'X'. 🧠
text
All ML models are designed to do one thing: learning a probability distribution in the form of P(y|X). In other words, they try to learn how to model an outcome 'y' given the input variables 'X'. 🧠
new_line
new_line
This probability distribution, P(y|X), is also called Concept. Therefore, if the Concept changes, the model may become invalid.
text
This probability distribution, P(y|X), is also called Concept. Therefore, if the Concept changes, the model may become invalid.
new_line
new_line
❓But how do we know if there is a new Concept in our data?
text
❓But how do we know if there is a new Concept in our data?
new_line
❓Or, more important, how do we measure if the new Concept is affecting the model's performance?
text
❓Or, more important, how do we measure if the new Concept is affecting the model's performance?
new_line
new_line
💡 We came up with a clever solution where the main ingredients are a reference dataset, one where the model's performance is known, and a dataset with the latest data we would like to monitor.
text
💡 We came up with a clever solution where the main ingredients are a reference dataset, one where the model's performance is known, and a dataset with the latest data we would like to monitor.
new_line
new_line
👣 Step-by-Step solution:
text
👣 Step-by-Step solution:
new_line
new_line
1️⃣ We start by training an internal model on a chunk of the latest data. ➡️ This allows us to learn the new possible Concept presented in the data.
text
1️⃣ We start by training an internal model on a chunk of the latest data. ➡️ This allows us to learn the new possible Concept presented in the data.
new_line
new_line
2️⃣ Next, we use the internal model to make predictions on the reference dataset.
text
2️⃣ Next, we use the internal model to make predictions on the reference dataset.
new_line
new_line
3️⃣ We then estimate the model's performance on the reference dataset, assuming the model's predictions on the monitoring data as ground truth.
text
3️⃣ We then estimate the model's performance on the reference dataset, assuming the model's predictions on the monitoring data as ground truth.
new_line
new_line
4️⃣ If the estimated performance of the internal model and the actual monitored model are very different, we then say that there has been a Concept Drift.
text
4️⃣ If the estimated performance of the internal model and the actual monitored model are very different, we then say that there has been a Concept Drift.
new_line
new_line
To quantify how this Concept impacts performance, we subtract the actual model's performance on reference from the estimated performance and report a delta of the performance metric. ➡️ This is what the plot below shows. The change of the F1-score due to Concept drift! 🚨
text
To quantify how this Concept impacts performance, we subtract the actual model's performance on reference from the estimated performance and report a delta of the performance metric. ➡️ This is what the plot below shows. The change of the F1-score due to Concept drift! 🚨
new_line
new_line
This process is repeated for every new chunk of data that we get. 🔁
text
This process is repeated for every new chunk of data that we get. 🔁
new_line
new_line
Where I work, we are obsessed with what happens to a model's performance after it has been deployed. We call this post-deployment data science.

Let me tell you about a post-deployment data science algorithm that we recently developed to measure the impact of Concept Drift on a model's performance.

How can we detect Concept Drift? 🤔

All ML models are designed to do one thing: learning a probability distribution in the form of P(y|X). In other words, they try to learn how to model an outcome 'y' given the input variables 'X'. 🧠

This probability distribution, P(y|X), is also called Concept. Therefore, if the Concept changes, the model may become invalid.

❓But how do we know if there is a new Concept in our data?
❓Or, more important, how do we measure if the new Concept is affecting the model's performance?

💡 We came up with a clever solution where the main ingredients are a reference dataset, one where the model's performance is known, and a dataset with the latest data we would like to monitor.

👣 Step-by-Step solution:

1️⃣ We start by training an internal model on a chunk of the latest data. ➡️ This allows us to learn the new possible Concept presented in the data.

2️⃣ Next, we use the internal model to make predictions on the reference dataset.

3️⃣ We then estimate the model's performance on the reference dataset, assuming the model's predictions on the monitoring data as ground truth.

4️⃣ If the estimated performance of the internal model and the actual monitored model are very different, we then say that there has been a Concept Drift.

To quantify how this Concept impacts performance, we subtract the actual model's performance on reference from the estimated performance and report a delta of the performance metric. ➡️ This is what the plot below shows. The change of the F1-score due to Concept drift! 🚨

This process is repeated for every new chunk of data that we get. 🔁


👍
ajibawa-2023 fblgit jessicagab Nhebo victor vishwask codito anujd9
❤️
gsarti clem dlicari samusenps Kukedlc