Explainable Anomaly Detection with RuleFit: An Intuitive Information

your anomaly detection outcomes to your stakeholders, the rapid subsequent query is all the time “why?”.

In follow, merely flagging an anomaly isn’t sufficient. Understanding what went incorrect is crucial to figuring out the very best subsequent motion.

But, most machine learning-based anomaly detection strategies cease at producing an anomaly rating. They’re black-box in nature, which makes it painful to make sense of their outputs-why does this pattern have the next anomaly rating than its neighbors?

To deal with this explainability problem, you might have probably already resorted to common eXplainable AI (XAI) methods. Maybe you’re calculating function significance to establish which variables are driving the abnormality, or you’re operating counterfactual evaluation to see how shut a case was to regular.

These are helpful, however what in case you might do extra? What in case you can derive a set of interpretable IF-THEN guidelines that characterize the recognized anomalies?

That is precisely what the RuleFit algorithm [1] guarantees.

On this put up, we’ll discover how the RuleFit algorithm works intuitively, how it may be utilized to elucidate detected anomalies, and stroll by a concrete case examine.

1. How Does It Work?

Earlier than diving into the technical particulars, let’s first make clear what we intention to have after making use of the algorithm: We need to have a set of IF-THEN guidelines that quantitatively characterize the irregular samples, in addition to the significance of these guidelines.

To get there, we have to reply two questions:

(1) How will we generate significant IF-THEN circumstances from the info?

(2) How will we calculate the rule significance rating to find out which of them truly matter?

The RuleFit algorithm addresses these questions by splitting the work into two complementary elements, the “Rule” and the “Match”.

1.1 The “Rule” in RuleFit

In RuleFit, a rule appears to be like like this:

IF x1 < 10 AND x2 > 5 THEN 1 ELSE 0

Would this construction look a bit extra acquainted if we visualize it like this:

Sure, it’s a choice tree! The rule right here is simply traversing one particular path by the tree, from the foundation node to the leaf node.

In RuleFit, the rule technology course of closely depends on constructing choice bushes, which predict the goal consequence given the enter options. As soon as the tree is constructed, any path from the foundation to a node in a tree might be transformed to a call rule, as we’ve got simply seen within the instance above.

To make sure the foundations are various, RuleFit doesn’t simply match one choice tree. As a substitute, it leverages tree ensemble algorithms (e.g., random forest, Gradient Boosting bushes, and many others.) to generate many various choice bushes.

Additionally, the depths of these bushes are, on the whole, completely different. This brings the advantages of producing guidelines with variable lengths, additional enhancing the variety.

Right here, we must always notice that though the ensemble bushes are constructed with predicting the goal consequence in thoughts, the RuleFit algorithm does probably not care concerning the finish prediction outcomes. It merely makes use of this tree-building train because the automobile to extract significant, quantitative guidelines.

Successfully, which means we are going to discard the anticipated worth in every node and solely preserve the circumstances that lead us to a node. These circumstances produce the foundations we care about.

Okay, we are able to now wrap up the primary processing step within the RuleFit algorithm: the rule constructing. The result of this step is a pool of candidate guidelines that would probably clarify the precise knowledge habits.

However out of all these guidelines, which of them truly deserve our consideration?

Effectively, that is the place the second step of RuleFit is available in. We “match” to rank.

1.2 The “Match” in RuleFit

Basically, RuleFit uncovers crucial guidelines by way of function choice.

First, RuleFit treats every rule as a brand new binary function, that’s, if the rule is happy for a selected pattern, it will get a worth of 1 for this binary function; in any other case, its worth is 0.

Then, RuleFit performs sparse linear regression with Lasso through the use of all of the “uncooked” options from the unique dataset, in addition to the newly engineered binary options derived from the foundations, to foretell the goal consequence. This manner, every function (uncooked options + binary rule options) will get a coefficient.

One key attribute of Lasso is that its loss operate forces the coefficients of these unimportant options to be precisely zero. This successfully means these unimportant options are faraway from the mannequin.

Consequently, by merely inspecting which binary rule options survived the Lasso evaluation, we might instantly know which guidelines are vital by way of getting correct predictions of the goal consequence. As well as, by wanting on the coefficient magnitudes related to the rule options, we might have the ability to rank the significance of the foundations.

1.3 Recap

We now have simply lined the important idea behind the RuleFit algorithm. To summarize, we are able to view this strategy as a two-step answer for offering explainability:

(1) It first extracts the foundations by coaching an ensemble of choice bushes. That’s the “Rule” half.

(2) It then cleverly converts these guidelines into binary options and performs customary function choice through the use of sparse linear regression (Lasso). That’s the “Match” half.

Lastly, the surviving guidelines with non-zero coefficients are vital ones which might be price our consideration.

At this level, you might have observed that “predicting goal consequence” pops up at each the “Rule” and “Match” steps. If we’re coping with a regression or classification downside, it’s simply comprehensible that the “goal consequence” is the numerical worth or the label we need to predict, and the foundations might be interpreted as patterns that drive the prediction.

However what about anomaly detection, which is essentially an unsupervised process? How can we apply RuleFit there?

2. Anomaly Clarification with RuleFit

2.1 Software Sample

To start with, we have to remodel the unsupervised explainability downside right into a supervised one. Right here’s how.

As soon as we’ve got our anomaly detection outcomes (doesn’t matter which algorithm we used), we are able to create binary labels, i.e., 1 for an recognized anomaly and 0 for a standard knowledge level, as our “goal consequence.” This manner, we’ve got precisely what RuleFit wants: the uncooked options, and the goal consequence to foretell.

Then, the RuleFit can work its magic to generate a pool of candidate guidelines and match a sparse linear regression mannequin to retain solely the vital guidelines. The coefficients of the ensuing mannequin would then point out how a lot every rule contributes to the log-odds of an occasion being categorized as an anomaly. To place it one other method, they inform us which rule combos most strongly push a pattern towards being labeled as anomalous.

Observe which you could, in idea, additionally use the anomaly rating (produced by the first anomaly detection mannequin) because the “goal consequence”. It will change the appliance of RuleFit from a classification setting to a regression setting.

Each approaches are legitimate, however they reply barely completely different questions: With the binary label classification setting, the RuleFit uncovers “What makes one thing an anomaly?“; With the anomaly rating regression setting, the RuleFit uncovers “What drives the severity of an anomaly?“.

In follow, the foundations generated by each approaches will most likely be very related. However, utilizing a binary anomaly label because the goal for a RuleFit is extra generally used for explaining detected anomalies. It’s easy by way of interpretation and direct applicability to creating enterprise guidelines for flagging future anomalies.

2.2 Case Examine

Let’s stroll by a concrete instance to see how RuleFit works in motion. Right here, we’ll create an anomaly detection state of affairs utilizing the Iris dataset [2] (licensed CC BY 4.0), the place every pattern consists of 4 options (sepal_length, sepal_width, petal_length, petal_width) and is labeled as one of many following three classes: Setosa, Versicolor, and Virginica.

Step 1: Knowledge Setup

First, we’ll use all Setosa samples (50) and all Versicolor samples (50) because the “regular” samples. For the “irregular” samples, we’ll use a subset of Virginica samples (10).

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, confusion_matrix
np.random.seed(42)

# Load the Iris dataset
iris = load_iris()
X = pd.DataFrame(iris.knowledge, columns=iris.feature_names)
y_true = iris.goal

# Get regular samples (Setosa + Versicolor)
normal_mask = (y_true == 0) | (y_true == 1)
X_normal_all = X[normal_mask].copy()

# Get Virginica samples
virginica_mask = (y_true == 2)
X_virginica = X[virginica_mask].copy()

# Randomly choose 10
anomaly_indices = np.random.selection(len(X_virginica), measurement=10, exchange=False)
X_anomalies = X_virginica.iloc[anomaly_indices].copy()

To make the state of affairs extra practical, we create a separate coaching set and take a look at set. The practice set incorporates pure “regular” samples, whereas the take a look at set consists of randomly sampled 20 “regular” samples and 10 “irregular” samples.

train_indices = np.random.selection(len(X_normal_all), measurement=80, exchange=False)
test_indices = np.setdiff1d(np.arange(len(X_normal_all)), train_indices)

X_train = X_normal_all.iloc[train_indices].copy()
X_normal_test = X_normal_all.iloc[test_indices].copy()

# Create take a look at set (20 regular + 10 anomalous)
X_test = pd.concat([X_normal_test, X_anomalies], ignore_index=True)
y_test_true = np.concatenate([
    np.zeros(len(X_normal_test)),   
    np.ones(len(X_anomalies))       
])

Step 2: Anomaly Detection

Subsequent, we carry out anomaly detection. Right here, we fake we don’t know the precise labels. On this case examine, we apply Native Outlier Issue (LOF) because the anomaly detection algorithm, which locates anomalies by measuring how remoted a knowledge level is in comparison with the density of its native neighbors. In fact, you can even attempt different anomaly detection algorithms, akin to Gaussian Combination Fashions (GMM), Ok-Nearest Neighbors (KNN), and Autoencoders, amongst others. Nonetheless, take into account that the intention right here is just to get the detection outcomes, our essential focus is the anomaly clarification in step 3.

Particularly, we’ll use the pyOD library to coach the mannequin and make inferences:

# Set up the pyOD library
#!pip set up pyod

from pyod.fashions.lof import LOF

# Standardize options
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.remodel(X_test)

# Native Outlier Issue
lof = LOF(n_neighbors=3)
lof.match(X_train_scaled)

train_scores = lof.decision_function(X_train_scaled)
test_scores = lof.decision_function(X_test_scaled)
threshold = np.percentile(train_scores_lof, 99)
y_pred = (test_scores > threshold).astype(int)

Discover that we’ve got used the 99% quantile of the anomaly scores obtained on the coaching set as the brink. For particular person take a look at samples, if its anomaly rating is greater than the brink, this pattern will likely be labeled as “anomaly”. In any other case, the pattern is taken into account “regular”.

At this stage, we are able to shortly verify the detection efficiency with:

classification_report(y_test_true, y_pred, target_names=['Normal', 'Anomaly'])

Not tremendous nice outcomes. Out of 10 true anomalies, solely 5 of them are caught. Nonetheless, the excellent news is that LOF didn’t produce any false positives. You possibly can additional enhance the efficiency by tuning the LOF mannequin hyperparameters, adjusting the brink, and even contemplating ensemble studying methods. However remember: our objective right here is to not get the very best detection accuracy. As a substitute, we intention to see if RuleFit can correctly generate guidelines to elucidate the anomalies detected by the LOF mannequin.

Step 3: Anomaly Clarification

Now we’re attending to the core subject. To use RuleFit, let’s first set up the library from imodels, which is a sklearn-compatible, Interpretable ML bundle for concise, clear, and correct predictive modeling:

pip set up imodels

On this case, we are going to take into account a binary label classification setting, the place the irregular samples (within the take a look at set) flagged by the LOF mannequin are labeled as 1, and different un-flagged regular samples (additionally within the take a look at set) are labeled as 0. Observe that we’re labeling based mostly on LOF’s detection outcomes, not the precise floor fact, which we fake we don’t know.

To provoke the RuleFit mannequin:

from imodels import RuleFitClassifier

rf = RuleFitClassifier(                 
        max_rules = 30,           
        lin_standardise=True,           
        include_linear=True,           
        random_state = 42
)

We are able to then proceed with becoming the RuleFit mannequin:

rf.match(
    X_test, 
    y_pred, 
    feature_names=X_test.columns
)

In follow, it’s often an excellent follow to do a fast sanity verify to judge how nicely the RuleFit mannequin’s predictions align with the anomaly labels decided by the LOF algorithm:

from sklearn.metrics import accuracy_score, roc_auc_score

y_label = rf.predict(X_test)               
y_prob  = rf.predict_proba(X_test)[:, 1]   

print("accuracy:", accuracy_score(y_pred, y_label))
print("roc-auc:", roc_auc_score (y_pred, y_prob))

For our case, we see that each printouts are 1. This confirms that the RuleFit mannequin has efficiently discovered the patterns that LOF used to establish anomalies. To your personal issues, in case you observe values a lot decrease than 1, you would wish to fine-tune your RuleFit hyperparameters.

Now let’s look at the foundations:

guidelines = rf._get_rules()
guidelines = guidelines[rules.coef != 0]                         
guidelines = guidelines[~rules.type.str.contains('linear')]      
guidelines['abs_coef'] = guidelines['coef'].abs()
guidelines = guidelines.sort_values('significance', ascending=False)

The RuleFit algorithm returns a complete of 24 guidelines. A snapshot is proven beneath:

Let’s first make clear the which means of the outcomes columns:

• The “rule” column and the “abs_coef” column are self-explanatory.
• The “sort” column has two distinctive values: “linear” and “rule”. The “linear” denotes the unique enter options, whereas “rule” denotes the “IF-THEN” circumstances generated from choice bushes.
• The “coef” column represents the coefficients produced by the Lasso regression evaluation. A optimistic worth signifies that if the rule applies, the log-odds of being categorized because the irregular class will increase. A bigger magnitude signifies a stronger affect of that rule on the prediction.
• The “assist” column data the fraction of information samples the place the rule applies.
• The “significance” column is calculated as absolutely the worth of the coefficient multiplied by the usual deviation of the binary (0 or 1) values that the rule takes on. So why this calculation? As we’ve got simply mentioned, a bigger absolute coefficient means a stronger direct impression on the log-odds. That’s clear. For the usual deviation time period, it successfully measures the “discriminative energy” of the foundations. For instance, if a rule is nearly all the time TRUE (very small customary deviation), it doesn’t break up your knowledge successfully. The identical holds if the rule is nearly all the time FALSE. In different phrases, the rule can not clarify a lot of the variation within the goal variable. Subsequently, the significance rating combines each the energy of the rule’s impression (coefficient magnitude) and the way nicely it discriminates between completely different samples (customary deviation).

For our particular case, we see just one high-impact rule (Rule #24):

If a flower’s petal is longer than 5.45 cm and wider than 2 cm, the percentages that LOF classifies it as “anomalous” enhance 85-fold. (Observe that exp(4.448999) ~= 85)

Guidelines #26 and #27 are nested inside Rule #24. That is widespread in follow, as RuleFit usually produces “households” of comparable guidelines as a result of they arrive from neighbouring tree splits. Subsequently, the one rule that really issues for characterizing the LOF-identified anomalies is Rule #24.

Additionally, we see that the assist for Rule #24 is 0.1667 (5/30). This successfully signifies that all 5 LOF-identified anomalies might be defined by this rule. We are able to see that extra clearly within the determine beneath:

There you could have it: the rule to explain the recognized anomalies!

3. Conclusion

On this weblog put up, we explored the RuleFit algorithm as a robust answer for explainable anomaly detection. We mentioned:

• The way it works: A two-step strategy the place choice bushes are first fitted to derive significant guidelines, adopted by a sparse linear regression to rank the rule significance.
• Tips on how to apply to anomaly clarification: Use the detection outcomes because the pseudo labels and use them because the “goal consequence” for the RuleFit mannequin.

With RuleFit in your modeling toolkit, the following time stakeholders ask “Why is that this anomaly?”, you’ll have concrete IF-THEN guidelines that they will perceive and act upon.

Reference

[1] Jerome H. Friedman, Bogdan E. Popescu, Predictive studying by way of rule ensembles, arXiv, 2008.

[2] Fisher, R. A., Iris [Data set]. UCI Machine Studying Repository, 1936.