Source
Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree
Anomaly detection finds points that deviate from the regular pattern in the data, the "is it weird behaviour?" question in Five Common Data Science Problems. The course uses a nearest neighbour method, the Local Outlier Factor (LOF).
Parameters
Two choices:
- : number of nearest neighbours to look at
- : expected fraction of anomalies (contamination), e.g. means about 5% are outliers, and 1% to 5% is a good range
Idea
A point is an anomaly if the density around it is much lower than the density around its neighbours. A point inside a crowd is normal, a point on its own is weird.
How It Works
For each data point:
- Find its nearest neighbours
- Compute the reachability distance to each neighbour, based on the distance to that neighbour's th nearest neighbour
- The local reachability density (LRD) is the inverse of the average reachability distance
- LOF is the average LRD of its neighbours divided by its own LRD
| LOF | Meaning |
|---|---|
| similar density to neighbours, normal | |
| denser than neighbours, inlier | |
| sparser than neighbours, possible outlier |
The fraction of points with the highest LOF are labelled anomalies.
More Variables
Using more dimensions (all six stats instead of only Total and Speed) makes detection more informative. A Pokemon that looks normal in every 2D scatter plot can still be an anomaly in the 6D space.
Code
from sklearn.neighbors import LocalOutlierFactor
X = pd.DataFrame(pkmndata[['Total', 'Speed']])
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.05)
labels = lof.fit_predict(X) # -1 = anomaly, +1 = normal
X_labeled = X.copy()
X_labeled['Anomaly'] = pd.Categorical(labels)
sb.scatterplot(x='Total', y='Speed', hue='Anomaly', data=X_labeled)