Source

Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree

Anomaly detection finds points that deviate from the regular pattern in the data, the "is it weird behaviour?" question in Five Common Data Science Problems. The course uses a nearest neighbour method, the Local Outlier Factor (LOF).

Parameters

Two choices:

  • : number of nearest neighbours to look at
  • : expected fraction of anomalies (contamination), e.g. means about 5% are outliers, and 1% to 5% is a good range

Idea

A point is an anomaly if the density around it is much lower than the density around its neighbours. A point inside a crowd is normal, a point on its own is weird.

How It Works

For each data point:

  1. Find its nearest neighbours
  2. Compute the reachability distance to each neighbour, based on the distance to that neighbour's th nearest neighbour
  3. The local reachability density (LRD) is the inverse of the average reachability distance
  4. LOF is the average LRD of its neighbours divided by its own LRD
LOFMeaning
similar density to neighbours, normal
denser than neighbours, inlier
sparser than neighbours, possible outlier

The fraction of points with the highest LOF are labelled anomalies.

More Variables

Using more dimensions (all six stats instead of only Total and Speed) makes detection more informative. A Pokemon that looks normal in every 2D scatter plot can still be an anomaly in the 6D space.

Code

from sklearn.neighbors import LocalOutlierFactor
 
X = pd.DataFrame(pkmndata[['Total', 'Speed']])
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.05)
labels = lof.fit_predict(X)      # -1 = anomaly, +1 = normal
 
X_labeled = X.copy()
X_labeled['Anomaly'] = pd.Categorical(labels)
sb.scatterplot(x='Total', y='Speed', hue='Anomaly', data=X_labeled)