Source
Adapted from my own handwritten and hand-typed notes from Year 1 to Year 3 of my EEE degree
Binary classification predicts a True or False label, e.g. is a Pokemon Legendary, given Total. Box plots and swarm plots of Total split by Legendary show that legendary Pokemon mostly have high Total, but the two groups overlap, so a clear rule is needed.
Decision Tree
A decision tree partitions the data space using consecutive binary decisions (e.g. ?). Each node shows:
- the split rule used at that node
- gini: the Gini index of the node
- samples: number of training points reaching the node
- value : how many of those are False () and True ()
- class: the majority class, which is the prediction for any point ending in that node
Gini Index
The Gini index is the chance of misclassifying a random point in the node if it is labelled at random by the node's class mix. For False and True out of :
Properties:
- gini = 0 means a pure node (all one class)
- gini = 0.5 is the worst case for two classes, a 50:50 mix
- the tree picks the split that lowers the gini of the child nodes the most
Example: Depth 2 Tree on Total
| Node | Rule or leaf | samples | value [False, True] | gini | class |
|---|---|---|---|---|---|
| root | 600 | [557, 43] | 0.133 | False | |
| root True | leaf | 516 | [516, 0] | 0.0 | False |
| root False | 84 | [41, 43] | 0.5 | True | |
| inner True | leaf | 60 | [36, 24] | 0.48 | False |
| inner False | leaf | 24 | [5, 19] | 0.33 | True |
Check the root: .
Worked prediction: a Pokemon with fails , goes right, then passes , and lands in the leaf . The majority is False, so it is predicted not legendary, though gini 0.48 says this leaf is quite uncertain.
Multivariate Trees
Using more predictors (Total, HP, Attack, Defense) lets each node split on whichever variable works best, and deeper trees (two, three levels) give purer leaves. Too deep a tree can overfit the train data.
Code
from sklearn.tree import DecisionTreeClassifier, plot_tree
y = pd.DataFrame(pkmndata['Legendary'])
X = pd.DataFrame(pkmndata['Total'])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)
dectree = DecisionTreeClassifier(max_depth=2)
dectree.fit(X_train, y_train)
f = plt.figure(figsize=(12, 12))
plot_tree(dectree, filled=True, rounded=True,
feature_names=['Total'], class_names=['False', 'True'])How good the predictions are is judged with a confusion matrix.