Extreme Learning Machines (ELM)

ELMs are essentially shallow feedforward neural networks that feature a single hidden layer.

The weights between the input layer and the hidden layers are randomly initialized and not updated during training.

The weights between the hidden layer and the output layer form a beta-matrix.

• ELM aims to find the matrix $\beta$ such that the error between the actual outputs (given by $H\cdot B$) and the target outputs (given by $y$) is minimised
• This can be formulated as an SLE given by $H\cdot \beta =y$
• Since $H$ is typically not square and may not have an exact inverse, we can approximate $\beta$ with $\beta =(H^{+})\cdot y$, where $H^{+}$ is the Moore-Penrose Pseudoinverse of $H$
• In practice, $H^{+}$ can be calculated using techniques like SVD

Code

2 Libraries we can use:

skelm, which gives a scikit-learn style interface:

from skelm import ELMClassifier
 
clf = ELMClassifier()
 
# Train the model using the training sets
clf.fit(train_review_tfidf, train_sent)
 
# Predict the response for test dataset
y_pred = clf.predict(test_review_tfidf)

hpelm, where the network is built by hand:

from hpelm import ELM
 
elm = ELM(X_train.shape[1], 1)
elm.add_neurons(10, 'sigm')  # Add 10 sigmoid neurons
elm.train(X_train, y_train)
 
predictions = elm.predict(X_test).round().flatten()

Library	Import	Train	Predict
skelm	from skelm import ELMClassifier	clf.fit(X, y)	clf.predict(X)
hpelm	from hpelm import ELM	elm.train(X, y)	elm.predict(X)

• hpelm names the training method train rather than fit, which is different from the usual sckitlearn pattern
• ELM(X_train.shape[1], 1) sets the input size from the number of features and the output size to 1
• add_neurons(10, 'sigm') creates the single hidden layer, with 10 neurons and a sigmoid activation, and this is the value of $L$
• hpelm returns continuous values, so .round() converts them to class labels and .flatten() reshapes the column into a flat array

Why there is no epoch count

ELM has no backpropagation and no training loop, because the input weights stay at their random values and only the output weights are solved. Any option about learning rate or epochs belongs to a different model.

The pseudoinverse that solves for the beta matrix is calculated in practice with Singular Value Decomposition (SVD).