Neurons

One Neuron

This is most fundamental state of a neural network. A single neuron takes in an input, multiplies each by a learned weight, sums them, and passes the result through an activation function. The operation is simply dot product.

Given:

• Input: x = [2, 3, 1]
• Weights: w = [0.5, -0.4, 0.8]

The neuron computes:

$$z=(2\times 0.5)+(3\times -0.4)+(1\times 0.8)=0.6$$

Fully-Connected (FC) Layer

A FC layer is a collection of neurons in parallel, all receiving the same input but each with their own weights, producing one output per neuron:

Input x₁ ──┬──► Neuron 1 ──► output 1
Input x₂ ──┼──► Neuron 2 ──► output 2
Input x₃ ──┴──► Neuron 3 ──► output 3
           └──► Neuron 4 ──► output 4

In matrix form:

$$\mathbf{h}=\text{activation}(W\cdot \mathbf{x}+\mathbf{b})$$

Where W is a weight matrix of shape [outputs × inputs] and b is a bias vector.

Fully-Connected Network

A FC network is many FC layers stacked together in sequence.

Input → [FC Layer 1 + ReLU] → [FC Layer 2 + ReLU] → [FC Layer 3 + Sigmoid] → Output

Each layer builds higher-level abstractions on top of the previous layer's output.

---

Activation Functions

Stacking multiple FC layers without Activation Functions collapses them to a single linear transformation:

$$W_2(W_1\mathbf{x})=(W_2{W}_1)\mathbf{x}=W_{\text{combined}}\mathbf{x}$$

• Two FC layers without activation results in one layer
• this is because matrix multiplication is associative
• Activations like ReLU introduce non-linearity, allowing the network to learn complex, curved decision boundaries
• with ReLU, when we zero out anything that is negative, we are already breaking that linearity

Common choices:

Activation	Output range	Typical use
Sigmoid	Between zero and one	Binary probabilities and recurrent gates
Tanh	Between minus one and one	Recurrent hidden states and candidate content
ReLU	Zero or positive values	Hidden features in many feed-forward networks

---

Code

One neuron with known weights

import numpy as np
 
x = np.array([2.0, 3.0, 1.0])
w = np.array([0.5, -0.4, 0.8])
bias = 0.0
score = x @ w + bias
output = max(0.0, score)
print(round(score, 2))   # 0.6
print(round(output, 2))  # 0.6

@ calculates the dot product. Adding bias shifts the score. max(0.0, score) applies ReLU. The positive score passes through unchanged.

A layer contains several neurons

import torch
layer = torch.nn.Linear(3, 4)
X = torch.tensor([[2.0, 3.0, 1.0]])
print(tuple(layer.weight.shape))  # (4, 3)
print(tuple(layer(X).shape))      # (1, 4)

Read Linear as values in, values out

Think of nn.Linear(12, 4) as 12 values in, 4 values out.
PyTorch stores its weight matrix as (4, 12): 4 output neurons, each with 12 weights. With the default bias=True, the 4 biases are stored separately with shape (4,).
nn.Linear(in_features, out_features) changes only the last input dimension. For example, (3, 5, 12) becomes (3, 5, 4). The other dimensions stay unchanged.

• here nn.Linear() creates 4 random biases for the 4 output neurons, where 3 is the number of input features per sample
• therefore, each output neuron has 3 weights and 1 bias

import tensorflow as tf
layer = tf.keras.layers.Dense(4)
X = tf.constant([[2.0, 3.0, 1.0]])
output = layer(X)
print(tuple(layer.kernel.shape))  # (3, 4) which is weight matrix
print(tuple(output.shape))        # (1, 4)

Both layers accept three input features and produce four output scores per sample. PyTorch stores the weight matrix as (outputs, inputs). Keras stores its kernel as (inputs, outputs). Both examples use randomly initialised weights.