Activation Functions

Activation functions introduce nonlinearity into neural networks. Without them, any depth of linear layers collapses to a single linear transformation. The choice of activation affects gradient flow, training speed, and representational capacity.

Requirements for a Good Activation

  • Nonlinear: otherwise stacking layers is useless.
  • Differentiable (almost everywhere): required for gradient-based training.
  • Non-saturating (ideally): saturating activations kill gradients in deep networks.
  • Computationally cheap: called billions of times per training step.

Sigmoid

$$ \sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1) $$

Derivative: $\sigma’(z) = \sigma(z)(1 - \sigma(z))$

Maximum derivative: $0.25$ at $z = 0$.

Problems:

  • Saturates at both tails: $\sigma’(z) \approx 0$ for $\lvert z \rvert \gg 0$. Gradients vanish in deep networks.
  • Output not zero-centered: all outputs positive; leads to zig-zagging gradient updates.
  • Expensive: requires exponential.

Use: output layer for binary classification; gating mechanisms (LSTM, GRU).

Tanh

$$ \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} = 2\sigma(2z) - 1 \in (-1, 1) $$

Derivative: $\tanh’(z) = 1 - \tanh^2(z)$

Maximum derivative: $1.0$ at $z = 0$.

Advantages over sigmoid: zero-centered output. Stronger gradient at origin.

Problems: still saturates at tails; vanishing gradients in deep networks.

Use: hidden layers in RNNs and shallow networks. Largely replaced by ReLU in feedforward layers.

ReLU (Rectified Linear Unit)

$$ \text{ReLU}(z) = \max(0, z) $$

Derivative:

$$ \text{ReLU}'(z) = \begin{cases} 1 & z > 0 \\ 0 & z < 0 \end{cases} $$

Advantages:

  • Non-saturating for $z > 0$: gradients pass through unchanged.
  • Computationally trivial (comparison + threshold).
  • Induces sparsity: about 50% of neurons are inactive on average.
  • Empirically accelerates convergence compared to sigmoid/tanh.

Problems:

  • Dying ReLU: if a neuron’s pre-activation is always negative (e.g., after a large negative gradient update), it outputs 0 permanently and receives zero gradient. Learning stops for that neuron.
  • Not zero-centered.
  • Not differentiable at $z = 0$ (subgradient ${0}$ used in practice).

Use: default activation for hidden layers in MLPs and CNNs.

Leaky ReLU

$$ \text{LeakyReLU}(z) = \begin{cases} z & z > 0 \\ \alpha z & z \leq 0 \end{cases} $$

Typically $\alpha = 0.01$. Fixes dying ReLU by allowing a small gradient for $z < 0$.

PReLU (Parametric ReLU)

Same as Leaky ReLU but $\alpha$ is a learned parameter. Adds minimal overhead; can improve accuracy on large datasets.

ELU (Exponential Linear Unit)

$$ \text{ELU}(z) = \begin{cases} z & z > 0 \\ \alpha(e^z - 1) & z \leq 0 \end{cases} $$
  • Smooth, differentiable everywhere.
  • Negative outputs bring mean activations closer to zero (reduces bias shift).
  • Slower to compute due to exponential.

SELU (Scaled ELU)

$$ \text{SELU}(z) = \lambda \cdot \text{ELU}(z) $$

With specific $\lambda \approx 1.0507$ and $\alpha \approx 1.6733$. Self-normalizing: activations converge to zero mean and unit variance under mild conditions, making batch normalization unnecessary. Requires LeCun Normal initialization and fully connected architectures.

GELU (Gaussian Error Linear Unit)

$$ \text{GELU}(z) = z \cdot \Phi(z) = z \cdot \frac{1}{2}\left[1 + \text{erf}\!\left(\frac{z}{\sqrt{2}}\right)\right] $$

where $\Phi$ is the standard normal CDF. Approximated as:

$$ \text{GELU}(z) \approx 0.5z\left(1 + \tanh\!\left(\sqrt{2/\pi}(z + 0.044715z^3)\right)\right) $$
  • Smooth, non-monotone.
  • Stochastic interpretation: $\text{GELU}(z) = z \cdot P(Z \leq z)$ for $Z \sim \mathcal{N}(0,1)$.
  • Default activation in BERT, GPT, and most modern Transformers.

SiLU / Swish

$$ \text{SiLU}(z) = z \cdot \sigma(z) $$

Smooth, non-monotone, unbounded above. Performs well in EfficientNet and many vision models. Close to GELU.

Softmax

$$ \text{softmax}(\mathbf{z})_k = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}} $$
  • Outputs sum to 1; interpreted as probabilities.
  • Numerically stable version: subtract $\max(\mathbf{z})$ before exponentiating.
  • Not an activation for hidden layers; used only in the output layer for multi-class classification.

Temperature scaling: $\text{softmax}(\mathbf{z} / T)$. Low $T \to 0$ sharpens to argmax; high $T \to \infty$ flattens to uniform. Used in knowledge distillation and language model decoding.

Comparison

Activation Range Saturates Zero-centered Dying neurons Typical Use
Sigmoid $(0,1)$ Yes (both) No No Output (binary), gates
Tanh $(-1,1)$ Yes (both) Yes No RNNs, shallow nets
ReLU $[0, \infty)$ No (pos) No Yes MLPs, CNNs
Leaky ReLU $\mathbb{R}$ No No No MLPs, CNNs
ELU $(-\alpha, \infty)$ No Near-zero mean No Deep MLPs
GELU $\mathbb{R}$ No No No Transformers
Swish/SiLU $\mathbb{R}$ No No No Vision models
Softmax $(0,1)^K$ Yes No No Output (multi-class)