Layer Normalization

Summary §

The paper introduces Layer Normalization, a method for normalization which standardizes each input across all of its features, instead of standardizing each feature across all inputs in a batch (batch normalization, BN). Unlike BN, there is no intra-batch dependency, no need to keep track of means and variances, and no changes in inference (so it works for online inference). This new method is also straightforward to apply to RNNs, unlike BN which needs different statistics per time step and layer.

Deep dive §

What is goal of the paper? §

• Why does it matter?
• What is the significance/impact of the conclusion?

What is the approach? §

• Is the approach well-motivated given existing literature?

Batch Normalization §

MLP case Assume $\text{X}\in R^{N,F}$ with $N$ being the batch size and $F$ being the number of features. Then, for each feature $i$ in $\{1,2,...,F\}$ :

1. ${\mu }_i=\frac{{\sum }_{n=1}^N{X}_{n,i}}{N}$
2. ${\sigma }_i^2={\sum }_{n=1}^N(X_{n,i}-{\mu }_i{)}^2$
3. ${\hat{\mathbf{x}}}_{\mathbf{i}}=\frac{{\mathbf{x}}_{\mathbf{i}}-{\mu }_i}{{\sigma }_i}$
4. ${\hat{\mathbf{x}}}_{\mathbf{i}}={\gamma }_i{\hat{\mathbf{x}}}_{\mathbf{i}}+{\beta }_i$

Extra Notes

• Highly dependant on batch size
• Hard to apply to RNNs
• Needs to keep track of running statistics for inference, uses momentum

Layer Normalization Standardizes each input example across all of its features.

• Does not depend on batch size
• Can be easily applied to RNNs
• Does not change during inference

Results §

• Are they correct?
• Are they rigorous?
• What else could be done?
• Datasets + experiments
• Evaluation metrics

Next steps §

• Where do we go from here?