What Is Batch Normalization? Batch normalization explained

Batch normalization is a deep learning method that uses mini-batches to normalize layer inputs in order to accelerate, stabilize, and increase learning rates. At a glance, here's what you need to know about this important technique:

• Batch normalization rescales the output of the intermediate layer in a deep neural network's training stage, effectively reducing dependency on the initial weights.

• Batch normalization can be used for a variety of neural networks, such as convolutional neural networks, but may not be appropriate for every kind.

• Professionals who use batch normalization include data scientists, machine learning engineers, and deep learning researchers.

Below, you'll learn more about batch normalization, including how it works, its real-world implications, and the professionals who use it. Afterward, if you want to learn more about deep learning, consider enrolling in DeepLearning.AI's  Deep Learning Specialization.

What is batch normalization?

Batch normalization is a machine learning technique that can speed up deep learning training and contribute to the speed and stability of neural networks.

In 2015, Sergey Ioffe and Christian Szegedy revolutionized the field of deep learning with a normalization model that addressed a significant challenge in training neural networks: internal covariate shift. Internal covariate shift occurs as the network activations’ distribution changes depending on network parameter alterations during training. Batch normalization aims to solve this problem by normalizing each layer of the network activations rather than normalizing the total batch. 

Batch normalization speeds up training rates and improves accuracy in a deep neural network. It does this by inserting a hidden layer between two activation network layers. Batch normalization reduces dependency on the initial weights by re-centering and re-scaling the output of the intermediate layer in the training stage of a deep neural network. The network can then train on inputs with a consistent distribution and isn’t slowed down by constantly changing parameters. 

How batch normalization works

Batch normalization operates in three stages and repeats those stages for each batch in the training set. Explore them in more detail: 

1. First, the hidden batch normalization layer calculates the input layer’s current mean and standard deviation. 

2. The second step alters these values to a mean of zero and a unit standard deviation, which you calculate by subtracting the mean from each input and dividing by the standard deviation.

3. Forcing these parameters can often be too restrictive, so the third step in batch normalization is adding two learnable parameters: gamma and beta. Gamma is a scaling factor that can modify the standard deviation, and beta offsets bias by shifting the curve left or right. When a neural network requires a fluctuation in the input distribution, gamma and beta continuously learn each mini batch's optimal value and can learn to revert to the actual distributions.

Batch normalization implementation

Batch normalization can be used in deep learning framework, but it's essential to determine the best technique for your neural network. Multilayer perceptrons and convolutional neural networks are candidates for batch normalization; however, recurrent neural networks may be too complex. 

Another consideration is whether to apply batch normalization before or after an activation. It is common to apply batch normalization before the activation function; however, some distributions yield better results when batch normalization is applied afterward. Also, factoring in the larger-than-normal learning rates can help determine if batch normalization makes sense for the neural network. 

Batch normalization benefits and drawbacks

The benefits of batch normalization solidify it as a universally accepted tool for machine learning engineers when training deep neural networks. However, batch normalization also has limitations and challenges to consider when choosing a normalization method.

Batch normalization advantages

• Accelerated training process: Batch normalization speeds up the training process, resulting in higher learning rates than models trained without a normalization tool. 

• Improved stability and performance: By reducing internal covariate shifts, the model is more stable and accurate in the training stage.

• Reduced dependence on initial parameters: Normalizing each batch rather than the entire training set makes the network less dependent on initial starting weights. The model continuously learns the ideal mean and standard deviation, increasing flexibility as it trains.

Batch normalization disadvantages

• Increases the complexity of the model: Adding additional layers to the training model makes it larger and more complex, which can increase operational costs.

• Requires large batch sizes: Batch normalization on small or varied batch sizes can incorrectly estimate the mean and deviation, creating an unstable training model.

• Not universally beneficial: Batch normalization works well with convolutional neural networks but can create detrimental results in training sequence data sets, such as recurrent neural networks.

Who uses batch normalization?

Jobs that create and interpret machine learning algorithms and models require understanding normalization techniques like batch normalization. Machine learning has applications in nearly all industries, with uses from automated order-taking (AOT) at drive-thrus to lung cancer screenings and self-driving cars. 

Below are three careers that work with machine learning and batch normalization.

1. Data scientist

Total median US pay (Glassdoor): $153,000 [1]

As a data scientist, your company or clients may look to you for recommendations on business decisions and processes based on your understanding of data. After collecting raw data, you can create algorithms and models and use machine learning to categorize data to make predictions and interpret the data.

2. Machine learning engineer

Average annual US salary (Glassdoor): $159,000 [2]

As a machine learning engineer, you often bridge the gap between software engineers and data scientists. You may develop software that automates predictive models and works with data scientists to input their data into machine learning algorithms. 

3. Deep learning model researcher

Average annual US salary (Glassdoor): $151,000 [3]

As a deep learning model researcher, your work may be similar to that of a machine learning engineer. Working with machine learning models, you would focus on developing models that let the machine train, teach, and think for itself. 

How to begin with batch normalization

If you want to have a career in deep learning, you will likely need to know batch normalization. You’ll likely need a bachelor's degree in mathematics, computer science, or statistics to enter the machine learning field. Many employers look for candidates with a bachelor's degree and relevant work experience, yet some require a master’s or doctoral degree. 

Learn more about deep learning on Coursera

Deep learning is critical to countless organizations around the world. Learn more about machine learning, deep learning, and AI with these resources on Coursera:

• Watch: Machine Learning Decision Trees

• Read more: AI and Machine Learning articles

• Explore: Popular Machine Learning Courses and Certifications

With Coursera Plus, you can learn and earn credentials at your own pace from over 350 leading companies and universities. With a monthly or annual subscription, you’ll gain access to over 10,000 programs—just check the course page to confirm your selection is included. 

订阅 Coursera Plus，培养就业技能

开始为期 7 天的免费试用

• 
• 
• 
• 

开始为期 7 天的免费试用