Applied Edge AI: Deep Learning Outside of the Cloud

1. 00:00Hello and welcome! In this video we will continue our topic on convolutional neural network architectures.
2. 00:08We will review Google's Inception networks and look at the batch normalization layer.
3. 00:16The convolutional neural network tries to learn the future in the 3D space with two spatial dimensions - width and height and
4. 00:25one channel dimension.
5. 00:27In this example the input image has three channels. And the convolutional Kernel also has three channels.
6. 00:35The task of a single convolution kernel is to learn the channel correlation and the spatial correlation at
7. 00:45the same time.
8. 00:48Inception modules assumption is that channel correlation and spatial correlation has been fully decoupled.
9. 00:57Therefore bottleneck layer can be used to reduce the number of channels.
10. 01:02On the other hand, it is into relatively more challenging to learn the special correlation and the channel correlation of
11. 01:11the filter simultaneously.
12. 01:12So if we try to learn two different things it will be more complicated than that.
13. 01:19We decouple them
14. 01:24and to try to learn it one by one. Thus the Inception module explicitly decomposes this process into four parallel computing
15. 01:35branches that independently consider channel correlation and spatial correlation, making the learning process easier and
16. 01:44more effective.
17. 01:47This concept also influenced the design of a theory of lightweight deep learning models and that also came from Google
18. 01:55Researchers later.
19. 02:00After GoogleNet which is sometimes also called InceptionNet V1, researchers from Google then proposed a new network
20. 02:08called BN-inceptionNet.
21. 02:11BM means batch normalization. In this model, a simple but efficient method is used as I said, it's called batch normalization.
22. 02:21Let's take a look first of all, when we are going to train a machine learning model, we will normally normalize the input
23. 02:29data.
24. 02:30Why do we do that?
25. 02:32Let's take a simple example, predicting a price of a house based on two features - the size and the square meter with a range
26. 02:44from 0 to 2000.
27. 02:46And other feature is the number of bedrooms which normally varies from 1 to 5. Without normalization, regarding the contour
28. 02:57of the cost function
29. 02:59J, in our case, with respect to the weight parameter θ, you will get a long and narrow, very narrow ellipses, which leads
30. 03:10to a zigzag routine in the direction of the gradient perpendicular to the contour when the gradient drops.
31. 03:19So this view makes the optimization process very slow.
32. 03:24On the other hand, if we normalize the data,
33. 03:32so if we normalize the feature data into the same value range, for example, between 0 and 1, you will get a more balanced
34. 03:41contours of the cost function and the gradient descent will be much faster, normalizing input data can speed up the convergence
35. 03:52normally.
36. 03:53So now input normalization is actually the standard processing step of a machine learning application and
37. 04:04it can also improve the accuracy, especially when it comes to some distance calculation algorithms.
38. 04:11The effect is significant.
39. 04:14Therefore normalizing is necessary.
40. 04:16It can make each feature contribute the same roughly the same to the result.
41. 04:26Thus the core idea of batch normalization is to normalize the input for each hidden layer, not just the input of the network
42. 04:34but for each hidden layer in the deep neural network.
43. 04:38This can help to mitigate interdependencies between distributions and hidden layers.
44. 04:46We can see that BN will always always create data distribution with zero mean and unit variance.
45. 04:55The effect in practice is that we can train the model faster and the convergence process is more stable.
46. 05:05This slide shows some comparison results. On the left hand side, the author shows several convergence curves on the ImageNet
47. 05:15classification task.
48. 05:17We can see that the black dotted line in the vanilla InceptionNet which trained with a very small initial learning rate of
49. 05:260.0015, very small number.
50. 05:31All the batch norm related methods
51. 05:34add a BN layer before the input of the non-linear activation function of each hidden layer and their inputs
52. 05:42are better than InceptionNet.
53. 05:45Their results are better than the InceptionNet.
54. 05:48And the batch norm baseline uses the same initial learning rate. And BNx5 and BNx30 use 5x and 30x larger
55. 06:00initial learning rate respectively.
56. 06:03BNx5 Sigmoid, instead ReLU,
57. 06:05it uses the sigmoid activation function but still without any convergence problem. On the right hand side, regarding the classification
58. 06:15accuracy, BN with larger initial learning rate regularly improve the accuracy and they converge much faster.
59. 06:26So here, I just want to give you an overview of how to implement a batch norm layer in the forward propagation.
60. 06:35In short, during the training we will per each day to batch to the following operations.
61. 06:42First we will calculate the mean and variance channel-wise across the batch and then we will subtract the mean and divide by
62. 06:50the standard deviation.
63. 06:52Finally, we will use two additional learnable parameters scale and shift the input.
64. 07:00During the testing,
65. 07:02We just use the fixed empirical mean and variance parameters during the training, for instance, they can be estimated by running
66. 07:10average.
67. 07:15BN relies on the batch, the 1st and 2nd statistical moments which are mean and variance to normalize hidden layer
68. 07:25activations, so the output values are then strongly tied to the current batch statistics, such transformation adds some
69. 07:36noise depending on the input examples used in the current batch. This behavior can be considered as a sort of regularization
70. 07:45effect.
71. 07:47So in the practice with batch norm we can make the training faster and more accurate. Batch normalization significantly improve
72. 07:56the training stability and reduces the strong dependence on good initialization.
73. 08:04So previously we have to pay attention to a initialization function.
74. 08:08Two different machine learning applications but now with the BN layer.
75. 08:13So actually we can easily achieve a similar result without choosing a numbers of initialization method and batch norm offers
76. 08:24regularization effects and to some extent to replace the use of dropout.
77. 08:34We also try to understand why BN works. Here to discuss with you the current three different hypothesis.
78. 08:43According to the explanation in the original batch norm paper, BN effectiveness is due to the reduction of the internal
79. 08:53covariate shift.
80. 08:56Covariate shift describes the shifting of a model's input distribution. By extension,
81. 09:03the internal covariate shift described this phenomenon when it happens in the hidden layers of a deep neural network.
82. 09:11The corresponding correction for those shifted intermediate distributions require more training steps as you can see the
83. 09:19back propagation adapt the ways to achieve the distribution adaption.
84. 09:26So if there is a huge variant shift in the input signal, the optimizer will have trouble to generalize.
85. 09:35While in contrast, if the input signal always follow the standard non distribution, the optimizer might be able to easily
86. 09:44generalize so, following the author's aspect, that is forcing to have 𝜇 = 0, σ = 1, and adding two trainable
87. 09:54parameter, gamma and beta to suggest the distribution.
88. 09:59This will help the network generalization.
89. 10:05The second hypothesis is normalizing the intermediate activations, reduces the interdependency between hidden layers.
90. 10:15This sound somehow similar to the previous hypothesis but it actually describes the phenomenon from the quite a different
91. 10:23perspective, the purpose of normalization is to reduce the interdependency between layers, which focus on the distribution
92. 10:32stability.
93. 10:34So the optimizer could choose the optimal distribution by adjusting only two parameters, gamma and beta.
94. 10:44As already mentioned in the previous session.
95. 10:46If all gradients are large and the gradient of B1 will be very large.
96. 10:52On the contrary, if all gradients are small, the gradient of B1 will be negligible.
97. 10:59So we can quickly figure out that hidden units are pretty dependent on each other. On modification of weight,
98. 11:09W1 will modify the input distribution of the neuron
99. 11:13node W2 eventually modifying subsequent neural nodes input signal
100. 11:20sequentially. If we want to adjust the input distribution of a specific hidden unit, we need to consider the whole sequence
101. 11:29of layers.
102. 11:31However batch normalization regulates that using just two parameters gamma and beta.
103. 11:38It is no longer necessary to consider all parameters to have clues about distribution inside the hidden units which significantly
104. 11:48is the training.
105. 11:53In this paper from machine learning conference neurIPS 2018 empirically demonstrated that batch norm effectiveness is likely
106. 12:02not related to the internal covariate shift against the original argument from the original authors.
107. 12:11So the third hypothesis here is that batch norm make the optimization landscape smoother.
108. 12:19The benefits of BN are due to this motion effect. In the left most figure,
109. 12:26The authors compares VGGNet
110. 12:28trained without and with BatchNorm, and with explicit "covariate shift" being added to BN layer, referring
111. 12:39to as noisy batch norm in the figure.
112. 12:43In the latter case, the author introduced distributional instability by adding time varying non zero mean and a non unit
113. 12:52variance noise independently to each batch normalization activation.
114. 12:58The figures show that noisy batch norm model nearly matches the performance of the standard BN model.
115. 13:06Despite substantial internal covariate shifts of the distributions and this finding significantly changes the hypothesis
116. 13:16from the original batch norm paper and the author showed that BN makes the optimization landscapes smoother while preserving
117. 13:25all the local minimum of the normal landscape.
118. 13:30So they realized the optimization landscape and observe such phenomenon.
119. 13:36In other words, BN re-parametrizes the optimization process which achieved the training faster and easier.
120. 13:45Furthermore, they also observe similar training performance using L1 and L2 normalization as well.
121. 13:53Thus, the authors speculate that BatchNorm makes the optimization landscape converge on more flat minima, which should have
122. 14:04better generalization ability.
123. 14:06We can find this effect in the right most figure.
124. 14:12As a summary,
125. 14:13however, please keep in mind all of those hypothesis presented in this session are mostly speculations
126. 14:22and the discussion are helpful for building your intuition regarding the effectiveness of BN, but
127. 14:32we still don't know exactly why batch norm is so efficient in practice.
128. 14:38So let's wait.
129. 14:40So let's wait patiently for the new research results to help ours reveal the mystery of batch normalization layer.

To enable the transcript, please select a language in the video player settings menu.