Applied Edge AI: Deep Learning Outside of the Cloud

1. 00:00Hello and welcome! In this video,
2. 00:03we will introduce Transformer model which first had a significant influence in the field of natural language processing and
3. 00:12then began to influence computer vision.
4. 00:16Because the attention mechanism plays an essential role in the transformer model structure.
5. 00:22Let's first learn the attention mechanism and the difference between the visual task and NLP task. In the year 2015, the
6. 00:34"show, attend and tell" paper maybe the first work that proposed the attention mechanism in the context of computer vision
7. 00:44it is more like the mechanism of the human retina.
8. 00:48There will be a focus in the visual image, where the focus is the clearest, and the surroundings are blurred.
9. 01:00The visual attention tries to imitate such a mechanism.
10. 01:04Find the focal area in the image and find the connection between those focal points and the natural language.
11. 01:13The algorithm tries to understand the semantic meaning of a sentence and uses the key words to be aligned with the visual
12. 01:22focus in the corresponding image.
13. 01:28The attention mechanism in NLP domain describes the correlations between tokens of sentences.
14. 01:36It is related to how to define query, key, and value vectors in the NLP problems.
15. 01:44For example, on the reading comprehension problem,
16. 01:48the query can be a representation of a question or a representation of a question, combine it with an option and for key
17. 01:57and value they are often the same and they all refer to the context information.
18. 02:03So for reading comprehension problem, the context is the article. For this case,
19. 02:10The purpose of attention is to find out the relevant fragments in the context, which is for example an article to the given
20. 02:20question to estimate the best answer.
21. 02:26Using this attention scores, you can get a weighted representation and then put it into a feed forward neural network to
22. 02:34get a new representation which takes into a current contextual information.
23. 02:41Another example here is a machine translation.
24. 02:45The input sentence will be processed by a neural network and to predict each output token,
25. 02:52it will not only get the information of the corresponding input token but also consider the context information from the
26. 03:00surrounding tokens but not all the input token will be equally considerate for certain output token and you will learn
27. 03:10a softmax attention distribution and to re-weight the correlation between an output token and several different input tokens.
28. 03:20Here, for example, the tokens located far away may be relatively less relevant.
29. 03:30Self attention for short will capture the correlation of a given tokens to all the other tokens in the same sentence.
30. 03:40For example, the token "it" here refers to "animal" or the "street".
31. 03:46This requires us to reach the context when we see the state is "tired" we should know that it refers to the "animal" with the
32. 03:57higher probability.
33. 03:59In the recurrent neural network, we need to process all the token step by step and when they are far apart, the
34. 04:08effect of recurrent neural network is often poor and its sequential processing efficiency is also very low and self attention
35. 04:18uses the attention mechanism to calculate the association between each token and all the other token in the sentence.
36. 04:26In the first sentence, the word animal has the highest attention score for the token
37. 04:32"it". On the contrary, adjective from "tired" changed to "wide" in the second sentence and token with the highest attention
38. 04:45score also changed from animal to the street.
39. 04:51So therefore we can see that the model trained by self attention can capture this context information very well and the
40. 05:00efficiency will not be drastically reduced
41. 05:03once the sentence became very long.
42. 05:09Transformer is essentially an encoder decoder structure and the encoder is composed of six encoding blocks and the decoder
43. 05:18is also composed of six decoding blocks.
44. 05:21Like all generative models, the output of the encoder will be used as the input of the decoder.
45. 05:29The encoder block consists of multi head self attention module and feed forward network layer.
46. 05:37Multi head attention is the combination of multiple self attention structures.
47. 05:43Each head learns features in different representation spaces.
48. 05:47As shown in the figure, the focus of attention learned by the two heads may be slightly different, giving the model more
49. 05:57capacity.
50. 05:58The difference between the decoder and the encoder block is that the decoder has one more connection attention
51. 06:07module between the encoder and the decoder.
52. 06:10The functional difference between these two different attention blocks are following.
53. 06:16In machine translation task self attention captures the relationship between the current translation and the previous text
54. 06:25that has been translated. The encoder decoder attention focused on the relationship between the current translation and the
55. 06:33encoded future vector.
56. 06:39We already introduced the three components for many NLP tasks.
57. 06:44They are query, key, and value.
58. 06:46How are they arranged in transformer?
59. 06:49Query, Key, and Value vectors are computed from the same word with the same length and in the self attention module. They are
60. 06:58obtained by multiplying the embedding vector by three different weight matrices. And the dimensions of three matrices are the
61. 07:07same.
62. 07:09The computation is according to the equation and softmax attention will be further multiplied by the value vector
63. 07:18and the output of multi head attention module will be fed into a classic feed forward network layer. Multi-head attention
64. 07:27may learn different features for each head and which can further enhance the expression ability of the model.
65. 07:37But it also introduced some more computation overhead.
66. 07:41So we need to make a trade off of accuracy and efficiency here.
67. 07:49As aforementioned, the convolutional neural network have two important inductive bias. The locality,
68. 07:57it has a locally restricted receptive field.
69. 08:01This means that the linear convolution filter can only see the neighbor values.
70. 08:07The second is weight sharing across the whole image, which makes the convolution filter translation invariance. Here invariants
71. 08:17means that you can recognize an object in an image regardless of the specific positions.
72. 08:25CNN works pretty well when the object's appearance change.
73. 08:29On the other hand, the transformer is by design permutation in variant transformer is for sequential data and the missing
74. 08:39position information of visual objects.
75. 08:44So if we want to also apply transformer architecture on image, we need to make some adaption on its structure.
76. 08:54Recently, the author of transformer further proposed vision transformer which reformulated the image classification problem
77. 09:03at the sequential problem using image patches as word tokens.
78. 09:08Let's take a look how it works.
79. 09:11First it splits an image into patches and then flattened the patches, create the linear embedding of the patches and then
80. 09:22define and add a positional embedding information on top of the linear embedding.
81. 09:28Note that the extra learnable class embedding are also utilized here.
82. 09:34It is fed into the sequence at the position zero.
83. 09:38Feed the sequential input into a standard transformer encoders.
84. 09:42And get the classification result from a MLP-Head
85. 09:46on top of the transformer encoders.
86. 09:52So with some simple adaption modification, transformer can also be used for image classification task.
87. 10:04They also claim that this structure does not directly produce impressive results on datasets such as CIFAR and ImageNet.
88. 10:12But if pre training is done on a larger data set, the situation will fundamentally change while larger ViT
89. 10:22model perform worse than BiT ResNets baseline.
90. 10:28And the pre-trained on the small data set.
91. 10:31It performs better and even much better when pre training on the larger dataset.
92. 10:37Similarly, larger ViT
93. 10:39variants overtake similar ones as the dataset grows. In the figure on the right hand side,
94. 10:47We can see that ViTs
95. 10:49outperform ResNet across the FLOPs landscapes.
96. 10:57ConvNets like AlexNet contains two separate stream of processing.
97. 11:02An apparent consequence of this architecture is that one stream developed the high frequency grayscale features and the
98. 11:11other low frequency color features.
99. 11:15The realization of the first linear embedding filters of visual transformer shows that early layer represents may share similar
100. 11:25features as ConvNets.
101. 11:26It demonstrates very well learned smooth filters and also compute attention distance at the average distance between
102. 11:37the query pixel and the rest of the patch, multiplied by attention weights. And they use 128 example images and average their
103. 11:48results.
104. 11:50So it can be seen that the area of inherent interest obtained in each photo is roughly consistent with the contour shape
105. 12:01of the object, indicating that the learned attention has a reasonable semantic meaning and the relatively higher interpretability.
106. 12:14Okay, in the practical session of this week, we will learn how to implement an image classification model using PyTorch.
107. 12:23The detailed task description can be found in the next learning unit and the time required to complete the practical task
108. 12:32is about 3-6 hours.
109. 12:34I wish you all have fun and have great success.
110. 12:41Thank you

To enable the transcript, please select a language in the video player settings menu.