Applied Edge AI: Deep Learning Outside of the Cloud

1. 00:00Hello and welcome. In the last video I mentioned it that an essential difficulty in Federated learning is how to minimize
2. 00:10the number of communications, even if doing so will increase the computation.
3. 00:16The need for overall model optimization is worth considering.
4. 00:21For example, the original Federated learning to train a model requests 1000 iterations.
5. 00:28A better algorithm makes it SGD
6. 00:31of the central server converge faster by generating a better gradient locally, which only needs for instance 100 iterations.
7. 00:42So the communication is saved 10 times, which is the goal of the current algorithm design the local computation is performed
8. 00:52when the device such as a mobile phone is charging so it will not cause too much inconvenience to the user.
9. 01:01Next I would like to introduce to you the Federated Average method first time proposed by google.
10. 01:11Let's shortly recap how the traditional distributed learning method works first at the edge.
11. 01:18Note, it receives the up to date model ways from the central server.
12. 01:25Then it uses the local data and obtain the ways to compute the gradients.
13. 01:31It further passes the gradient to the central server
14. 01:35for a global update.
15. 01:38The central server receives gradient from all the edge nodes, then sum them up and updates the weights according to the formula
16. 01:48actually learned before theta represents the weight parameters.
17. 01:53Theta is the learning rate.
18. 01:55So after its weight updates we just finished the current iteration, it may require many iterations to get the model to converge.
19. 02:06Now let's take a look at the Federated Average method.
20. 02:11Federated Average is a communication efficient algorithm requiring fewer iterations to make the model converge.
21. 02:20First the edge node worker will also get the ways from the central server.
22. 02:26Then it will repeat the following iterations at first use the local data and receive the ways to compute the gradient, then
23. 02:36use the gradient to update the ways we call this step
24. 02:40the local update.
25. 02:42It will generally continue to update for several epochs and each epoch needs to traverse all the local data.
26. 02:50Google's paper suggests that local updates repeat 1 - 5 times.
27. 02:56There's always this way the local weight parameters are already different from received from the central server.
28. 03:05After the local updates,
29. 03:06the new weight parameters ascend to the server.
30. 03:10Central server know that what is sent here is no longer the gradient.
31. 03:17Why do we do that?
32. 03:18Because doing so we can make more weights changed in one communication, not just one time gradient descent. After the central
33. 03:27server received the ways of all edge nodes, it does not need to perform gradient descent.
34. 03:34Still, it performs a simple calculation or weight average
35. 03:39to obtain the new model weights.
36. 03:41Now we can see why the name of the algorithm is called
37. 03:45Federated averaging?
38. 03:49The paper reports.
39. 03:51The following experimental comparison results of SGD
40. 03:55and Federated averaging
41. 03:59the horizontal axis in the number of communications and the vertical axis
42. 04:04they know the classification accuracy.
43. 04:07You can see that fit average converge faster and has higher accuracy with the same communication overhead.
44. 04:15This is exactly the goal of federated average, which is no to achieve a faster convergence rate and fewer communication times
45. 04:26between two connections. Federal average allows the edge.
46. 04:31nodes, do a lot of local computation in exchange for more efficient communication.
47. 04:39So suppose we now require federal it average and SGD
48. 04:43to do the same amount of computation at the edge
49. 04:46node, then the convergence speed of Federated average will be slower than that of SGD.
50. 04:54We know that in the scenario of Federated Learning the computational cost is relatively small while the communication cost
51. 05:03is high.
52. 05:05So Federated average algorithm is very practical.
53. 05:09Another point from the figure is that with Federated Average one can use higher learning rate.
54. 05:16So the result of the learning rates equals to 0.25 is better than that of 0.05.
55. 05:24On the contrary, SGD
56. 05:25with higher learning rates results in worse learning curves.
57. 05:35According to the previous example, we can find that the communication content between age nodes and central server in Federated
58. 05:44Learning is mainly the models ways
59. 05:47and the gradient information, which is computed at the edge
60. 05:52node. User data does not leave the local device, but the question is, are weight and gradient information secure enough?
61. 06:02Let's take a look at the formula of gradient computation, we know that gradient is a derivative of the loss function with
62. 06:12respect to a weight parameter. Here X,
63. 06:16is a sample,.
64. 06:17and Y represents a label. The mathematical transformation of local data obtains the gradients.
65. 06:25Therefore, gradient carriers, certain information from the training data.
66. 06:32So if the gradient contains the information of training data, we can reverse engineer the data from the gradient somehow.
67. 06:40Unfortunately, the foreign literature shows that the training data information can be severelly leaked through the model ways
68. 06:50and gradients. In another paper, the authors devise a new set of attacks to compromising influence data privacy in collaborative
69. 07:03AI system when deep network model and the corresponding inference task are split and distributed to different nodes.
70. 07:13So one malicious participants can accurately recover and arbitrary input fed into this system, even if he has no access
71. 07:24to another participants data or computations. it can also recover the prediction APIs
72. 07:31from querying this system.
73. 07:37This paper, demonstrates that Federated learnings update leak unintended information about participants training data.
74. 07:46And it develops several inference attacks to exploit this leakage.
75. 07:52It shows that adversarial participants can infer the presence of exact data points.
76. 08:00For example, the specific locations in another training data.
77. 08:05In other words the membership inflerence. Second,
78. 08:10the authors show how the adversary can infer properties that hold only for a subset.
79. 08:17For example, the adversary can infer when a specific person first appears in the photos used to train the binary gender classifier,
80. 08:28in this way by using the gradient information as features to train the binary classifier
81. 08:34one can further in for other properties like age, race, disease, etcetera.
82. 08:41All very sensitive personal data, personal information that must be protected.
83. 08:48So, there are many ways to obtain data information through model ways or gradient information.
84. 08:55This video only introduced
85. 08:57simplest one, privacy protection algorithms are indeed
86. 09:01which shows research directions in the Federated Learning.
87. 09:07How is the different defense also critical? for example, adding noise to gradient to strengthen protection
88. 09:16this method is called differential privacy.
89. 09:20However, the sort of method secures the information but also decreases the accuracy and convergence speed of the model.
90. 09:30Therefore, the privacy protection of Federated Learning is still a challenging task first of all, ensuring the robustness
91. 09:38of learning and combining barriers
92. 09:42security attacks is also a good research direction. Such as the poisoning attack.
93. 09:49It is an attacker who pretends to be a participant to join the Federation and insert poison in data.
94. 09:58This kind of poison data information will disrupt other participants and make the entire learning task a fail.
95. 10:07For example, this kind of attack can let the Global Model make a specific mistake by adding a customized adversarial examples
96. 10:17or living an attack back door. Overall how to defend the privacy leakage and improve the robustness are
97. 10:27essential research directions in the Federated Learning.
98. 10:33Thank you for watching

To enable the transcript, please select a language in the video player settings menu.