Künstliche Intelligenz und Maschinelles Lernen in der Praxis

1. 00:00Welcome to Exkurs Reinforcement Learning.
2. 00:02In this unit, we want to implement Repeat learning briefly what the
3. 00:07theoretical and a practical example on reinforcement learning in practice.
4. 00:15We would have used OpenAI Gym, a Development platform for enforcement learning.
5. 00:23First, however, a short Repeat for enforcement learning.
6. 00:29Reinforcement Learning is one of the four paradigms in machine learning,
7. 00:32in addition to supervised, unsupervised and semisupervised learning.
8. 00:37In Reinforcement Learning, an agent that uses various actions interacts with the environment.
9. 00:42The environment provides rewards and the current status as an observation.
10. 00:48In the learning process, Agent tried the strategy or adjust and improve its policy.
11. 00:57These phases or learning can basically be in exploration and exploitation.
12. 01:02On the one hand, existing strategies are wanted to a certain extent
13. 01:07but also new strategies test and try.
14. 01:11This is usually done with a parameter specified or set.
15. 01:16At the beginning of the training the Exploration, of course, very high.
16. 01:20That means we try a lot of strategies.
17. 01:23In the end, rather less exploration , and we try rather previous
18. 01:32strategies. We have a our beginner course artificial intelligence and
19. 01:37machine learning for beginners a unit for enforcement learning.
20. 01:41If you are interested here also the link to it.
21. 01:48Let's go back to the Enviroment that we use here.
22. 01:50We use the OpenAI open source environment GYM.
23. 01:55This is an open source environment for the Development of Reinforcement Learning Models.
24. 02:00And there are many so-called enviroments.
25. 02:03Among them you can use Atari games. Nintendo games as well as environment for robots that make it
26. 02:09in reality.
27. 02:11Let me give you an example of how several of these things work. environment.
28. 02:16So for example, here we have several robots. which have to perform a certain task.
29. 02:21Or, yes, a robot arm that, for example, can Dice or several eggs, for example.
30. 02:29or juggling.
31. 02:34Use in our Use Case like the Super Mario Bros. environment.
32. 02:38The was developed by other developers and developers and develops and we can
33. 02:44framework of an open source project is also useful here.
34. 02:47Define in the first step we have this environment and
35. 02:50then we carry a certain number of steps. At the beginning here we show only
36. 02:56And I'm going to show you the environment.
37. 02:58There's no learning going on here. first of all, just the environment itself.
38. 03:04If you want to interrupt the execution, you can in the menu above, simply stop the cell executions.
39. 03:12This is the code for the simple definition, just such an enviroment.
40. 03:19We specify what kind of an enviroment we and also what kind of movements we want to use
41. 03:24or what to allow for actions like.
42. 03:26Then we run a each action in this environment
43. 03:38and get the current status environment or enviroment
44. 03:43a reward, the information whether we're already at the end of the world
45. 03:47or fulfill our task and more information.
46. 03:51At the beginning we lead a random action.
47. 03:56You'll see that in a second.
48. 03:58We are doing particularly well with This is not exactly the strategy.
49. 04:06Our environment opens in a parallel windows, and we see the
50. 04:11Super Mario world and Mario in the Fall jumps like wildly confused.
51. 04:17That's a pretty good fit, but it's actually only random action taken here
52. 04:22and not really or not at all the environment.
53. 04:32Don't be surprised.
54. 04:33I lead or have the execution here briefly interrupted with a keyboard interrupt.
55. 04:38So that we can sort of take this graphic Do not see execution in the background.
56. 04:45For our correct reinforcement Learning model we must
57. 04:48but take further steps.
58. 04:50First of all, let's look at what kind of possible actions as at all available
59. 04:55in this enviroment.
60. 04:56We will then do this in just two actions to train faster and
61. 05:01the solution space, or to minimize the solution space.
62. 05:07We only give our agents the Possibility to go right, so walk right,
63. 05:13and jump right, so jump and A usually on the console.
64. 05:21Let's look at it. that there are other
65. 05:26There are ways to run. We could also run left, jump left.
66. 05:30We could also Bend or get up.
67. 05:36However, like here first use a simple enviroment and accordingly also
68. 05:41only a few actions.
69. 05:42That means we limit our action space on two actions only.
70. 05:50In this case, we want to how we can learn from this enviroment as
71. 05:56Get back information and give us here for once, the return values of the environment.
72. 06:04We see here that we can next state we can see the reward
73. 06:10and whether our agent is ready.
74. 06:13As well as information we collected coins.
75. 06:17How many lives do we have left?
76. 06:19Which stage are we on right now? What time has passed?
77. 06:23And where are they? Are we just in our world?
78. 06:28But that's still Very little information.
79. 06:31Here we have to do something called wrapper define that we also provide feedback in the form of
80. 06:36Get pictures. Also if we introduce it just in week four,
81. 06:41we use here a convolutional neural Network, i.e. an image analysis network
82. 06:47to understand the current state of the game and take action accordingly.
83. 06:53That we just take these pictures of the environment , we have to write the following wrapper
84. 06:59or make the following adjustment that our enviroment gives us images, in that case not
85. 07:06for every frame, but for me, I believe only every fourth frame returns an image, which we also
86. 07:13can be analyzed as feedback or observation of our enviroment.
87. 07:22Since we haven't really learned yet, here is a hint to our approach to learning.
88. 07:27In this case, we are using the so-called Q-Learning approach or Q-Learning strategy.
89. 07:34We will not go into the details.
90. 07:36However, we want to say that in Q-Learning the agent receives the
91. 07:41using the environment and over time learn which actions in a particular
92. 07:46State of enviroment is best.
93. 07:48The state of the enviroment is mainly our pictures, that means in which
94. 07:54Position am I right now?
95. 07:56For example, if enemies are in close proximity Look, is something above me, below me?
96. 08:01All this can be used as input to adjust my actions accordingly.
97. 08:09We want our agents to be part of the First steps in learning, so to speak.
98. 08:16It is then thought that we will have been trained to date, and
99. 08:25at certain times, see how far we can go in the Training, which level or which results
100. 08:30can already deliver.
101. 08:32At the beginning we will be very Quotation marks, moving experimentally.
102. 08:38That means a lot of things won't work yet.
103. 08:40Then we also look at an agent 100 epochs of training, after 1000 epochs,
104. 08:46Training and after a more or less fully trained model.
105. 08:52Episode training is called that because it's a Pass or run of our agent
106. 09:01better in this case just dies Or fall down somewhere.
107. 09:04And then an episode of and the counter counts one up.
108. 09:09Here the note that for a really good model several 10,000 episodes of training are needed.
109. 09:16To train a thousand episodes alone, This can take several hours.
110. 09:20That's why we're here the trained models.
111. 09:27We see our Mario directly here at the beginning of the training, we see that he
112. 09:38often sticks to such tubes very strongly or often runs into the monsters.
113. 09:47Yes, the best is not yet.
114. 09:49We are still very much at the beginning of our Training and see how it works
115. 09:56behaves after for example 100 Episodes or 1000 episodes training.
116. 10:05I also interrupt here, it will execute. that we are using our other models
117. 10:09of the second generation.
118. 10:11Right, here we are the trained models.
119. 10:16First, we try to get some 100 episodes to use trained model
120. 10:24and load that. To load this, But then we have to
121. 10:28reset our environment.
122. 10:30And we do this like this. Now we first load our model with 100
123. 10:37Episodes training and see once how our Mario beats himself in this world.
124. 10:51We see after a hundred episodes Training, and that's really not much
125. 10:55Training, we see the Mario already here very simple things fail, that is to say
126. 11:02Example of a tube at the very beginning can't really be overcome.
127. 11:06And yes, time goes by, so it's optimal This is far from being the case.
128. 11:18When we think of a model with 1000 episodes watch training,
129. 11:22It's certainly a step forward.
130. 11:25But not much progress, we still see significant errors and
131. 11:31a strategy that is not optimal, overcome the obstacles.
132. 11:43You can see here, over the tubes. will go a little better.
133. 11:47However, yes, you see, you fail still a little bit on the tubes.
134. 11:52Maybe we'll see if he can do that again.
135. 11:57Well...
136. 12:00So even after a thousand episodes have We still don't have a really good model.
137. 12:09Now let's look at a finished model where we have significantly better behavior
138. 12:15and higher reward, i.e. the achievement of the target, Collecting coins, shorter times etc. .
139. 12:22The model shown here was for several 10,000 episodes, is
140. 12:27but still by no means perfect.
141. 12:29Here is also the note, of course we train our Agents at a very simple level of the game.
142. 12:36In addition, at the beginning of the Actions limited to two.
143. 12:40Give the agent more actions to choose from and also more episodes in training, then we get
144. 12:45certainly a better model and a Model, what perhaps even higher levels
145. 12:49can do it.
146. 13:19Right, you're looking at the At the beginning, it took a while until the model
147. 13:24It's sort of come over the tubes.
148. 13:26However, we see here significantly better behavior.
149. 13:29Straight behind out if the something more difficult parts of the level.
150. 13:37That means yes, we still have by far not an optimal model.
151. 13:41However, our Agent much further.
152. 13:44Maybe we'll even see that until to the end, however, as long as we want
153. 13:49then do not wait.
154. 13:51Right, so here we are. that the agent comes very far.
155. 14:01That's it for the Exkurs Reinforcement Learning.
156. 14:04If you are using the or some of the applications, you can use
157. 14:09OpenAI Gym with these Start applications very easily.
158. 14:13And, yes, we wish you much Have fun trying it out yourself.
159. 14:16trying out

To enable the transcript, please select a language in the video player settings menu.