{"id":846,"date":"2019-01-09T09:14:16","date_gmt":"2019-01-09T09:14:16","guid":{"rendered":"http:\/\/csp15.epgpbooks.inflibnet.ac.in\/?post_type=chapter&p=846"},"modified":"2019-01-09T09:48:35","modified_gmt":"2019-01-09T09:48:35","slug":"basics-of-reinforcement-learning-i","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/chapter\/basics-of-reinforcement-learning-i\/","title":{"rendered":"Basics of Reinforcement Learning-I"},"content":{"raw":"
\"epgp<\/a>\r\n<\/span><\/div>\r\n\r\n
\r\n\r\n \r\n\r\nLearning Objectives<\/strong>\r\n\r\n \r\n\r\nThe learning objectives of the module are as follows:\r\n\r\n\u2022\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 A Basic Introduction to Reinforcement Learning\r\n\r\n\u2022\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 To explain the Elements of Reinforcement Learning\r\n\r\n\u2022\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 To discuss the applications of ReinforcementLearning\r\n\r\n \r\n\r\n37.1\u00a0 Reinforcement learning<\/strong>\r\n\r\n \r\n

Reinforcement learning is the problem of getting an agent to act in the world so as to maximize its rewards. For example, consider teaching a monkey a new trick: you cannot instruct the monkey what is to be done, but you can reward (a banana) or punish (scold) it depending on whether it does the right\/wrong thing. Here the monkey has to figure out what it did that made it get the reward or punishment, which is known in reinforcement learning context as the credit assignment problem. Learningtakesplaceasa resultofinteraction between an agentand the world. In other words percept received by an agent should be used not only for understanding, interpreting or prediction, as in the machine learning tasks we have discussed so far, butalsoforacting. Reinforcement learning is more general than supervised and unsupervised learning and learn from interaction with the environment to achieve a goal and getting an agent to act in the world so as to maximize its rewards. It allows agents to automatically determine the ideal behaviour within a specific context, in order to maximize its performance. Simple reward feedback is required for the agent to learn its behaviour; this is known as the reinforcement signal.<\/p>\r\n \r\n

The motivation behind reinforcement learning is that it allows an agent to learn its behavior based on feedback from the environment. This behavior can be learnt once and for all, or keep on adapting as time goes by. If the problem is modelled with care, some reinforcement learning algorithms can converge to the global optimum; this is the ideal behavior that maximizes the reward.<\/p>\r\n \r\n

It is a trial-and-error learning paradigm which learns from rewards and punishments. Reinforcement learning is not just an algorithm but a new paradigm in itself. Its objective is to learn about a system from minimal feedbacklike its behavior, control. It is inspired by behavioral psychology.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n\"\"\r\n

Figure: 37.1<\/strong><\/p>\r\n \r\n

As mentioned in figure 37.1, the reinforcement learning consists of the agent and the environment. The agent performs the action under the policy being followed and the environment is everything else other than the agent. Reinforcement learning is learning from interaction and it is a goal-oriented learning. It is learning about, from, and while interacting with an external environment. It is a learning which tells you, what to do (what action to take), how to map situations to actions so as to maximize a numerical reward signal.<\/p>\r\n \r\n

The general definition of reinforcement learning is as follows \"Reinforcement learning is learning what to do \u2014 how to map situations to actions \u2014 so as to maximize a numerical reward signal. The learner is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them.\"<\/p>\r\n \r\n\r\n37.2 Difference<\/strong> between Reinforcement, Supervised and Unsupervised learning<\/strong>\r\n\r\n \r\n

In supervised learning, the training information contains the desired (target) outputs. In reinforcement learning the training information contains evaluations (\u201crewards\u201d \/ \u201cpenalties\u201d) and the output are actions of the agent (Figure 37.2). In supervised learning we get the target output and if we subtract the actual output from the target output we will get the error of the system but that is not the case with reinforcement learning. In reinforcement learning the objective is to get as much reward as possible.<\/p>\r\n \r\n

37.2.1 Supervised Learning<\/strong><\/p>\r\n \r\n

In supervised learning we have a situation in which sample (input, output) pairs of the function to be learned can be perceived or are given. We can think as if there is a kind teacher who makes training data: (X,Y). (features, label) available and the job of the learning system is to predict Y, minimizing some loss.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n\"\"\r\n

Figure 37.2 Comparison Of Supervised Learning and Reinforcement Learning 37.2.2 Unsupervised Learning<\/strong><\/p>\r\n \r\n

In unsupervised learning the training data contains X ( the features only) and the job of the learning system is to find \u201csimilar\u201d points in high-dimensional X-space.<\/p>\r\n \r\n\r\n37.2.3 Reinforcement Learning<\/strong>\r\n\r\n \r\n

In reinforcement learning the agent acts on its environment, it receives some evaluation of its action (reinforcement), but is not told of which actionisthecorrectonetoachieveitsgoal. The training data: (S, A, R). (State-Action-Reward) and the learning system needs to develop an optimal policy(sequence of decision rules) so as to maximize its long-term reward.<\/p>\r\n \r\n\r\n37.3\u00a0 General Reinforcement Learning Algorithm<\/strong>\r\n\r\n \r\n

The first step is to initialise the learner\u2019s internal state . Then the algorithm should repeat forever the following steps<\/p>\r\n \r\n\r\n1.Observe current state s<\/em>\r\n\r\n \r\n\r\n2.Choose action a<\/em> using some evaluation function\r\n\r\n \r\n\r\n3.Execute action a<\/em>\r\n\r\n \r\n\r\n4.\u00a0 Let r<\/em> be immediate reward, s\u2019<\/em> new state 5.Update internal state based on s,a,r,s\u2019.<\/em>\r\n\r\n \r\n

Here learning is concerned with what to do that is how to map situations to actionsso as to maximize a numerical reward signal. Therefore learning to choose an action based on an evaluation function is the learning part.<\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n37.4 Key Features of Reinforcement Learning<\/strong>\r\n\r\n \r\n

The following are the key features of reinforcement learning. The learner is not told which actions to take and in that sense it is a trial-and-error search. It has the possibility of delayed reward (sacrifice short-term gains for greater long -term gains). In reinforcement learning there is a need to explore and exploit. It considers the whole problem of a goal-directed agent interacting with an uncertain environment. The reinforcement model assumes that each percept(e<\/em>) is enough to determine the State(the state is accessible) and the agent can decompose the Reward component from a percept. Therefore the agent task: to find a optimal policy, mapping states to actions, that maximize long-run measure of the reinforcement.<\/p>\r\n \r\n\r\n37.5 Agent-Environment Interface<\/strong>\r\n\r\n\"\"\r\n\r\n \r\n

Figure 37.3 Agent-Environment Interface<\/strong><\/p>\r\n \r\n

We have the agentenvironment interface as shown in Figure 37.3. Associated with this interface is s<\/strong>t<\/strong> which is the state at time t, r<\/strong>t<\/strong>which is the reward by the environment to the agent at time t and a<\/strong>t<\/strong> which is the action taken by the agent at time t. After taking the action at time t, the environment goes to the next state s<\/strong>t+1<\/strong> at time t+1, and gives the reward r<\/strong>t+1<\/strong>at time t+1.<\/p>\r\n \r\n

The task of reinforcement learning is to learn how to behave successfully so as to achieve a goal while interacting with an external environment and learn through experience from trial and error. Some examples of reinforcement learning are:<\/p>\r\n \r\n