{"id":625,"date":"2019-01-08T10:51:55","date_gmt":"2019-01-08T10:51:55","guid":{"rendered":"http:\/\/csp15.epgpbooks.inflibnet.ac.in\/?post_type=chapter&p=625"},"modified":"2019-01-08T11:13:06","modified_gmt":"2019-01-08T11:13:06","slug":"bayes-learning","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/chapter\/bayes-learning\/","title":{"rendered":"Bayes Learning"},"content":{"raw":"
\r\n\r\n \r\n\r\nLearning Objectives:<\/strong>\r\n\r\n \r\n\r\nThe learning objectives of this module are as follows:\r\n\r\n \r\n\r\n\u2022\u00a0\u00a0\u00a0\u00a0\u00a0 To understand Probability, Bayes Theorem and Bayes Learning\r\n\r\n\u2022\u00a0\u00a0\u00a0\u00a0\u00a0 To understand Bayes Learning Framework\r\n\r\n\u2022\u00a0\u00a0\u00a0\u00a0\u00a0 To understand Probabilistic Classification\r\n\r\n \r\n\r\n29.1 Introduction<\/strong>\r\n\r\n \r\n

Before we discuss Bayes learning let us consider the important issue of fitting a model to data. This is important in machine learning before based on this model built from data and associated target values, we predict target values for unknown hitherto unseen data. Some of the important criteria when we fit a model are the choice of weights and thresholds. In some cases we need to incorporate prior knowledge and in other cases we need to merge multiple sources of information. Another important factor is the modelling of uncertainty. Bayesian reasoning provides solutions to the above issues. Bayesian reasoning is associated with probability, statistics and data fitting. Figure 29.1 shows the idea of frequentist or orthodox statistics where probability is defined as the frequency of occurrences in possibly infinite number of trials.<\/p>\r\n \r\n\r\nPrinciple #1:<\/strong> The first principle of probability as stated by Pierre-Simon Laplace (Figure 29.2) in 1814 is given below:\r\n\r\n \r\n\r\n\u201cProbability theory is nothing more than common sense reduced to calculation.\u201d\r\n\r\n<\/div>\r\n

\r\n\r\n\"\"\r\n

Figure 29.1 Probability as Frequency of Occurences<\/strong><\/p>\r\n\"\"\r\n

Figure 29.2 Pierre-Simon Laplace in 1814<\/strong><\/p>\r\n \r\n\r\n29.2 Probability Basics<\/strong>\r\n\r\n \r\n

Let us recall the basic laws of probability given in Figure 29.3. We talk about prior or independent probability of a variable X as P(X). Conditional probability P(X1|X2) is defined as the probability of X1 given X2. Joint probability is defined as the probability of both X1 and X2 occurring together. The relation between joint probability and conditional probability is also given. If X1 and X2 are independent<\/p>\r\n \r\n\r\n\"\"\r\n

Figure 28.3 Basics of Probability Bayesian Rule<\/strong><\/p>\r\n \r\n

then the conditionally probability of X1 given X2 reduces to P(X1) and similarly the conditional probability of X2 given X1 reduces to P(X2). The joint probability when X1 and X2 are independent is given by the product of P(X1) and P(X2).<\/p>\r\n \r\n\r\n29.3 Bayesian Rule<\/strong>\r\n\r\n \r\n

The Baye\u2019s Rule or Bayes theorem is a very important concept that is used in many different aspects of machine learning. Figure 29.4 shows a general definition of the rule where the conditional probability of C given X is given by the conditional probability of X given C multiplied by the independent probability of C divided by the independent probability of X. Here the conditional probability of C given X (that is the probability to be determined) is called the posterior probability, the conditional probability of X given C (usually determined from the labelled data) is called likelihood, independent probability of C is called prior probability and the independent probability of X is called evidence. This very important theorem is further explained in Figure 29.5. Here we explain that the posterior probability of the target vector W given the training data D can be determined knowing the prior probability of target vector W, likelihood or probability of observed data D given target vector W and the evidence or the probability of the data D.<\/p>\r\n

<\/p>\r\n\r\n<\/div>\r\n\"\"<\/strong>\r\n

Figure 29.5 Explanation of Bayes Theorem<\/p>\r\n\r\n

\r\n\r\n \r\n

then the conditionally probability of X1 given X2 reduces to P(X1) and similarly the conditional probability of X2 given X1 reduces to P(X2). The joint probability when X1 and X2 are independent is given by the product of P(X1) and P(X2).<\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n29.4 Discrete and Continuous Variables \u2013 Modelling the World<\/strong>\r\n\r\n \r\n

Let us first discuss probabilities over discrete variables. A probability function maps the possible values of x against their respective probabilities of occurrence, P(x). P(x) is a number from 0 to 1.0. Let us assume the simple case where the discrete variable C can take on two values Heads or Tails. We know that in such cases the probability of C=Heads and C=Tails i.e P(C=Heads) = P(C=Tails) is 0.5 and the P(C=Heads) + P(C=Tails) = 1.<\/p>\r\n\"\"\r\n\r\n \r\n

Now let consider the case of continuous variables. The probability function that accompanies a continuous random variable is a continuous mathematical function that integrates to 1. Let x be a real number. Then we need to describe beliefs over x where x may be joint angles, price of stock etc..<\/p>\r\n \r\n\r\n\"\"\r\n

In the case of continuous variables we describe the data in terms of a probability distribution function (PDF) or marginal probability. One common distribution that is used to describe the data is the Gaussian or normal distribution. This distribution can be defined by the parameters mean m and standard deviation s (Figure 29.6). This is a bell shaped curve with different centers and spreads depending on m and s.<\/p>\r\n\"\"\r\n

Figure 29.6 Gaussian Distribution<\/strong><\/p>\r\n

Now the question is why do we use Gaussians? This is due to the fact that it has convenient analytic properties, it is governed by the central limit theorem, works reasonably well for most real data. Though it does not suit all types of data, it acts as a good building block. The values of of the data point x is given by the function (Figure 29.7).<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n\"\"Figure 29.7 Gaussian Distribution Function<\/strong>\r\n\r\n29.5 Inference<\/strong>\r\n\r\n \r\n

How do we reason about the world from observations? In this context there are three important sets of variables namely observations, unknowns and auxiliary (\u201cnuisance\u201d) variables. Given the observations, what are the probabilities of the unknowns? The first step is parameter estimation. Here we are given lots of data, and using this data we need to determine unknown parameters. The various estimators used to determine these parameters include Maximum A Posteriori (MAP), Maximum likelihood and Unbiased estimators. We will discuss these estimators later.<\/p>\r\n \r\n

A word about finding a model that fits the data. A model that fits the data well but does not generalize is said to be over fitting. This does not augur well for machine learning since these type of models do not predict well on unseen data. This occurs when an estimate is obtained from a \u201cspread-out\u201d posterior.<\/p>\r\n \r\n\r\n29.6 Probability \u201cPrinciples\u201d<\/strong>\r\n\r\n \r\n\r\n29.6.1 Basic Probability:<\/strong>\r\n\r\n \r\n\r\nThe following are some of the facts about probability:\r\n\r\n \r\n\r\n1.\u00a0\u00a0\u00a0 Probability theory is common sense reduced to calculation.\r\n\r\n \r\n\r\n2.\u00a0\u00a0\u00a0 Given a model, we can derive any probability\r\n\r\n \r\n\r\n3.\u00a0\u00a0\u00a0 Describe a model of the world, and then compute the probabilities of the unknowns with Bayes\u2019 Rule\r\n\r\n \r\n\r\n4.\u00a0\u00a0\u00a0 Parameter estimation leads to over-fitting when the posterior isn\u2019t \u201cpeaked.\u201d However, it is easier than\r\n\r\nBayesian prediction.\r\n\r\n \r\n\r\n5.\u00a0\u00a0\u00a0 Least-squares estimation is a special case of MAP, and can suffer from over- and under-fitting\r\n\r\n \r\n\r\n6.\u00a0\u00a0 You can learn (or marginalize out) all parameters.<\/span>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n29.6.2 Benefits of the Bayesian approach<\/strong>\r\n\r\n \r\n

The following are some of the benefits of the general Bayesian approach: This approach allows principled modeling of noise and uncertainty. It defines a unified model for learning and synthesis where essentially all parameters can be learnt and it is possible to get reasonably good results from simple models. This is an area where lots of good research is being carried out and several good algorithms are available.<\/p>\r\n \r\n

Some of the applications where Bayesian methods have been employed include data mining, robotics, signal processing, bioinformatics, text analysis, etc.<\/p>\r\n \r\n\r\n29.7 Bayesian Framework<\/strong>\r\n\r\n \r\n

The Bayesian framework allows us to combine observed data and prior knowledge. It provides practical learning algorithms. Moreover it is a generative (model based) approach, which offers a useful conceptual framework. This essentially means that any kind of object (e.g. time series, trees, etc.) can be classified, based on a probabilistic model specification.<\/p>\r\n

The Bayesian framework assumes that we always have a prior distribution for everything. However the usage of prior knowledge alone may not be effective. When we observe some data, we combine our prior distribution with a likelihood term to get a posterior distribution. The likelihood term takes into account how probable the observed data is given the parameters of the model. The framework favors parameter settings that make the data likely. It in a way changes the estimate based solely on the prior and fights the prior. With sufficient data, the likelihood terms always win.<\/p>\r\n \r\n\r\n29.8 What is Bayesian Learning?<\/strong>\r\n\r\n \r\n

Bayesian learning uses probability to model<\/strong> data and quantify uncertainty of predictions. As already discussed it facilitates incorporation of prior knowledge and allows for decision-theoretic reasoning and a probabilistic approach to inference. Some of the assumptions are that the quantities of interest are governed by probability distribution. The optimal decisions are based on reasoning about probabilities and observations. It provides quantitative approach to weighing how evidence supports alternative hypotheses<\/p>\r\n \r\n\r\n29.8.1 Features of Bayesian Learning<\/strong>\r\n\r\n \r\n

Each observed training data can incrementally decrease or increase the estimated probability of a hypothesis rather than completely eliminating a hypothesis if it is found to be inconsistent with a single example. Prior<\/strong> knowledge <\/strong>can be combined with observed data to determine the final\u00a0probability of a hypothesis. Moreover new instances can be classified by combining predictions of multiple hypotheses. Even in computationally intractable cases, Bayesian optimal classifier provides a standard of optimal decision against which other practical methods can be compared.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n29.8.2 Context of Bayesian Learning<\/strong>\r\n\r\n \r\n

Bayesian Decision Theory <\/em><\/strong>came long before Version Spaces, Decision Tree Learning and Neural Networks. It was studied in the field of Statistical Theory and more specifically, in the field of Pattern Recognition<\/em><\/strong>. Bayesian Decision Theory is at the basis of important learning schemes such as the Na\u00efve Bayes<\/em><\/strong> Classifier<\/em><\/strong>, Bayesian Belief Networks <\/em><\/strong>and the Expectation Maximization (EM) Algorithm. <\/em><\/strong>Bayesian Decision Theory is also useful as it provides a framework within which many non-Bayesian classifiers can be studied.<\/p>\r\n \r\n\r\n29.8.3 Advantages of Bayesian Learning<\/strong>\r\n\r\n \r\n

Bayesian approaches, including the Naive Bayes classifier, are among the most common and practical ones in machine learning. Bayesian decision theory allows us to revise probabilities based on new evidence. Bayesian methods provide a useful perspective for understanding many learning algorithms that do not manipulate probabilities.<\/p>\r\n \r\n\r\n29.9 Relook at Bayes Theorem<\/strong>\r\n\r\n \r\n\r\n\"\"\r\n\r\n \r\n

Now let us relook at the Bayes theorem from the viewpoint of Bayesian Learning. The goal is to determine the most probable hypothesis, given the data D<\/em> plus any initial knowledge about the prior probabilities of the various hypotheses in H<\/em>. With this premise, Prior probability of h, P(h)<\/em><\/strong> reflects any background knowledge we have about the chance that h<\/em> is a correct hypothesis. This is before having observed the data. Prior probability of D,<\/em><\/strong> P(D) <\/em><\/strong>reflects the probability that training data D <\/em>will be observed given no knowledge about which hypothesis h<\/em> holds. Then we have Conditional<\/em><\/strong> Probability of observation D, P(D|h) <\/em><\/strong>which denotes the probability of observing data D<\/em> given some world in which hypothesis h<\/em> holds. Finally we have Posterior probability of h, P(h|D)<\/em><\/strong> which represents the probability that h<\/em> holds given the observed training data D<\/em>. It reflects our confidence that h<\/em> holds after we have seen the training data D<\/em> and it is the quantity that Machine Learning researchers are interested in.<\/p>\r\n \r\n

Bayes Theorem <\/em><\/strong>allows us to compute P(h|D). <\/em><\/strong>We will now discuss the different ways to carry out this computation. We will discuss two methods namely<\/p>\r\n \r\n

M<\/span>aximum A Posteriori (MAP) & Maximum Likelihood (ML).<\/strong> The goal is to find the most probable hypothesis <\/span>h<\/em> from a set of candidate hypotheses <\/span>H<\/em> given the observed data <\/span>D<\/em>.<\/span><\/p>\r\n \r\n

In the M<\/span>aximum A Posteriori (MAP)<\/strong> method we need to find the hypothesis h among the set of hypothesis H that maximizes the posterior probability <\/span>P(h|D).<\/em><\/strong> Now by applying Bayes theorem, we need to find the hypothesis h that maximizes the likelihood of the data D for a given h and the prior probability of h. The prior probability of the data D does not affect the maximization for finding h so that is not considered. Hence we have the MAP hypothesis as given below:<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n\"\"\r\n\r\n \r\n

If every hypothesis in H<\/em> is equally probable a priori, we only need to consider the likelihood of the data D<\/em> given h<\/em>, P(D|h).<\/em><\/strong> This gives rise to h<\/em>ML<\/em> or the Maximum Likelihood<\/em><\/strong>, as follows:<\/p>\r\n\"\"\r\n\r\n \r\n\r\n26.9.1 Bayes Example<\/strong>\r\n\r\n \r\n\r\nNow let us look at an example of Bayes Theorem. We have data on past student performance. For each student we know the following:\r\n\r\n \r\n\r\n\u2013\u00a0 If student\u2019s GPA > 3.0 (G)\r\n\r\n \r\n\r\n\u2013\u00a0 If student had a strong math background (M)\r\n\r\n \r\n\r\n\u2013\u00a0 If student is a hard worker (H)\r\n\r\n \r\n\r\n\u2013\u00a0 If student passed or failed course\r\n\r\n \r\n\r\n\"\"\r\n\r\n \r\n

We are given the data as in the Table of Figure 29.8 where p is the number of binary features. Let us assume that the probability to pass or fail are equal i.e<\/p>\r\n \r\n

P(pass)=P(fail)=0.5. Now if we are given that the student has GPA,3, has Maths, and is not a Hard worker that is the value of the features are x={0,1,0}. From the table we know that for this value of x the probability of passing is 0.05 and the probability of failing is 0.20.Therefore we can find f(x)=0.25 (Figure 29.8) which means that the classification gives the value as fail.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n\"\"\r\n\r\n \r\n

Figure 29.8 Example for Bayes Theorem<\/strong><\/p>\r\n \r\n\r\n29.10 Bayesian Classification: Why?<\/strong>\r\n\r\n \r\n\r\nThe following are the reasons why Bayesian classification s important.\r\n\r\n \r\n

\u2022 Probabilistic learning: We can calculate explicit probabilities for hypothesis, and is among the most practical approaches to certain types of learning problems.<\/p>\r\n \r\n

\u2022 Incremental: Each training example can incrementally increase\/decrease the probability that a hypothesis is correct. Prior knowledge can also be combined with observed data.<\/p>\r\n \r\n

\u2022 Probabilistic prediction: It is possible to predict multiple hypotheses, weighted by their probabilities.<\/p>\r\n \r\n

\u2022 Standard: Even when Bayesian methods are computationally intractable, they can provide a standard of optimal decision making against which other methods can be measured<\/p>\r\n \r\n\r\n29.10.1 Bayes Optimal Classifier<\/strong>\r\n\r\n \r\n

We will now briefly outline the different types of Bayesian learning. The forthcoming modules will discuss these methods in detail. One great advantage of Bayesian Decision Theory is that it gives us a lower bound on the classification error that can be obtained for a given problem.<\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n \r\n

In Bayes optimal classification<\/strong> the most probable classification of a new instance is obtained by combining the predictions of all hypotheses, weighted by their posterior probabilities as given below:<\/p>\r\n \r\n\r\n\"\"\r\n\r\n \r\n

where V<\/em> is the set of all the values a classification can take and v<\/em>j<\/em> is one of the possible classifications from this set. Unfortunately, Bayes Optimal Classifier is usually too costly to apply! The compromise is the Na\u00efve Bayes Classifier<\/em><\/strong> which we will discuss in detail later.<\/p>\r\n \r\n\r\n29.10.2 Bayesian Belief Networks<\/strong>\r\n\r\n \r\n

The Bayes Optimal Classifier<\/em><\/strong> is often too costly to apply. The Na\u00efve Bayes<\/em><\/strong> Classifier <\/em><\/strong>uses the conditional independence assumption to defray these costs. However, in many cases, such an assumption is overly restrictive. Bayesian belief networks <\/em><\/strong>provide an intermediate <\/em><\/strong>approach which allows stating conditional independence assumptions that apply to subsets<\/em><\/strong> of the variable.<\/p>\r\n \r\n\r\n29.10.3 Expectation-Maximization (EM)<\/strong>\r\n\r\n \r\n

Now we will have a brief look at Expectation-Maximization approach. Consider learning a na\u00efve Bayes classifier using unlabeled<\/em> data. How can we estimate e.g. <\/em>P(A|C)? (Figure 29.9)<\/p>\r\n\r\n<\/div>\r\n\"\"\r\n

Figure 29.9 Dependencies of the variable A,B & C<\/strong><\/p>\r\n \r\n\r\nSummary<\/strong>\r\n

    \r\n \t
  • Explained Probability, Bayes Theorem and Bayes Learning<\/li>\r\n \t
  • Discussed Bayes Learning Framework<\/li>\r\n \t
  • Outlined Maximum A Posteriori (MAP) & Maximum Likelihood (ML)<\/li>\r\n<\/ul>","rendered":"
    \n

     <\/p>\n

    Learning Objectives:<\/strong><\/p>\n

     <\/p>\n

    The learning objectives of this module are as follows:<\/p>\n

     <\/p>\n

    \u2022\u00a0\u00a0\u00a0\u00a0\u00a0 To understand Probability, Bayes Theorem and Bayes Learning<\/p>\n

    \u2022\u00a0\u00a0\u00a0\u00a0\u00a0 To understand Bayes Learning Framework<\/p>\n

    \u2022\u00a0\u00a0\u00a0\u00a0\u00a0 To understand Probabilistic Classification<\/p>\n

     <\/p>\n

    29.1 Introduction<\/strong><\/p>\n

     <\/p>\n

    Before we discuss Bayes learning let us consider the important issue of fitting a model to data. This is important in machine learning before based on this model built from data and associated target values, we predict target values for unknown hitherto unseen data. Some of the important criteria when we fit a model are the choice of weights and thresholds. In some cases we need to incorporate prior knowledge and in other cases we need to merge multiple sources of information. Another important factor is the modelling of uncertainty. Bayesian reasoning provides solutions to the above issues. Bayesian reasoning is associated with probability, statistics and data fitting. Figure 29.1 shows the idea of frequentist or orthodox statistics where probability is defined as the frequency of occurrences in possibly infinite number of trials.<\/p>\n

     <\/p>\n

    Principle #1:<\/strong> The first principle of probability as stated by Pierre-Simon Laplace (Figure 29.2) in 1814 is given below:<\/p>\n

     <\/p>\n

    \u201cProbability theory is nothing more than common sense reduced to calculation.\u201d<\/p>\n<\/div>\n

    \n

    \"\"<\/p>\n

    Figure 29.1 Probability as Frequency of Occurences<\/strong><\/p>\n

    \"\"<\/p>\n

    Figure 29.2 Pierre-Simon Laplace in 1814<\/strong><\/p>\n

     <\/p>\n

    29.2 Probability Basics<\/strong><\/p>\n

     <\/p>\n

    Let us recall the basic laws of probability given in Figure 29.3. We talk about prior or independent probability of a variable X as P(X). Conditional probability P(X1|X2) is defined as the probability of X1 given X2. Joint probability is defined as the probability of both X1 and X2 occurring together. The relation between joint probability and conditional probability is also given. If X1 and X2 are independent<\/p>\n

     <\/p>\n

    \"\"<\/p>\n

    Figure 28.3 Basics of Probability Bayesian Rule<\/strong><\/p>\n

     <\/p>\n

    then the conditionally probability of X1 given X2 reduces to P(X1) and similarly the conditional probability of X2 given X1 reduces to P(X2). The joint probability when X1 and X2 are independent is given by the product of P(X1) and P(X2).<\/p>\n

     <\/p>\n

    29.3 Bayesian Rule<\/strong><\/p>\n

     <\/p>\n

    The Baye\u2019s Rule or Bayes theorem is a very important concept that is used in many different aspects of machine learning. Figure 29.4 shows a general definition of the rule where the conditional probability of C given X is given by the conditional probability of X given C multiplied by the independent probability of C divided by the independent probability of X. Here the conditional probability of C given X (that is the probability to be determined) is called the posterior probability, the conditional probability of X given C (usually determined from the labelled data) is called likelihood, independent probability of C is called prior probability and the independent probability of X is called evidence. This very important theorem is further explained in Figure 29.5. Here we explain that the posterior probability of the target vector W given the training data D can be determined knowing the prior probability of target vector W, likelihood or probability of observed data D given target vector W and the evidence or the probability of the data D.<\/p>\n

    \n<\/div>\n

    \"\"<\/strong><\/p>\n

    Figure 29.5 Explanation of Bayes Theorem<\/p>\n

    \n

     <\/p>\n

    then the conditionally probability of X1 given X2 reduces to P(X1) and similarly the conditional probability of X2 given X1 reduces to P(X2). The joint probability when X1 and X2 are independent is given by the product of P(X1) and P(X2).<\/p>\n<\/div>\n

    \n

     <\/p>\n

    29.4 Discrete and Continuous Variables \u2013 Modelling the World<\/strong><\/p>\n

     <\/p>\n

    Let us first discuss probabilities over discrete variables. A probability function maps the possible values of x against their respective probabilities of occurrence, P(x). P(x) is a number from 0 to 1.0. Let us assume the simple case where the discrete variable C can take on two values Heads or Tails. We know that in such cases the probability of C=Heads and C=Tails i.e P(C=Heads) = P(C=Tails) is 0.5 and the P(C=Heads) + P(C=Tails) = 1.<\/p>\n

    \"\"<\/p>\n

     <\/p>\n

    Now let consider the case of continuous variables. The probability function that accompanies a continuous random variable is a continuous mathematical function that integrates to 1. Let x be a real number. Then we need to describe beliefs over x where x may be joint angles, price of stock etc..<\/p>\n

     <\/p>\n

    \"\"<\/p>\n

    In the case of continuous variables we describe the data in terms of a probability distribution function (PDF) or marginal probability. One common distribution that is used to describe the data is the Gaussian or normal distribution. This distribution can be defined by the parameters mean m and standard deviation s (Figure 29.6). This is a bell shaped curve with different centers and spreads depending on m and s.<\/p>\n

    \"\"<\/p>\n

    Figure 29.6 Gaussian Distribution<\/strong><\/p>\n

    Now the question is why do we use Gaussians? This is due to the fact that it has convenient analytic properties, it is governed by the central limit theorem, works reasonably well for most real data. Though it does not suit all types of data, it acts as a good building block. The values of of the data point x is given by the function (Figure 29.7).<\/span><\/p>\n<\/div>\n

    \n

     <\/p>\n

    \"\"Figure 29.7 Gaussian Distribution Function<\/strong><\/p>\n

    29.5 Inference<\/strong><\/p>\n

     <\/p>\n

    How do we reason about the world from observations? In this context there are three important sets of variables namely observations, unknowns and auxiliary (\u201cnuisance\u201d) variables. Given the observations, what are the probabilities of the unknowns? The first step is parameter estimation. Here we are given lots of data, and using this data we need to determine unknown parameters. The various estimators used to determine these parameters include Maximum A Posteriori (MAP), Maximum likelihood and Unbiased estimators. We will discuss these estimators later.<\/p>\n

     <\/p>\n

    A word about finding a model that fits the data. A model that fits the data well but does not generalize is said to be over fitting. This does not augur well for machine learning since these type of models do not predict well on unseen data. This occurs when an estimate is obtained from a \u201cspread-out\u201d posterior.<\/p>\n

     <\/p>\n

    29.6 Probability \u201cPrinciples\u201d<\/strong><\/p>\n

     <\/p>\n

    29.6.1 Basic Probability:<\/strong><\/p>\n

     <\/p>\n

    The following are some of the facts about probability:<\/p>\n

     <\/p>\n

    1.\u00a0\u00a0\u00a0 Probability theory is common sense reduced to calculation.<\/p>\n

     <\/p>\n

    2.\u00a0\u00a0\u00a0 Given a model, we can derive any probability<\/p>\n

     <\/p>\n

    3.\u00a0\u00a0\u00a0 Describe a model of the world, and then compute the probabilities of the unknowns with Bayes\u2019 Rule<\/p>\n

     <\/p>\n

    4.\u00a0\u00a0\u00a0 Parameter estimation leads to over-fitting when the posterior isn\u2019t \u201cpeaked.\u201d However, it is easier than<\/p>\n

    Bayesian prediction.<\/p>\n

     <\/p>\n

    5.\u00a0\u00a0\u00a0 Least-squares estimation is a special case of MAP, and can suffer from over- and under-fitting<\/p>\n

     <\/p>\n

    6.\u00a0\u00a0 You can learn (or marginalize out) all parameters.<\/span><\/p>\n<\/div>\n

    \n

     <\/p>\n

    29.6.2 Benefits of the Bayesian approach<\/strong><\/p>\n

     <\/p>\n

    The following are some of the benefits of the general Bayesian approach: This approach allows principled modeling of noise and uncertainty. It defines a unified model for learning and synthesis where essentially all parameters can be learnt and it is possible to get reasonably good results from simple models. This is an area where lots of good research is being carried out and several good algorithms are available.<\/p>\n

     <\/p>\n

    Some of the applications where Bayesian methods have been employed include data mining, robotics, signal processing, bioinformatics, text analysis, etc.<\/p>\n

     <\/p>\n

    29.7 Bayesian Framework<\/strong><\/p>\n

     <\/p>\n

    The Bayesian framework allows us to combine observed data and prior knowledge. It provides practical learning algorithms. Moreover it is a generative (model based) approach, which offers a useful conceptual framework. This essentially means that any kind of object (e.g. time series, trees, etc.) can be classified, based on a probabilistic model specification.<\/p>\n

    The Bayesian framework assumes that we always have a prior distribution for everything. However the usage of prior knowledge alone may not be effective. When we observe some data, we combine our prior distribution with a likelihood term to get a posterior distribution. The likelihood term takes into account how probable the observed data is given the parameters of the model. The framework favors parameter settings that make the data likely. It in a way changes the estimate based solely on the prior and fights the prior. With sufficient data, the likelihood terms always win.<\/p>\n

     <\/p>\n

    29.8 What is Bayesian Learning?<\/strong><\/p>\n

     <\/p>\n

    Bayesian learning uses probability to model<\/strong> data and quantify uncertainty of predictions. As already discussed it facilitates incorporation of prior knowledge and allows for decision-theoretic reasoning and a probabilistic approach to inference. Some of the assumptions are that the quantities of interest are governed by probability distribution. The optimal decisions are based on reasoning about probabilities and observations. It provides quantitative approach to weighing how evidence supports alternative hypotheses<\/p>\n

     <\/p>\n

    29.8.1 Features of Bayesian Learning<\/strong><\/p>\n

     <\/p>\n

    Each observed training data can incrementally decrease or increase the estimated probability of a hypothesis rather than completely eliminating a hypothesis if it is found to be inconsistent with a single example. Prior<\/strong> knowledge <\/strong>can be combined with observed data to determine the final\u00a0probability of a hypothesis. Moreover new instances can be classified by combining predictions of multiple hypotheses. Even in computationally intractable cases, Bayesian optimal classifier provides a standard of optimal decision against which other practical methods can be compared.<\/span><\/p>\n<\/div>\n

    \n

     <\/p>\n

    29.8.2 Context of Bayesian Learning<\/strong><\/p>\n

     <\/p>\n

    Bayesian Decision Theory <\/em><\/strong>came long before Version Spaces, Decision Tree Learning and Neural Networks. It was studied in the field of Statistical Theory and more specifically, in the field of Pattern Recognition<\/em><\/strong>. Bayesian Decision Theory is at the basis of important learning schemes such as the Na\u00efve Bayes<\/em><\/strong> Classifier<\/em><\/strong>, Bayesian Belief Networks <\/em><\/strong>and the Expectation Maximization (EM) Algorithm. <\/em><\/strong>Bayesian Decision Theory is also useful as it provides a framework within which many non-Bayesian classifiers can be studied.<\/p>\n

     <\/p>\n

    29.8.3 Advantages of Bayesian Learning<\/strong><\/p>\n

     <\/p>\n

    Bayesian approaches, including the Naive Bayes classifier, are among the most common and practical ones in machine learning. Bayesian decision theory allows us to revise probabilities based on new evidence. Bayesian methods provide a useful perspective for understanding many learning algorithms that do not manipulate probabilities.<\/p>\n

     <\/p>\n

    29.9 Relook at Bayes Theorem<\/strong><\/p>\n

     <\/p>\n

    \"\"<\/p>\n

     <\/p>\n

    Now let us relook at the Bayes theorem from the viewpoint of Bayesian Learning. The goal is to determine the most probable hypothesis, given the data D<\/em> plus any initial knowledge about the prior probabilities of the various hypotheses in H<\/em>. With this premise, Prior probability of h, P(h)<\/em><\/strong> reflects any background knowledge we have about the chance that h<\/em> is a correct hypothesis. This is before having observed the data. Prior probability of D,<\/em><\/strong> P(D) <\/em><\/strong>reflects the probability that training data D <\/em>will be observed given no knowledge about which hypothesis h<\/em> holds. Then we have Conditional<\/em><\/strong> Probability of observation D, P(D|h) <\/em><\/strong>which denotes the probability of observing data D<\/em> given some world in which hypothesis h<\/em> holds. Finally we have Posterior probability of h, P(h|D)<\/em><\/strong> which represents the probability that h<\/em> holds given the observed training data D<\/em>. It reflects our confidence that h<\/em> holds after we have seen the training data D<\/em> and it is the quantity that Machine Learning researchers are interested in.<\/p>\n

     <\/p>\n

    Bayes Theorem <\/em><\/strong>allows us to compute P(h|D). <\/em><\/strong>We will now discuss the different ways to carry out this computation. We will discuss two methods namely<\/p>\n

     <\/p>\n

    M<\/span>aximum A Posteriori (MAP) & Maximum Likelihood (ML).<\/strong> The goal is to find the most probable hypothesis <\/span>h<\/em> from a set of candidate hypotheses <\/span>H<\/em> given the observed data <\/span>D<\/em>.<\/span><\/p>\n

     <\/p>\n

    In the M<\/span>aximum A Posteriori (MAP)<\/strong> method we need to find the hypothesis h among the set of hypothesis H that maximizes the posterior probability <\/span>P(h|D).<\/em><\/strong> Now by applying Bayes theorem, we need to find the hypothesis h that maximizes the likelihood of the data D for a given h and the prior probability of h. The prior probability of the data D does not affect the maximization for finding h so that is not considered. Hence we have the MAP hypothesis as given below:<\/span><\/p>\n<\/div>\n

    \n

    \"\"<\/p>\n

     <\/p>\n

    If every hypothesis in H<\/em> is equally probable a priori, we only need to consider the likelihood of the data D<\/em> given h<\/em>, P(D|h).<\/em><\/strong> This gives rise to h<\/em>ML<\/em> or the Maximum Likelihood<\/em><\/strong>, as follows:<\/p>\n

    \"\"<\/p>\n

     <\/p>\n

    26.9.1 Bayes Example<\/strong><\/p>\n

     <\/p>\n

    Now let us look at an example of Bayes Theorem. We have data on past student performance. For each student we know the following:<\/p>\n

     <\/p>\n

    \u2013\u00a0 If student\u2019s GPA > 3.0 (G)<\/p>\n

     <\/p>\n

    \u2013\u00a0 If student had a strong math background (M)<\/p>\n

     <\/p>\n

    \u2013\u00a0 If student is a hard worker (H)<\/p>\n

     <\/p>\n

    \u2013\u00a0 If student passed or failed course<\/p>\n

     <\/p>\n

    \"\"<\/p>\n

     <\/p>\n

    We are given the data as in the Table of Figure 29.8 where p is the number of binary features. Let us assume that the probability to pass or fail are equal i.e<\/p>\n

     <\/p>\n

    P(pass)=P(fail)=0.5. Now if we are given that the student has GPA,3, has Maths, and is not a Hard worker that is the value of the features are x={0,1,0}. From the table we know that for this value of x the probability of passing is 0.05 and the probability of failing is 0.20.Therefore we can find f(x)=0.25 (Figure 29.8) which means that the classification gives the value as fail.<\/span><\/p>\n<\/div>\n

    \n

    \"\"<\/p>\n

     <\/p>\n

    Figure 29.8 Example for Bayes Theorem<\/strong><\/p>\n

     <\/p>\n

    29.10 Bayesian Classification: Why?<\/strong><\/p>\n

     <\/p>\n

    The following are the reasons why Bayesian classification s important.<\/p>\n

     <\/p>\n

    \u2022 Probabilistic learning: We can calculate explicit probabilities for hypothesis, and is among the most practical approaches to certain types of learning problems.<\/p>\n

     <\/p>\n

    \u2022 Incremental: Each training example can incrementally increase\/decrease the probability that a hypothesis is correct. Prior knowledge can also be combined with observed data.<\/p>\n

     <\/p>\n

    \u2022 Probabilistic prediction: It is possible to predict multiple hypotheses, weighted by their probabilities.<\/p>\n

     <\/p>\n

    \u2022 Standard: Even when Bayesian methods are computationally intractable, they can provide a standard of optimal decision making against which other methods can be measured<\/p>\n

     <\/p>\n

    29.10.1 Bayes Optimal Classifier<\/strong><\/p>\n

     <\/p>\n

    We will now briefly outline the different types of Bayesian learning. The forthcoming modules will discuss these methods in detail. One great advantage of Bayesian Decision Theory is that it gives us a lower bound on the classification error that can be obtained for a given problem.<\/p>\n<\/div>\n

    \n

     <\/p>\n

     <\/p>\n

    In Bayes optimal classification<\/strong> the most probable classification of a new instance is obtained by combining the predictions of all hypotheses, weighted by their posterior probabilities as given below:<\/p>\n

     <\/p>\n

    \"\"<\/p>\n

     <\/p>\n

    where V<\/em> is the set of all the values a classification can take and v<\/em>j<\/em> is one of the possible classifications from this set. Unfortunately, Bayes Optimal Classifier is usually too costly to apply! The compromise is the Na\u00efve Bayes Classifier<\/em><\/strong> which we will discuss in detail later.<\/p>\n

     <\/p>\n

    29.10.2 Bayesian Belief Networks<\/strong><\/p>\n

     <\/p>\n

    The Bayes Optimal Classifier<\/em><\/strong> is often too costly to apply. The Na\u00efve Bayes<\/em><\/strong> Classifier <\/em><\/strong>uses the conditional independence assumption to defray these costs. However, in many cases, such an assumption is overly restrictive. Bayesian belief networks <\/em><\/strong>provide an intermediate <\/em><\/strong>approach which allows stating conditional independence assumptions that apply to subsets<\/em><\/strong> of the variable.<\/p>\n

     <\/p>\n

    29.10.3 Expectation-Maximization (EM)<\/strong><\/p>\n

     <\/p>\n

    Now we will have a brief look at Expectation-Maximization approach. Consider learning a na\u00efve Bayes classifier using unlabeled<\/em> data. How can we estimate e.g. <\/em>P(A|C)? (Figure 29.9)<\/p>\n<\/div>\n

    \"\"<\/p>\n

    Figure 29.9 Dependencies of the variable A,B & C<\/strong><\/p>\n

     <\/p>\n

    Summary<\/strong><\/p>\n

      \n
    • Explained Probability, Bayes Theorem and Bayes Learning<\/li>\n
    • Discussed Bayes Learning Framework<\/li>\n
    • Outlined Maximum A Posteriori (MAP) & Maximum Likelihood (ML)<\/li>\n<\/ul>\n","protected":false},"author":3,"menu_order":28,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":[],"pb_section_license":""},"chapter-type":[],"contributor":[],"license":[],"class_list":["post-625","chapter","type-chapter","status-publish","hentry"],"part":3,"_links":{"self":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/chapters\/625","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/wp\/v2\/users\/3"}],"version-history":[{"count":5,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/chapters\/625\/revisions"}],"predecessor-version":[{"id":647,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/chapters\/625\/revisions\/647"}],"part":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/parts\/3"}],"metadata":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/chapters\/625\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/wp\/v2\/media?parent=625"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/pressbooks\/v2\/chapter-type?post=625"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/wp\/v2\/contributor?post=625"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/wp-json\/wp\/v2\/license?post=625"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}