{"id":322,"date":"2018-08-27T09:18:29","date_gmt":"2018-08-27T09:18:29","guid":{"rendered":"http:\/\/csp15.epgpbooks.inflibnet.ac.in\/?post_type=chapter&p=322"},"modified":"2019-01-02T06:56:21","modified_gmt":"2019-01-02T06:56:21","slug":"neural-networks-i","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/csp15\/chapter\/neural-networks-i\/","title":{"rendered":"Neural Networks \u2013 I"},"content":{"raw":"
\"epgp<\/a>\r\n<\/span><\/div>\r\n
\r\n\r\n \r\n\r\n \r\n\r\n \r\n

Welcome to the e-PG Pathshala Lecture Series on Machine Learning. In this and the next modules we will be discussing another important and interesting machine learning technique \u2013 Neural networks.<\/p>\r\n \r\n\r\nLearning Objectives:<\/strong>\r\n\r\n \r\n\r\nThe learning objectives of this module are as follows:\r\n\r\n \r\n

\u2022 To understand the Artificial Neural Network Classification Methods<\/p>\r\n

\u2022 To explain the working of ANN, types of ANN and its applications<\/p>\r\n

\u2022 To discuss about back propagation\u00a0 and feed forward networks.<\/p>\r\n \r\n\r\n19.1 Introduction<\/strong>\r\n\r\n \r\n

Neural networks are considered as models of biological neural networks. The main aim of neural networks was to design artificial systems that tried to mimic the intelligent and complex behaviour of the human brain. Another objective was the possibility of enhancing our own understanding of the human brain. Most neural networks are based on some sort of training rule. In other words, neural networks learn from examples similar to the way children learn to recognize objects by observing examples of the objects. They also need to have the capability of generalization so that they can predict about unseen data beyond the training data.<\/p>\r\n \r\n\r\n19.1.1 Real Neurons<\/strong>\r\n\r\n \r\n

In order to understand the working of neural networks it is necessary to have an understanding about the components of the real neuron of the human brain. We will describe the Cell structure of the neuron (Figure 19.1). It consists of basically four components. The first component namely the cell body or SOMA is responsible for processing the inputs. The second component consists of a host of fine structures called dendrites that collects the input signals and passes it to the third component which is the the axon. The neuron sends out electrical signals through the axon which splits it into thousands of branches. Finally we\u00a0have the synaptic terminals which are responsible for the electro-chemical contacts between the neurons and inhibit or excite activity of the neuron.<\/span><\/p>\r\n\r\n<\/div>\r\n\"\"\r\n\r\nStructure 19.1.2 The Neuron Metaphor<\/strong>\r\n\r\n \r\n

The main purpose of neurons is to receive, analyze and transmit further information x1,x2,x3, \u2026\u2026.xN in the form of signals (electric pulses). The multiple inputs are multiplied by weights wi1,wi2, \u2026..wiN\u00a0 and summed to get an input ui.This ui is given as input to the function to get the output yi (Figure 19.2) . There is an additional input unit that corresponds to an nonexisting attribute xo = 1. This is called the bias.<\/p>\r\n\"\"\r\n

\r\n\r\n \r\n\r\n19.2 <\/strong>Goals of Neural Computation<\/strong>\r\n\r\n \r\n

Before we go into the details of neural networks let us discuss some possible goals of neural computation. The first and possibly lofty goal is to understand how the brain actually works. As of now we know that it is a big, complex collection of cells. The second and practical goal is to understand a new style of computation that is inspired by neurons and their adaptive connections. Essentially this style of computation is different from sequential computation. This neural computation approach is good for cognitive processes such as computer vision. While sequential process is suitable for large scale computation, neural style of computation is not suited for such computation. Another goal is to carry out the learning for practical problems where the algorithms learnt are useful even they do not depict how the brain works.<\/p>\r\n \r\n\r\n19.3 Artificial Neural Network<\/strong>\r\n\r\n \r\n

Artificial neural networks or ANNs as they are called are built of densely interconnected set of simple units, each taking several real-valued inputs and producing single-valued output. ANN is characterized by its architecture, its training or learning algorithm and its activation functions. One motivation is to capture highly parallel computations on distributed processes. However most ANN software run only on sequential machines that emulate distributed processes.<\/p>\r\n \r\n\r\n19.4 Appropriate Problems for Neural Network Learning<\/strong>\r\n\r\n \r\n

Now let us discuss the type of problems that are suitable to be solved by ANNs. ANNs are well suited for problems that take noisy complex sensor data such as from cameras and microphones. Here the input instances are represented as attribute-value pair. The target output may be discrete-valued, real-valued, or a vector of several real or discrete-valued attributes. Training samples may not always be accurate and may contain errors. The long training time due to noisy complex data is generally acceptable. It is after the training phase, that fast evaluation of the target function is required. There is no need for humans to understand or interpret the learning of the target function.<\/p>\r\n \r\n\r\n19.5 Pros and Cons of Neural Networks<\/strong>\r\n\r\n \r\n\r\n19.5.1 Pros<\/strong>\r\n\r\n \r\n

Basically neural networks are suited to problems that take continuous data especially in domains with little initial knowledge. Neural networks are easy to use and basically learns by example and need very little user domain\u2010specific expertise. Neural networks have the ability to solve new kinds of problems that are difficult and sometimes impossible to explicitly define. ANNs can be used when a good functional model is not known. Neural computation provides\u00a0human characteristics to problem solving that are otherwise difficult to simulate. It is flexible and easy to maintain. Although training may take time, once trained it exhibits fast processing speed.<\/span><\/p>\r\n \r\n

ANNs are generally robust and have the ability to cope with incomplete or fuzzy data. As ANNs consist of a large number of massively interconnected parallel processing units, operating on the same problem and are fast when parallel processors are used for implementation.<\/span><\/p>\r\n\r\n<\/div>\r\n \r\n\r\nCons<\/strong>\r\n\r\n \r\n

The main disadvantage of ANNs is that they do not produce an explicit model. ANNs do not provide explanation capabilities as the results are based on connection weights which usually do not have obvious interpretations. The main disadvantage of ANNs is that the solution is not interpretable, that is it acts as a \u201cblack box\u201d. The learning during the training phase is generally slow. For achieving good generalization many input data points may be required for training.<\/p>\r\n \r\n\r\n19.6 Linear Models<\/strong>\r\n\r\n \r\n

In a simple problem of linear regression we have the training data X = {x1k },<\/p>\r\n

k=1,..,N with corresponding output Y = {yk }, k=1,..,N and our job is to find the parameters that predict the output Y from the data X in a linear fashion that is<\/p>\r\n \r\n\r\nyk \u2248 wo + w1 x1k.\r\n\r\n \r\n\r\nLet\u00a0 us\u00a0 first\u00a0 consider\u00a0 the\u00a0 general\u00a0 form of\u00a0 linear\u00a0 models\u00a0 for\u00a0 regression\u00a0 and classification as given below:\r\n\r\n\"\"\r\n\r\n \r\n

such as Gaussian functions, or sigmoid basis functions, W are the weights we are trying to learn.As we have already discussed for regression f is identity function<\/em> (in the case of Linear Regression), while for classification f is a nonlinear activation<\/em> function as specified in generalized linear regression). If the function f is sigmoid than the regression is called logistic regression and is as given below:<\/p>\r\n\"\"\r\n\r\n19.6.1 Extending Linear Models<\/strong>\r\n\r\n \r\n

The linear models that we have discussed are useful for discussing analytical and computational properties but however have limited applicability mainly due to the curse of dimensionality. For example the number of polynomial co-efficients that are needed, like finding means of Gaussians is difficult. Therefore we extend the scope by adapting basis functions \u03d5j to the parameters. This approach is suited for large scale problems.<\/p>\r\n \r\n

Both SVMs and Neural Networks address this limitation. In the case of SVM a varying number of basis functions M,<\/em> centred around training data points are used and a subset of these are selected during training. In the case of ANNs the number of basis functions M is fixed<\/em> but \u03d5j have their own parameters {wji} which are adapted appropriately during training.<\/p>\r\n\"\"\r\n

\r\n\r\n \r\n\r\n19.7 SVM versus Neural Networks<\/strong>\r\n\r\n \r\n

Now let us compare SVM and ANNs. In the case of SVM, training involves non-linear optimization. The objective function is convex, but however the optimization is straightforward usually with a single minimum. The number of basis functions to be considered is much smaller than the number of training points. SVMs produce probabilistic outputs at the expense of non-convex optimization.<\/p>\r\n \r\n

Neural Network on the other hand has only a fixed number of basis functions, and parametric forms. Multilayer perceptron uses layers of logistic regression models. ANNs also involve non-convex optimization during training (many minima). At the expense of longer periods of training, we get a more compact and faster model after the learning process is completed.<\/strong><\/p>\r\n \r\n\r\n19.8 Origin of Neural Networks<\/strong>\r\n\r\n \r\n

The origin of neural networks was based on the desire to find information processing models of biological systems. The term covers a wide range of models and most them make exaggerated claims of biological plausibility. Moreover the biological realism imposes unnecessary constraints. However neural networks are efficient models for machine learning especially multilayer perceptrons. The neural network parameters can be obtained using the\u00a0maximum likelihood framework. It is basically a nonlinear optimization problem and requires evaluating derivative of log-likelihood function w.r.t network parameters which can be performed efficiently using error back propagation.<\/span><\/p>\r\n\r\n<\/div>\r\n \r\n\r\n19.9 The Structure of the Neural Network<\/strong>\r\n\r\n \r\n

The neural network can be viewed as a generalization of linear models. Each ANN is composed of a collection of perceptrons grouped in layers. A typical and simple network consists of three layers namely input, intermediate (called the hidden layer<\/em><\/strong>) and output. Several hidden layers can be placed between the input and output layers.<\/p>\r\n \r\n\r\n19.10 Types of Neural Network<\/strong>\r\n\r\n \r\n

The simplest and widely used neural network is the Feed-forward neural network where the information travels in one direction only (Figure 19.3). In other words there is no feedback or no cycles that is it is a Directed Acyclic graph. These types of neural networks are used for pattern recognition.<\/p>\r\n\"\"\r\n

In another type of neural networks, the signals travel in both directions and the networks become complicated and dynamic (Figure 19.4). These networks have context units that act as internal memory to store a part of the inputs and use this internal memory to process arbitrary sequences of inputs. Recurrent neural networks are usually used to learn temporal patterns<\/p>\r\n \r\n\r\n\"\"\r\n\r\n19.11 Types of Neuron<\/strong>\r\n\r\n \r\n

Now let us see the different types of neurons that can be designed based on the function used. We show (Figure 19.5) only a few but potentially more functions can be designed. The simplest is the linear neuron where the function\"\"<\/p>\r\n \r\n

used is linear. Then we have the use of the sigmoid function resulting in the logistic neuron. We also show a simple step function resulting in the perceptron.<\/p>\r\n \r\n

19.12 Feed Forward Network Functions<\/strong><\/p>\r\n \r\n

A neural network can also be represented like linear models but where the basis functions are generalized. The activation function used for regression is the identity function while for classification a nonlinear function like the sigmoid function is used. The weight coefficients wj are adjusted during training. Please note that there can be several activation functions.<\/p>\r\n \r\n

The basis function \u03d5j (x) is a nonlinear function, such as tanh, and is a linear combination of D inputs and its parameters are also adjusted during training. Here f is the activation function and \u03d5j (x) is the basis function.<\/p>\r\n \r\n\r\n\"\"\r\n

Here the superscript (1) indicates that the parameters we are talking about are in the first layer of the network. The parameters wji are referred to as weights while wj0 with input 1 are referred to as biases. The quantities aj which consider the combination of D weighted inputs and the bias are known as activations.<\/p>\r\n \r\n\r\n\"\"\r\n

Each activation aj is transformed using differentiable nonlinear activation functions zj=h(aj). The zj correspond to outputs of basis functions \u03d5j(x) or the first layer of network or hidden units. The nonlinear functions h are chosen to be sigmoidal. Two examples of the above activation functions are<\/p>\r\n\r\n