{"id":257,"date":"2018-07-19T10:33:27","date_gmt":"2018-07-19T10:33:27","guid":{"rendered":"http:\/\/csp4.epgpbooks.inflibnet.ac.in\/?post_type=chapter&#038;p=257"},"modified":"2018-08-01T11:18:52","modified_gmt":"2018-08-01T11:18:52","slug":"why-do-we-need-to-pre-process-the-data","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/chapter\/why-do-we-need-to-pre-process-the-data\/","title":{"rendered":"WHY DO WE NEED TO PRE PROCESS THE DATA?"},"content":{"raw":"<div>\r\n\r\n<strong>WHY DO WE NEED TO PREPROCESS THE DATA?<\/strong>\r\n\r\n&nbsp;\r\n<p style=\"text-align: justify\">Much\u00a0 of\u00a0 the\u00a0 raw\u00a0\u00a0 data\u00a0 contained\u00a0 in\u00a0\u00a0 databases\u00a0\u00a0 is\u00a0\u00a0 unpreprocessed,\u00a0 incomplete,\u00a0\u00a0 and\u00a0\u00a0 noisy.For example, the databases may contain:Fields that are obsolete or redundant. Missing values Outliers.Data in a form not suitable for data mining models. Values not consistent with policy or common sense.To be useful for data mining purposes, the databases need to undergo preprocessing, in the form of <em>data cleaning<\/em> and <em>data transformation<\/em>. Data mining often deals with data that hasn\u2019t been looked at for years, so that much of the data contains field values that have expired, are no longer relevant, or are simply missing. The overriding objective is to <em>minimize GIGO<\/em>: to minimize the \u201cgarbage\u201d that gets into our model so that we can minimize the amount of garbage that our models give out. Dorian Pyle, in his book <em>Data Preparation for Data Mining<\/em> , estimates that data preparation alone accounts for 60% of all the time and effort expanded in the entire data mining process.<\/p>\r\n&nbsp;\r\n\r\n<strong>DATA CLEANING<\/strong>\r\n\r\n&nbsp;\r\n<p style=\"text-align: justify\">To illustrate the need to clean up data, let\u2019s take a look at some of the types of errors that could creep into even a tiny data set, such as that in Table 1. Let\u2019s discuss, attribute by attribute, some of the problems that have found their way into the data set in Table 1. The <em>customer ID<\/em> variable seems to be fine. What about <em>zip<\/em>?<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">Let\u2019sassumethatweareexpectingallofthecustomersinthedatabasetohavethe usual five-numeral U.S. zip code. Now, customer 1002 has this strange (to American eyes) zip code of <em>J2S7K7<\/em>. If we were not careful, we might be tempted to classify this unusual value as an error and toss it out, until we stop to think that not all countries use the same zip code format. Actually, this is the zip code of St. Hyancinthe, Quebec, Canada, so probably represents real data from a real customer. What has evidently occurred is that a French-Canadian customer has made a purchase and put their home zip code down in the field required. Especially in this era of the North American Free Trade Agreement, we must be ready to expect unusual values in fields such as zip codes, which vary from country to country. What about the zip code for customer 1004? We are unaware of any countries that have four-digit zip codes, such as the <em>6269<\/em> indicated here, so this must be an error, right? Probably not. Zip codes for the New England states begin with the numeral 0.<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">Unless the zip code field is defined to be character (text) and not numeric, the software will probably chop off the leading zero, which is apparently what happened here. The zipcodeisprobably 06269, which refers to Storrs, Connecticut, home of the University of Connecticut. The next field, gender, contains a missing value for customer 1003. The income field, which we assume is measuring annual gross income, has three potentially\u00a0<span style=\"text-align: initial;font-size: 1em\">anomalous values. First, customer 1003 is shown as having an income of Rs.10,000,000 per year. Although entirely possible, especially when considering the customer\u2019s zip code (90210, Beverly Hills), this value of income is nevertheless an outlier, an extreme data value. Certain statistical and data mining modeling techniques do not function smoothly in the presence of outliers;<\/span><\/p>\r\n\r\n<\/div>\r\n<div>\r\n\r\nTable 1: Can You Find Any Problems in This Tiny Data Set?\r\n<p style=\"text-align: center\"><img class=\"size-full wp-image-258 aligncenter\" src=\"http:\/\/csp4.epgpbooks.inflibnet.ac.in\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124.png\" alt=\"\" width=\"623\" height=\"173\" \/><\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">Poverty is one thing, but it is rare to find an income that is negative, as our poor customer 1004 has. Unlike customer 1003\u2019s income, customer 1004\u2019s reported income of -Rs.40,000 lies beyond the field bounds for income and therefore must be an error. It is unclear how this error crept in, with perhaps the most likely explanation being that the negative sign is a stray data entry error. However, we cannot be sure and should approach this value cautiously, attempting to communicate with the database manager most familiar with the database history. So what is wrong with customer 1005\u2019s income of Rs.99,999? Perhaps nothing; it may in fact be valid. But if all the other incomes are rounded to the nearest Rs.5000, why the precision with customer 1005? Often, in legacy databases, certain specified values are meant to be codes for anomalous entries, such as missing values. Perhaps 99999 was coded in an old database to mean missing. Again, we cannot be sure and should again refer to the \u201cwetware.\u201d<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">Finally, are we clear as to which unit of measure the income variable is measured in? Databases often get merged, sometimes without bothering to check whether such merges are entirely appropriate for all fields. For example, it is quite possible that customer 1002, with the Canadian zip code, has an income measured in Canadian dollars, not U.S. dollars.<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">The age field has a couple of problems. Although all the other customers have numerical values for age, customer 1001\u2019s \u201cage\u201d of C probably reflects an earlier categorization of this man\u2019s age into a bin labeled C. The data mining software will definitely not like this categorical value in an otherwise numerical field, and we will have to resolve this problem somehow. How about customer 1004\u2019s age of 0? Perhaps there is a newborn male living in Storrs, Connecticut, who has made a transaction of Rs.1000.<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">More likely, the age of this person is probably missing and was coded as 0 to indicate this or some other anomalous condition (e.g., refused to provide the age information). Of course, keeping an age field in a database is a minefield in itself, since the passage of time will quickly make the field\u00a0<span style=\"text-align: initial;font-size: 1em\">values obsolete and misleading. It is better to keep date-type fields (such as birthdate) in a database, since these are constant and may be transformed into ages when needed.<\/span><\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\"><span style=\"text-align: initial;font-size: 1em\">The marital status field seems fine, right? Maybe not. The problem lies in the meaning behind these symbols. We all think we know what these symbols mean, but are sometimes surprised. For example, if you are in search of cold water in a rest room in Montreal and turn on the faucet marked C, you may be in for a surprise, since the C stands for chaud, which is French for hot. There is also the problem of ambiguity. In Table 1, for example, does the S for customers 1003 and 1004 stand for single or separated? The transaction amount field seems satisfactory as long as we are confident that we know what unit of measure is being used and that all records are transacted in this unit.<\/span><\/p>\r\n\r\n<\/div>\r\n<div>\r\n\r\n<strong>\u00a0 \u00a0<\/strong>\r\n\r\n<strong> HANDLING MISSING DATA<\/strong>\r\n\r\n&nbsp;\r\n<p style=\"text-align: justify\">Missing data is a problem that continues to plague data analysis methods. Even as our analysis methods gain sophistication, we continue to encounter missing values in fields, especially in databases with a large number of fields. The absence of information is rarely beneficial. All things being equal, more data is almost always better. Therefore, we should think carefully about how we handle the thorny issue of missing data. To help us tackle this problem, we will introduce ourselves to a new data set, the cars data set, originally compiled by Barry Becker and Ronny Kohavi of Silicon Graphics, and available at the SGI online data repository at www.sgi.com\/tech\/mlc\/db. The data set, also available on the book series Web site accompanying the text, consists of information about 261 automobiles manufactured in the 1970s and 1980s, including gas mileage, number of cylinders, cubic inches, horsepower, and so on.<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">A common method of handling missing values is simply to omit from the analysis the records or fields with missing values. However, this may be dangerous, since the pattern of missing values may in fact be systematic, and simply deleting records with missing values would lead to a biased subset of the data. Further, it seems like a waste to omit the information in all the other fields, just because one field value is missing. Therefore, data analysts have turned to methods that would replace the missing value with a value substituted according to various criteria. Insightful Miner offers a choice of replacement values for missing data:<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">1.\u00a0 Replace the missing value with some constant, specified by the analyst.<\/p>\r\n<p style=\"text-align: justify\">2. Replace the missing value with the field mean (for numerical variables) or the mode (for categorical variables).<\/p>\r\n<p style=\"text-align: justify\">3. Replace the missing values with a value generated at random from the variable distribution observed.<\/p>\r\n&nbsp;\r\n\r\n<strong>IDENTIFYING MISCLASSIFICATIONS\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/strong>\r\n\r\n&nbsp;\r\n<p style=\"text-align: justify\">Let us look at an example of checking the classification labels on the categorical variables, to make sure that they are all valid and consistent. One of the functions of Insightful Miner\u2019s missing\u00a0 values\u00a0 node is\u00a0 to\u00a0 display\u00a0 a\u00a0 frequency distribution\u00a0 of\u00a0\u00a0<span style=\"text-align: initial;font-size: 1em\">the categorical variables available. For example, the frequency distribution of the categorical variable origin, where Insightful Miner\u2019s missing values node is applied to the cars data set, classes: India, USA, France, US, Europe, and Japan. However, two of the classes, USA and France, have a count of only one automobile each. What is clearly happening here is that two of the records have been classified inconsistently with respect to the origin of manufacture. To maintain consistency with the remainder of the data set, the record with origin USA should have been labeled US, and the record with origin France should have been labeled Europe.<\/span><\/p>\r\n\r\n<\/div>\r\nMETHODS FOR IDENTIFYING OUTLIERS\r\n\r\n&nbsp;\r\n\r\nOutliers are extreme\u00a0 values\u00a0 that\u00a0 lie\u00a0 near\u00a0 the\u00a0 limits\u00a0 of the\u00a0 data range or go\u00a0 against the\u00a0 trend\u00a0 of\u00a0 the\u00a0 remaining\u00a0 data.\u00a0 Identifying outliers is important because they may represent errors in data\u00a0 entry.\u00a0 Also,\u00a0 even\u00a0 if\u00a0 an\u00a0 outlier is\u00a0 a\u00a0 valid\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 data point\u00a0 and\u00a0 not in\u00a0 error,\u00a0 certain\u00a0 statistical\u00a0 methods\u00a0 are\u00a0 sensitive\u00a0 to\u00a0 the\u00a0 presence of outliers and may deliver\u00a0 unstable\u00a0 results.\u00a0 Neural\u00a0 networks\u00a0 benefit\u00a0 from\u00a0 normalization,\u00a0 \u00a0 \u00a0 \u00a0as do algorithms that make use of distance measures, such as the k-nearest neighbor algorithm.\r\n\r\n&nbsp;\r\n\r\n<strong>DATA TRANSFORMATION<\/strong>\r\n\r\n&nbsp;\r\n<p style=\"text-align: justify\">Variables\u00a0 tend\u00a0 to\u00a0 have\u00a0 ranges\u00a0 that\u00a0 vary\u00a0 greatly\u00a0 from\u00a0 each\u00a0 other.\u00a0 For\u00a0 example,\u00a0 if\u00a0 we\u00a0 are interested in major league baseball, players\u2019 batting averages\u00a0 will\u00a0 range\u00a0 from\u00a0 zero\u00a0 to less\u00a0 than\u00a0 0.400,\u00a0 while\u00a0 the number\u00a0 of\u00a0 home\u00a0 runs\u00a0 hit\u00a0 in\u00a0\u00a0\u00a0 a\u00a0 season\u00a0 will\u00a0 range\u00a0 from\u00a0 zero to\u00a0 around 70.\u00a0 For some data\u00a0 mining\u00a0 algorithms,\u00a0 such differences\u00a0 in\u00a0 the\u00a0 ranges\u00a0 will\u00a0lead to a tendency for the variable with greater range to have undue influence on the results.<\/p>\r\n&nbsp;\r\n<p style=\"text-align: justify\">Therefore, data miners should normalize their numerical variables, to standardize the scale of effect each variable has on the results. There are several techniques for normalization, and we shall examine two of the more prevalent methods. Let X refer to our original field value and X \u2217 refer to the normalized field value.<\/p>\r\n&nbsp;\r\n<p style=\"text-align: center\"><img class=\"size-full wp-image-259 aligncenter\" src=\"http:\/\/csp4.epgpbooks.inflibnet.ac.in\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125.png\" alt=\"\" width=\"630\" height=\"356\" \/><\/p>","rendered":"<div>\n<p><strong>WHY DO WE NEED TO PREPROCESS THE DATA?<\/strong><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Much\u00a0 of\u00a0 the\u00a0 raw\u00a0\u00a0 data\u00a0 contained\u00a0 in\u00a0\u00a0 databases\u00a0\u00a0 is\u00a0\u00a0 unpreprocessed,\u00a0 incomplete,\u00a0\u00a0 and\u00a0\u00a0 noisy.For example, the databases may contain:Fields that are obsolete or redundant. Missing values Outliers.Data in a form not suitable for data mining models. Values not consistent with policy or common sense.To be useful for data mining purposes, the databases need to undergo preprocessing, in the form of <em>data cleaning<\/em> and <em>data transformation<\/em>. Data mining often deals with data that hasn\u2019t been looked at for years, so that much of the data contains field values that have expired, are no longer relevant, or are simply missing. The overriding objective is to <em>minimize GIGO<\/em>: to minimize the \u201cgarbage\u201d that gets into our model so that we can minimize the amount of garbage that our models give out. Dorian Pyle, in his book <em>Data Preparation for Data Mining<\/em> , estimates that data preparation alone accounts for 60% of all the time and effort expanded in the entire data mining process.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>DATA CLEANING<\/strong><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">To illustrate the need to clean up data, let\u2019s take a look at some of the types of errors that could creep into even a tiny data set, such as that in Table 1. Let\u2019s discuss, attribute by attribute, some of the problems that have found their way into the data set in Table 1. The <em>customer ID<\/em> variable seems to be fine. What about <em>zip<\/em>?<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Let\u2019sassumethatweareexpectingallofthecustomersinthedatabasetohavethe usual five-numeral U.S. zip code. Now, customer 1002 has this strange (to American eyes) zip code of <em>J2S7K7<\/em>. If we were not careful, we might be tempted to classify this unusual value as an error and toss it out, until we stop to think that not all countries use the same zip code format. Actually, this is the zip code of St. Hyancinthe, Quebec, Canada, so probably represents real data from a real customer. What has evidently occurred is that a French-Canadian customer has made a purchase and put their home zip code down in the field required. Especially in this era of the North American Free Trade Agreement, we must be ready to expect unusual values in fields such as zip codes, which vary from country to country. What about the zip code for customer 1004? We are unaware of any countries that have four-digit zip codes, such as the <em>6269<\/em> indicated here, so this must be an error, right? Probably not. Zip codes for the New England states begin with the numeral 0.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Unless the zip code field is defined to be character (text) and not numeric, the software will probably chop off the leading zero, which is apparently what happened here. The zipcodeisprobably 06269, which refers to Storrs, Connecticut, home of the University of Connecticut. The next field, gender, contains a missing value for customer 1003. The income field, which we assume is measuring annual gross income, has three potentially\u00a0<span style=\"text-align: initial;font-size: 1em\">anomalous values. First, customer 1003 is shown as having an income of Rs.10,000,000 per year. Although entirely possible, especially when considering the customer\u2019s zip code (90210, Beverly Hills), this value of income is nevertheless an outlier, an extreme data value. Certain statistical and data mining modeling techniques do not function smoothly in the presence of outliers;<\/span><\/p>\n<\/div>\n<div>\n<p>Table 1: Can You Find Any Problems in This Tiny Data Set?<\/p>\n<p style=\"text-align: center\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-258 aligncenter\" src=\"http:\/\/csp4.epgpbooks.inflibnet.ac.in\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124.png\" alt=\"\" width=\"623\" height=\"173\" srcset=\"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124.png 623w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124-300x83.png 300w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124-65x18.png 65w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124-225x62.png 225w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-124-350x97.png 350w\" sizes=\"auto, (max-width: 623px) 100vw, 623px\" \/><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Poverty is one thing, but it is rare to find an income that is negative, as our poor customer 1004 has. Unlike customer 1003\u2019s income, customer 1004\u2019s reported income of -Rs.40,000 lies beyond the field bounds for income and therefore must be an error. It is unclear how this error crept in, with perhaps the most likely explanation being that the negative sign is a stray data entry error. However, we cannot be sure and should approach this value cautiously, attempting to communicate with the database manager most familiar with the database history. So what is wrong with customer 1005\u2019s income of Rs.99,999? Perhaps nothing; it may in fact be valid. But if all the other incomes are rounded to the nearest Rs.5000, why the precision with customer 1005? Often, in legacy databases, certain specified values are meant to be codes for anomalous entries, such as missing values. Perhaps 99999 was coded in an old database to mean missing. Again, we cannot be sure and should again refer to the \u201cwetware.\u201d<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Finally, are we clear as to which unit of measure the income variable is measured in? Databases often get merged, sometimes without bothering to check whether such merges are entirely appropriate for all fields. For example, it is quite possible that customer 1002, with the Canadian zip code, has an income measured in Canadian dollars, not U.S. dollars.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">The age field has a couple of problems. Although all the other customers have numerical values for age, customer 1001\u2019s \u201cage\u201d of C probably reflects an earlier categorization of this man\u2019s age into a bin labeled C. The data mining software will definitely not like this categorical value in an otherwise numerical field, and we will have to resolve this problem somehow. How about customer 1004\u2019s age of 0? Perhaps there is a newborn male living in Storrs, Connecticut, who has made a transaction of Rs.1000.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">More likely, the age of this person is probably missing and was coded as 0 to indicate this or some other anomalous condition (e.g., refused to provide the age information). Of course, keeping an age field in a database is a minefield in itself, since the passage of time will quickly make the field\u00a0<span style=\"text-align: initial;font-size: 1em\">values obsolete and misleading. It is better to keep date-type fields (such as birthdate) in a database, since these are constant and may be transformed into ages when needed.<\/span><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\"><span style=\"text-align: initial;font-size: 1em\">The marital status field seems fine, right? Maybe not. The problem lies in the meaning behind these symbols. We all think we know what these symbols mean, but are sometimes surprised. For example, if you are in search of cold water in a rest room in Montreal and turn on the faucet marked C, you may be in for a surprise, since the C stands for chaud, which is French for hot. There is also the problem of ambiguity. In Table 1, for example, does the S for customers 1003 and 1004 stand for single or separated? The transaction amount field seems satisfactory as long as we are confident that we know what unit of measure is being used and that all records are transacted in this unit.<\/span><\/p>\n<\/div>\n<div>\n<p><strong>\u00a0 \u00a0<\/strong><\/p>\n<p><strong> HANDLING MISSING DATA<\/strong><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Missing data is a problem that continues to plague data analysis methods. Even as our analysis methods gain sophistication, we continue to encounter missing values in fields, especially in databases with a large number of fields. The absence of information is rarely beneficial. All things being equal, more data is almost always better. Therefore, we should think carefully about how we handle the thorny issue of missing data. To help us tackle this problem, we will introduce ourselves to a new data set, the cars data set, originally compiled by Barry Becker and Ronny Kohavi of Silicon Graphics, and available at the SGI online data repository at www.sgi.com\/tech\/mlc\/db. The data set, also available on the book series Web site accompanying the text, consists of information about 261 automobiles manufactured in the 1970s and 1980s, including gas mileage, number of cylinders, cubic inches, horsepower, and so on.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">A common method of handling missing values is simply to omit from the analysis the records or fields with missing values. However, this may be dangerous, since the pattern of missing values may in fact be systematic, and simply deleting records with missing values would lead to a biased subset of the data. Further, it seems like a waste to omit the information in all the other fields, just because one field value is missing. Therefore, data analysts have turned to methods that would replace the missing value with a value substituted according to various criteria. Insightful Miner offers a choice of replacement values for missing data:<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">1.\u00a0 Replace the missing value with some constant, specified by the analyst.<\/p>\n<p style=\"text-align: justify\">2. Replace the missing value with the field mean (for numerical variables) or the mode (for categorical variables).<\/p>\n<p style=\"text-align: justify\">3. Replace the missing values with a value generated at random from the variable distribution observed.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>IDENTIFYING MISCLASSIFICATIONS\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/strong><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Let us look at an example of checking the classification labels on the categorical variables, to make sure that they are all valid and consistent. One of the functions of Insightful Miner\u2019s missing\u00a0 values\u00a0 node is\u00a0 to\u00a0 display\u00a0 a\u00a0 frequency distribution\u00a0 of\u00a0\u00a0<span style=\"text-align: initial;font-size: 1em\">the categorical variables available. For example, the frequency distribution of the categorical variable origin, where Insightful Miner\u2019s missing values node is applied to the cars data set, classes: India, USA, France, US, Europe, and Japan. However, two of the classes, USA and France, have a count of only one automobile each. What is clearly happening here is that two of the records have been classified inconsistently with respect to the origin of manufacture. To maintain consistency with the remainder of the data set, the record with origin USA should have been labeled US, and the record with origin France should have been labeled Europe.<\/span><\/p>\n<\/div>\n<p>METHODS FOR IDENTIFYING OUTLIERS<\/p>\n<p>&nbsp;<\/p>\n<p>Outliers are extreme\u00a0 values\u00a0 that\u00a0 lie\u00a0 near\u00a0 the\u00a0 limits\u00a0 of the\u00a0 data range or go\u00a0 against the\u00a0 trend\u00a0 of\u00a0 the\u00a0 remaining\u00a0 data.\u00a0 Identifying outliers is important because they may represent errors in data\u00a0 entry.\u00a0 Also,\u00a0 even\u00a0 if\u00a0 an\u00a0 outlier is\u00a0 a\u00a0 valid\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 data point\u00a0 and\u00a0 not in\u00a0 error,\u00a0 certain\u00a0 statistical\u00a0 methods\u00a0 are\u00a0 sensitive\u00a0 to\u00a0 the\u00a0 presence of outliers and may deliver\u00a0 unstable\u00a0 results.\u00a0 Neural\u00a0 networks\u00a0 benefit\u00a0 from\u00a0 normalization,\u00a0 \u00a0 \u00a0 \u00a0as do algorithms that make use of distance measures, such as the k-nearest neighbor algorithm.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>DATA TRANSFORMATION<\/strong><\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Variables\u00a0 tend\u00a0 to\u00a0 have\u00a0 ranges\u00a0 that\u00a0 vary\u00a0 greatly\u00a0 from\u00a0 each\u00a0 other.\u00a0 For\u00a0 example,\u00a0 if\u00a0 we\u00a0 are interested in major league baseball, players\u2019 batting averages\u00a0 will\u00a0 range\u00a0 from\u00a0 zero\u00a0 to less\u00a0 than\u00a0 0.400,\u00a0 while\u00a0 the number\u00a0 of\u00a0 home\u00a0 runs\u00a0 hit\u00a0 in\u00a0\u00a0\u00a0 a\u00a0 season\u00a0 will\u00a0 range\u00a0 from\u00a0 zero to\u00a0 around 70.\u00a0 For some data\u00a0 mining\u00a0 algorithms,\u00a0 such differences\u00a0 in\u00a0 the\u00a0 ranges\u00a0 will\u00a0lead to a tendency for the variable with greater range to have undue influence on the results.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: justify\">Therefore, data miners should normalize their numerical variables, to standardize the scale of effect each variable has on the results. There are several techniques for normalization, and we shall examine two of the more prevalent methods. Let X refer to our original field value and X \u2217 refer to the normalized field value.<\/p>\n<p>&nbsp;<\/p>\n<p style=\"text-align: center\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-259 aligncenter\" src=\"http:\/\/csp4.epgpbooks.inflibnet.ac.in\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125.png\" alt=\"\" width=\"630\" height=\"356\" srcset=\"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125.png 630w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125-300x170.png 300w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125-65x37.png 65w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125-225x127.png 225w, https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-content\/uploads\/sites\/47\/2018\/07\/a2-125-350x198.png 350w\" sizes=\"auto, (max-width: 630px) 100vw, 630px\" \/><\/p>\n","protected":false},"author":4,"menu_order":34,"template":"","meta":{"pb_show_title":"on","pb_short_title":"","pb_subtitle":"","pb_authors":["dr-r-baskaran"],"pb_section_license":""},"chapter-type":[],"contributor":[58],"license":[],"class_list":["post-257","chapter","type-chapter","status-publish","hentry","contributor-dr-r-baskaran"],"part":3,"_links":{"self":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/chapters\/257","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/wp\/v2\/users\/4"}],"version-history":[{"count":4,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/chapters\/257\/revisions"}],"predecessor-version":[{"id":441,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/chapters\/257\/revisions\/441"}],"part":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/parts\/3"}],"metadata":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/chapters\/257\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/wp\/v2\/media?parent=257"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/pressbooks\/v2\/chapter-type?post=257"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/wp\/v2\/contributor?post=257"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/ebooks.inflibnet.ac.in\/csp4\/wp-json\/wp\/v2\/license?post=257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}