{"id":40,"date":"2018-07-18T07:31:08","date_gmt":"2018-07-18T07:31:08","guid":{"rendered":"http:\/\/csp2.epgpbooks.inflibnet.ac.in\/?post_type=chapter&p=40"},"modified":"2018-08-03T06:54:35","modified_gmt":"2018-08-03T06:54:35","slug":"summarizing-performance-amdahls-law-and-benchmarks","status":"publish","type":"chapter","link":"https:\/\/ebooks.inflibnet.ac.in\/csp2\/chapter\/summarizing-performance-amdahls-law-and-benchmarks\/","title":{"rendered":"Summarizing Performance, Amdahl\u2019s law and Benchmarks"},"content":{"raw":"
\r\n\r\n \r\n

The objectives of this module are to discuss ways and means of reporting and summarizing performance, look at Amdahl\u2019s law and discuss the various benchmarks for performance evaluation.<\/p>\r\n \r\n

We've already looked at the performance equation in the earlier module. You know that the CPU execution time is the most consistent measure of performance and the CPU execution time per program is defined as the number of instructions per program multiplied by the average number of clock cycles per instruction multiplied by the clock cycle time. By improving any of these factors, by paying attention to the parameters that affect these factors, you can have an improvement in performance. Suppose you have only one processor, you can just find out the execution time with respect to that processor, with respect to a particular program, and even compare with another processor which executes the same program. But, what happens when these processors execute multiple programs? How do you summarize the performances and compare the performances? While comparing two processors, A and B, you will have to be careful in saying whether processor A is 10 times faster than or the other way round. You will have to be very careful here because wrong summary can be confusing. This module clarifies issues related to such queries.<\/p>\r\n \r\n

We know that the total execution time is a consistent measure of performance and the relative execution times for the same workload can also be informative. To show an example here for the importance of summarization, suppose if you have three different computer systems, which run two different programs, P1 and P2.Assume that computer A takes 1 second for executing P1 and 1000 seconds to execute P2. Similarly, B takes 10 seconds and 100 seconds respectively and c takes 20 seconds to execute both P1 and P2.<\/p>\r\n \r\n\r\n\"\"\r\n\r\n<\/div>\r\n \r\n

\r\n\r\nIf you look at the individual performances,\r\n\r\n \r\n\r\n\u2022 A is 10 times faster than B for program P1\r\n\r\n\u2022 A is 20 times faster than C for program P1\r\n\r\n\u2022 B is 10 times faster than A for program P2\r\n\r\n\u2022 B is\u00a0 2 times faster than\u00a0 C for program P1\r\n\r\n\u2022 C is 50 times faster than A for program P2\r\n\r\n\u2022 C is 5 times faster than B for program P2\r\n\r\n \r\n

How do you summarize and say which computer is better? You can calculate the total execution time as 1001 seconds, 110 seconds and 40 seconds and based on these values, you can make the following conclusions:<\/p>\r\n \r\n\r\n\u2022 Using total execution time:\r\n\r\n\u2013\u00a0\u00a0 B is 9.1 times faster than A\r\n\r\n\u2013\u00a0\u00a0 C is 25 times faster than A\r\n\r\n\u2013\u00a0\u00a0 C is 2.75 times faster than B\r\n\r\n \r\n

If you want a single number to summarize performance, you can take the average of the execution timesand try to come up with one value which will summarize the performance. The average execution time is the summation of all the times divided by the total number of programs. For the given example, we have<\/p>\r\n \r\n\r\nAvg(A) = 500.5\r\n\r\nAvg(B) = 55\r\n\r\nAvg(C) = 20\r\n\r\n \r\n

Here again, if you're trying to make a judgementbased on this, you have a problem because you know that P1 and P2 are not run equal number of times. So that may mislead your summary. Therefore, you could assign weights per program. Weighted arithmetic mean summarizes performance while tracking the execution time. Weights can adjust for different running times, balancing the contribution of each program.<\/p>\r\n \r\n

One more issue that has to be mentioned before we go deeper into summarization of performance is the question of the types of programs chosen for evaluation. Each person can pick his own programs and report performance. And this is obviously not correct. So, we normally look at a set of programs, a benchmark suite to evaluate performance. Later in this module, we shall discuss in detail about benchmarks. One of the most popular benchmarks is the SPEC (Standard Performance Evaluation Corporation) benchmark. So the evaluation of processors is done with respect to this.<\/p>\r\n \r\n

Even with the benchmark suite, and considering weighted arithmetic mean, the problem would be how to pick weights; since SPEC is a consortium of competing companies, each company might have their own favorite set of weights, which would make it hard to reach consensus. One approach is to use weights that make all\u00a0programs execute an equal time on some reference computer, but this biases the results to the performance characteristics of the reference computer.<\/span><\/p>\r\n \r\n

Rather than pick weights, we could normalize execution times to a referencecomputer by dividing the time on the reference computer by the time on the computerbeing rated, yielding a ratio proportional to performance. SPEC uses thisapproach, calling the ratio the SPECRatio. For example, suppose that theSPECRatio of computer A on a benchmark was 1.25 times higher than computerB; then you would know,<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n\r\n\"\"\r\n

Also observe that the execution times on the reference computer drop out and thechoice of the reference computer is irrelevant when the comparisons are made asa ratio.Because a SPECRatio is a ratio rather than an absolute execution time, themean must be computed using the geometric<\/em> mean. (Since SPECRatios have nounits, comparing SPECRatios arithmetically is meaningless.) The formula is<\/p>\r\n \r\n\r\n\"\"\r\n\r\n \r\n\r\nUsing the geometricmean ensures two important properties:\r\n

1.\u00a0 The geometric mean of the ratios is the same as the ratio of the geometricmeans.<\/p>\r\n \r\n

2.\u00a0 The ratio of the geometric means is equal to the geometric mean of the performance ratios, which implies that the choice of the reference computer isirrelevant.<\/p>\r\n \r\n

Hence, the motivations to use the geometric mean are substantial, especiallywhen we use performance ratios to make comparisons.<\/p>\r\n \r\n

Now that we have seen how to define, measure, and summarize performance, we shall explore certain guidelines and principles that areuseful in the design and analysis of computers. This section introduces importantobservations about design, as well as an important equation to evaluate alternatives. The most important observations are:<\/p>\r\n \r\n

1. Take advantage of parallelism : We have already discussed the different types of parallelism that exist in applications and how architecture designers should exploit them. To mention a few \u2013 having multiple processors, having multiple threads of execution, having multiple execution units to exploit the data level parallelism, exploiting ILP through pipelining.<\/p>\r\n \r\n

2. Principle of Locality: This comes from the properties of programs. A widely held rule of thumb is that a program spends 90% of its execution time in only 10% of the code. This implies that we can predict with reasonable accuracy what instructions and data a program will use in the near future based on its accesses in the recent past. Two different types of locality have been observed. <\/span>Temporal<\/em> locality <\/em>states that recently accessed items are likely to be accessed in the near future. <\/span>Spatial locality<\/em> says that items whose addresses are near one another tend to be referenced close together in time.<\/span><\/p>\r\n \r\n

3. Focus on the common case: This is one of the most important principles of computer architecture. While making design choices among various alternatives, always favour the most frequent case. Focusing on the common case works for power as well as for resource allocationand performance. For example, when performing the addition of two numbers, there might be overflow, but obviously not so frequent. Optimize the addition operation without overflow. When overflow occurs, it might slow down the processor, but it is only a rare occurrence and you can afford it.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n

In applying the simple principle of focusing on the common cases, we have to decide what the frequent case is and how much performancecan be improved by making that case faster. A fundamental law, calledAmdahl\u2019s Law,<\/em> can be used to quantify this principle.Gene Amdahl, chief architect of IBM's first mainframe series and founder of Amdahl Corporation and other companies found that there were some fairly stringent restrictions on how much of a speedup one could get for a given enhancement. These observations were wrapped up in Amdahl's Law<\/em>. It basically states that the performance enhancement possible with a given improvement is limited by the amount that the improved feature is used. For example, you have a floatingpoint unit and you try to speed up the floatingpoint unit many times, say a 10 X speedup, but all this is going to matter only the if the floatingpoint unit is going to be used very frequently. If your program does not have floatingpoint instructions at all, then there is no point in increasing the speed of the floatingpoint unit.<\/p>\r\n \r\n\r\nThe performance gain from improving some portion of\u00a0 a computer is calculated by:\r\n\r\n \r\n\r\n\"\"\r\n\r\n \r\n

Amdahl\u2019s Law gives us a quick way to find the speedup from some enhancement,which depends on two factors:<\/p>\r\n \r\n

1.\u00a0<\/span>The fraction of the computation time in the original computer that can be converted to take advantage of the enhancement - <\/em>For example, if 30 seconds of the execution time of a program that takes 60 seconds in total can use an enhancement, the fraction is 30\/60. This value, which we will call Fractionenhanced, is always less than or equal to 1.<\/span><\/p>\r\n\r\n<\/div>\r\n

\r\n\r\n \r\n

2.\u00a0The improvement gained by the enhanced execution mode; that is, how much faster the task would run if the enhanced mode were used for the entire program - <\/em>This value is the time of the original mode over the time of the enhanced mode. If the enhanced mode takes, say, 3 seconds for a portion of the program, while it is 6 seconds in the original mode, the improvement is 6\/3. We will call this value, which is always greater than 1, Speedupenhanced.<\/p>\r\n \r\n\r\n\"\"\r\n\r\n \r\n

That is, suppose you define an enhancement that accelerates a fraction F of the execution time by a factor S and the remainder of the time is unaffected, then you can say, execution time with the enhancement is equal to 1 minus F, where F is the fraction of the execution time for which the enhancement is active, plus F by S, multiplied by the execution time without the enhancement. Speedup is going to be execution time without the enhancement divided by the execution time with enhancement. This is as shown above.<\/p>\r\n \r\n

Amdahl\u2019s Law can serve as a guide to how much an enhancement willimprove performance and how to distribute resources to improve cost - performance.The goal, clearly, is to spend resources proportional to where timeis spent. Amdahl\u2019s Law is particularly useful for comparing the overall systemperformance of two alternatives, but it can also be applied to compare two processordesign alternatives.<\/p>\r\n \r\n\r\nConsider the following example to illustrate Amdahl\u2019s law.\r\n\r\nFor the RISC machine with the following instruction mix:\r\n\r\n \r\n\r\n<\/div>\r\n\"\"\r\n

\r\n\r\nCPI = 2.2\r\n\r\n \r\n\r\nIf a CPU design enhancement improves the CPI of load instructions from 5 to 2, what isthe resulting performance improvement from this enhancement?\r\n\r\n \r\n\r\nFraction enhanced = F =\u00a0 45% or\u00a0 .45\r\n\r\n \r\n\r\nUnaffected fraction = 1- F = 100% - 45% = 55% or .55\r\n\r\n \r\n\r\nFactor of enhancement = S =\u00a0 5\/2 =\u00a0 2.5\r\n\r\n \r\n\r\nUsing Amdahl\u2019s Law:\r\n\r\n \r\n\r\n\"\"\r\n\r\nYou can also alternatively calculate using the CPU performance equation.\r\n\r\n \r\n\r\nOld CPI = 2.2\r\n\r\nNew CPI =\u00a0 .5 x 1 + .2 x 2 +\u00a0 .1 x 3 + .2 x 2\u00a0 = 1.6\r\n\r\nSpeed up = 2.2 \/ 1.6 = 1.37\r\n\r\n \r\n\r\nwhich is the same speedup obtained from Amdahl\u2019s Law.\r\n\r\n \r\n

The same concept can be extended even when there are multiple enhancements. Suppose three CPU performance enhancements are proposed with the following speedups and percentage of the code execution time affected:<\/p>\r\n \r\n\r\n\"\"\r\n

While all three enhancements are in place in the new design, each enhancement affects a different portion of the code and only one enhancement can be used at a time. What is the resulting overall speedup?<\/p>\r\n \r\n\r\n\"\"\r\n\r\nObserve that:\r\n\r\n \r\n

\u2022 The performance of any system is constrained by the speed or capacity of the slowest point.<\/p>\r\n\r\n