For a computer that took a variable number of clock cycles to execute floating point instructions, what would be the formula to estimate its performance in FLOPS assuming that it can execute either A add instructions, or M multiply instructions, or D divide instructions per second in average?

Is it reasonable to use just the value of M, as was done in the first line of the table, using "About 2400 IBM 7030 Stretch supercomputers [...] IBM 7030 Stretch performs one floating-point multiply every 2.4 microseconds." to represent the performance of 1 GFLOPS, or were there better formulas?

Also, converting from Whetstones to FLOPS appears unreliable. The ratio of MWIPS to MFLOPS in this table varies substantially even for the same family of processors.

I don't care that In the past, FLOPS was considered a marketing term and thus subject to rather over-optimistic reporting, as mentioned in an answer below. I'm asking how it was computed.

  • 2
    FLOPS is easy to trick into exactly say what you want, especially if the computer you are measuring has no native floating point unit as it very often was the case in the early days. Simply count the number of (arbitrary) floating point operations your computer can do in a second - This would, however, vary wildly with the number of significant digits, the precision of your computations and even with the input data (it's much more easy to divide 4.00 by 2.00 than 10.00 / 3.14). Some kind of "agreed standard" was linpack (which still could be tweaked) as pointed out in @scruss' answer below. – tofro Jul 6 '17 at 19:08
  • @tofro But it won't be easy to sell that number to anyone knowledgeable enough to ask how exactly was that number achieved. One way to compute FLOPS would be to use the average latency of multiplication as the representative operation (because additions are somewhat faster, divisions are somewhat slower, so they cancel each other out). Another way could be to say, well, if you mostly compute polynomials using the Horner scheme, your FLOPS would be mostly determined by the mean of the latency of multiplication and the latency of addition, etc. – Leo B. Jul 6 '17 at 19:19
  • But it's easy to compute something and write it on glossy marketing paper. Linpack at least calculates FLOPS by doing something useful: Solving linear equation systems using Gaussian elimination. Even then, the compiler you use to translate linpack can have very significant impact on what you get as a result. – tofro Jul 6 '17 at 19:24
  • @tofro True. Then, as far as the time spent performing Gaussian elimination is overwhelmingly dominated by the floating point instructions or routines, as was the case for retro bit-serial CPUs or CPUs without an FPU, the FLOPS number would be correlated highly enough with the (average) latency of "FADD", "FMUL" and "FDIV", that a formula could be derived. – Leo B. Jul 6 '17 at 20:38
  • The latency of each FP op doesn't make much difference if there is a lot a parallism (e.g. 1000 vector units). – hotpaw2 Jul 7 '17 at 1:05

The TOP500 project uses the Linpack Benchmark to determine FLOPS ratings. In the past, FLOPS was considered a marketing term and thus subject to rather over-optimistic reporting.

  • Nowadays, when the f. p. throughput depends mostly on memory bandwidth, cache size and policy, and branch prediction algorithms, it makes sense. in the old days it were the actual f. p. instructions that determined the performance. There must have been some conventions even to marketing reporting. – Leo B. Jul 6 '17 at 15:44
  • Your answer is skirting the question. How exactly was FLOPS computed in the past to be reported over-optimistically as a marketing term? – Leo B. Jul 6 '17 at 17:15
  • @LeoB., it was computed in whatever manner would give the biggest number for a specific computer. – Mark Jul 6 '17 at 21:49
  • 1
    Compiler optimizations and memory/cache policies in hardware that were done specifically for LINPACK inner loops code and vector sizes (and not other types of typical FP code) were rumored or reported to be done. – hotpaw2 Jul 7 '17 at 1:02
  • 1
    @LeoB, I'm too young to have had firsthand experience of this, but someone who would know is Roy Longbottom. He's pretty much "Mr Benchmark", and he's collated a lot of historical information here: Whetstone Benchmark History and Results – scruss Jul 7 '17 at 13:57
up vote 2 down vote accepted

According to Roy Longbottom (pers. comm.), the proper way to calculate the real-life number of MFLOPS is to run the Whetstone benchmark and to take the geometric mean of the three floating point results in Millions of Floating Point Operations Per Second.

My mistake was using an outdated version of the benchmark.

For example, according to my experiments with simulated CU/ALU pipes of the BESM-6, this would come out to about 30% less than the geometric mean of A, M, and D, and 40% less than just M.

For the curious, here's how the BESM-6 results look like:

 WHETSTONE BENCHMARK FOR  100.00 SECONDS DURATION

      8 PASSES USED (X 100)

 FORTRAN WHETSTONE BENCHMARK - SINGLE PRECISION
Month run         7/2017
Supplier/model    IPMCE, USSR                                
CPU chip type     BESM-6                               
Clock MHz         9                                   
Cache size        16 words                                 
Chipset/options   CU/ALU pipes emulated using interlocks and ave. timings from the manual                        
OS/DOS            DISPAK (user mode emulated)
Compiler          F O R E X ИПM AH CCCP 4.12 OT 25.06.85           
Options           default

 LOOP CONTENT                   RESULT                MFLOPS      MOPS   SECONDS

 N1 FLOATING POINT       -1.12398256285086973          0.524               0.293
 N2 FLOATING POINT       -1.12187081181764370          0.402               2.674
 N3 IF THEN ELSE          1.00000000000000000                    0.185     4.477
 N4 FIXED POINT          12.00000000000000000                    0.346     7.280
 N5 SIN,COS ETC.          0.49902906717352380                    0.036    18.361
 N6 FLOATING POINT        0.99999958804255584          0.121              35.720
 N7 ASSIGNMENTS           3.00000000000000000                    0.106    13.963
 N8 EXP,SQRT ETC.         0.75100162294984329                    0.018    16.087

 MWIPS                                                 0.809              98.856

MFLOPS per the benchmark come out to 0.295; the theoretical max numbers are A=0.820, M=0.500, and D = 0.180, based on the CPU manual; their geometric mean is 0.418.

The question seems to be skirting round the definition of FLOPS compared with the measurements of performance achieved by benchmarks. FLOPS is defined as a 'best case' measurement, as is evident from the way it is calculated. There is no allowance for making any use of the floating-point result (load/store), and no allowance for any control flow (branches and integer maths). These approximations may not have been too significant when the benchmark was first envisaged, and make some sense in the context of a separate floating-point unit which once loaded performs its work, and returns a result (that being the count for cycles required).

Where this starts to fall over is cores which are able to perform as many floating point operations as data transfers - but this is where benchmark results become relevant. Take a standardised workload, and use that as a measure of both the floating point workload, and the other operations which are necessary to feed the workload. Benchmarks are more real numbers, but subject to over-fitting of the micro-architecture (pick a gain of 5% on three benchmarks, or a loss of 2% on another for example).

Taking the numbers as presented to get a best case number is not unreasonable, the performance metric relates to the fastest possible floating point operation (which makes the result almost entirely meaningless, but still easy to quote). I believe that at the time, these numbers were rapidly identified as meaningless, and replaced by a small number of synthetic benchmarks - we're still evolving what we use to benchmark processors now.

  • I would accept either definition, as far as it is clear whether the result represents the "best case" or the "real life" performance. Please see an answer I've just posted, based on my communication with Ray Longbottom, the maintainer of the Whetstone benchmark. – Leo B. Jul 8 '17 at 18:54

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.