36

It's clear where many of the design decisions of IEEE 754 floating point come from. For example, the binary format maximizes efficiency on binary hardware. And 32 and 64 bits for single and double precision make sense on machines with power of two word sizes.

But the number of bits for sign+exponent is a free parameter, and the actual values chosen, being 9 and 12 respectively, seem terribly random. To the extent I would've any prior expectation, I would've expected them to be powers of two, like 8 or 16, to make it easier to implement them in software, since the standard was written at a time when most computers did not have floating-point hardware.

Where did those values come from?

  • 9
    The History section of Wikipedia's entry on IEEE 754-1985 suggests the standard evolved out of (or was at least influenced by) Intel's designs for a floating-point co-processor. Pursuing the rationale behind that design might throw some light on the numbers chosen. – TripeHound Jan 27 at 9:40
  • 2
    @TripeHound Yes, that matches my understanding, and then I'm curious about where Intel got the numbers. I mean, if the answer is that an Intel engineer held his finger up to the wind and decided those numbers felt just about right, fair enough, but I would be curious about exactly who and when. – rwallace Jan 27 at 10:26
  • 2
    "Most computers did not have FP hardware" -- not true in my experience, but then microprocessors were not ubiquitous when I started programming, As the answer from @phuclv suggests, IEEE 754 was guided by earlier implementations. – another-dave Jan 27 at 13:05
  • 5
    Most computers at the time - and all of the ones where numeric processing were predominant - did have floating point hardware. They were the mainframes and minicomputers, not the microprocessors. Any kind of scientific or engineering applications ran on them. Choices were made based on the expectations of the numerical software that was being run - there was a lot of it already running against a variety of floating point formats - and on numerical analysis of those floating point formats (e.g., the IBM format's base-16 exponent was deemed a loser from that POV). – davidbak Jan 27 at 19:43
  • 1
    Rumor has it that Seymour Cray designed the Control Data Cyber mainframes with 60 bit words instead of 64 bit because he found it easier and/or quicker to do floating point arithmetic at that size. – Mark Ransom Feb 5 at 16:25
56

From an interview with Dr. William Kahan, the IEEE-754 formats were based on the VAX F and G formats, which have 8 and 11-bit exponents respectively. In fact Dr. Kahan also said that previously VAX has a double precision D format which has the same 8-bit exponent as the single precision F format which proved too limited in practical use, therefore DEC had to introduced 3 more bits to create the new G format to avoid the underflow/overflow issues

But why did DEC use 8 bits for the exponent in the F and D formats? The reason is to be able to represent all important physical constants, including the Plank constant (6.626070040 × 10-34) and the Avogradro constant (6.022140857 × 1023), as stated in PDP-11/40 Technical Memorandum #16

Another rationale for the 64-bit format as described by David Stephenson ("A Proposed Standard for Binary Floating-Point Arithmetic", IEEE Computer, Vol. 14, No. 3, March 1981, pp. 51-62) is that

For the 64-bit format, the main consideration was range; as a minimum, the desire was that the product of any two 32-bit numbers should not overflow the 64-bit format. The final choice of exponent range provides that a product of eight 32-bit terms cannot overflow the 64-bit format — a possible boon to users of optimizing compilers which reorder the sequence of arithmetic operations from that specified by the careful programmer.

In fact nowadays the rule for IEEE-754 interchange format the size for the exponent is round(4 log2(k)) − 13 bits so every time we double the width of the type, the exponent will be have ~4 more bits which allows for 16 multiplications of the narrower type without overflow

The encoding scheme for these binary interchange formats is the same as that of IEEE 754-1985: a sign bit, followed by w exponent bits that describe the exponent offset by a bias, and p − 1 bits that describe the significand. The width of the exponent field for a k-bit format is computed as w = round(4 log2(k)) − 13. The existing 64- and 128-bit formats follow this rule, but the 16- and 32-bit formats have more exponent bits (5 and 8) than this formula would provide (3 and 7 respectively).

https://en.wikipedia.org/wiki/IEEE_754#Interchange_formats

Those are the things I summarized from the below links which you should read to get more details

|improve this answer|||||
  • 5
    One should remember that the main ieee-754 standard pusher and creator was intel itself. So their choices, while being seemingly well-argumented nowadays, may have included different reasoning those days. – lvd Jan 27 at 13:06
  • 5
    Intel wanted to push its x87, so it hired Dr. Kahan to draft a standard as well as lead the implementation, but it doesn't define the standard (which were the result of a series of meetings between CPU manufacturers and numerical analyzing scientists) – phuclv Jan 27 at 14:13
  • 4
    The 11/40 memos are a little frustrating to read: on Aug 21 they were in favour of a S/360-style hex exponent, and by Oct 1 they'd switched to a binary exponent, but the intervening arguments and rationale don't appear. – another-dave Jan 28 at 0:39
  • 3
    @phuclv: The 80-bit format was more efficient than 64-bit format not only on the 8x87, but also for most processors without floating-point unit. In the 68000-based Standard Apple Numerical Environment, when not using an FPU, the slowest data type to work with was 64-bit double-precision. The 80-bit extended type was not only more precise, but also faster. – supercat Jan 28 at 17:55
14

Well, the first thing to remember about these binary formats you're talking about (there were also decimal formats) is that they are the interchange formats; it's not required that hardware, or even software, use these for internal calculations. It's perfectly reasonable for an implementation to use an entire separate byte for the sign and one or more separate bytes for the exponent, if that's felt to be a good memory vs. speed tradeoff by the designer. (And in fact that's exactly what many 6502 floating point routines did!)

The binary interchange formats were chosen first to fit into common modern binary machine words: 16, 32, 64, 128 and 256 bits. The next decision to be made is the balance between exponent bits and significant bits: too many of the former and you have a format with perhaps more range you need and not enough precision; too many of the latter and you have the opposite problem.

An exponent of eight bits as used by binary32 (single-precision) interchange format gives you a binary exponent range of only -126 to +127, which limits the absolute sizes of your values to somewhere around 10^38 decimal. That's not a bad range, but it's certainly not enough for many applications. Reducing it to 7 bits (so that the sign and all exponent bits could be in a single byte) would make the problem even worse, and probably wouldn't help much with calculations in software since you still have to split out the sign, and anybody concerned about speed is going to use an internal format with a separate byte for that anyway.

Once we move to binary64 (double-precision), the problem goes in the opposite direction: increasing the 11 bit exponent to 16 bits exponent would give an enormous absolute range of something like 10^5000 decimal and, because it's "stealing" 5 bits that could otherwise be used for more precision, would reduce precision from almost 16 digits to just over 14. The current scheme was felt to be a better tradeoff.

Following is a table I made a while back when I was thinking about this topic myself. p is precision (in bits) of significand, including an implicit leading 1. prec is number of digits of decimal precision (c*log₁₀(2)), demax is the maximum exponent in decimal (2^(e-1)-1 * log₁₀(2)).

byt bits p   e   prec    demax      notes
2   16  11   5   3.31     4.51      IEEE 754 half precision (not basic)
3   24  16   8   4.81    38.23
3   24  17   7   5.11    18.96
3   24  18   6   5.42     9.33
3   24  19   5   5.72     4.51
4   32  24   8   7.22    38.23      IEEE 754 single precision
5   40  32   8   9.63    38.23
5   40  31   9   9.33    76.76
5   40  30  10   9.03   153.8
6   48  40   8  12.04    38.23
6   48  39   9  11.74    76.76
6   48  38  10  11.44   153.8
7   56  48   8  14.45    38.23
8   64  56   8  16.86    38.23
8   64  53  11  15.95   308.0       IEEE 754 double precision

MS 6502 BASIC shipped with 32-bit 6 digit FP (8 KB) in 1976 (8 KB), later expanded to 40-bit 9 digit (9 KB) in 1977.

|improve this answer|||||
  • 3
    As the other answer notes, one of the key reasons for an 8-bit exponent is that it can represent both Avogadro's constant (the largest constant commonly used in physics) and the reduced Planck constant (the smallest). – Mark Jan 28 at 2:41
  • 1
    @Mark Yes, I hadn't thought in detail about what useful numbers the smaller exponents could represent, but it's great reasoning. In fact, phuclv's answer is, overall, much more comprehensive than mine (I upvoted it myself). This one was basically just a quick dump of some thoughts and notes of mine from last year, and I hope it does have a little to contribute. – cjs Jan 28 at 3:19
4

I can see logic in selecting 23 mantissa bits for 32-bit float: together with the implied "one" it fits exactly in 24 bits, which is convenient for emulating floats on a purely integer processor.

And another pressure is the precision -- they had to trade precision for exponent size, and the precision (which was not particularly high for float-32) won a bit or two.

64-bit float decisions might be more random since the precision it gives and the maximum/minimum exponent it allows seemed to be less of importance at the time (both were perceived to be "more than enough", and the exponent still is).

Handling both exponent sizes within a 16-bit cpu (which was i8086 for intel) was obviously easy.

|improve this answer|||||
  • 5
    The 64-bit double is lousy to work with for processors without floating-point hardware. A 48+15+1 with an explicit leading 1 would have been much faster, since it would require a lot less bit shifting and masking. Multiplying two values on a CPU with 16x16->32 multiply would take 9 multiples with a 48-bit significand, but takes 16 with a 53-bit one. – supercat Jan 28 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.