The original Intel 8087 supported an 80-bit floating point format with 64 bits of mantissa and 15 exponent bits. This is more than the double-precision format of IEEE-754. This 8087 architecture is still with us in modern Intel and AMD CPUs, even though it is largely overtaken by the SSE2 extension that supports only regular single and double precision floating point numbers. The 80-bit extended precision float format is slower than the modern SSE2 instructions, but it is still available to C programs.
Other widely available CPUs, such as ARM and RISC-V do not have this extended precision format at all. RISC-V defines a full 128-bit quad-precision floating point extension, but the number of CPUs that actually implement this in hardware is zero. POWER-9 and IBM zSystem CPUs do implement 128-bit quad precision in hardware, plus decimal floating point. These are not in your regular desktop PC of phone.
For all practical purposes, 64-bit double-precision is enough. The 53 bits of precision are enough to express the length of a trip to Mars to within one millimetre. That said, modern atomic clocks reach a precision of 10-14, getting somewhat close to the 15-digit precision our double floating point numbers have. Extended precision may be useful for providing maximally precise implementations of logarithmic and trigonometric functions for programs using double precision. The round-off errors that occur when computing these functions in extended precision, disappear in double precision. Of course there is number theory study. It is super important to know the statistical distribution of digits of Pi. 128-bit floating point does not cut that either and we use arbitrary-precision libraries for it.Pi has been computed to a several trillion digits (1012 ). Therefore the use of 128-bit floating point is rather limited and
With the old Intel CPUs, single-precision and double-precision did not compute any faster than 80-bit extended precision. The 32-bit single and 64-bit double floating point formats were just storage formats, the 80×87 computed everything in extended precision anyway. The downside was, that programmers in high-level languages had no idea how and when the compiler stored intermediate results into memory (thereby rounding them) and when it kept them in internal registers, keeping the full precision.
The original C standard even specified that intermediate results had to be double precision, restricting the use of faster 32-bit single precision operations on CPUs on which these were indeed faster.
When doing large numerical operations, like linear equation solving and numerical integration, round-off errors add up. You may lose several digits of precision. Single-precision may not be enough for some applications. If you start with 7 digits and end with just 3 digits of precision. But in many cases, even 3 digits of precision of the end result, is just enough. Some microcontrollers. like the ARM Cortex-M33, have only single-precision floating point support, no double precision in hardware. If you can get the job done with just single precision, it makes the difference between a viable system or not. On modern CPUs with SIMD or vector hardware, throughput of single precision operations is at least twice that of double precision, just because you can do twice as many operations in parallel.
Enter the world of 3D rendering. Pixel values can have a high dynamic range, maybe 1 to 1 billion or more. 16-bit floating point numbers offer a dynamic range greater than 32-bit integers, but requiring half the bandwidth. The half-precision format has 5 bits of exponent and 11 bits of mantissa (implicit leading one). Modern CPUs offer half-precision floating point numbers, twice as many in parallel as single-precision numbers. GPUs (Graphics Processing Units) are typically built around half-precision floating point numbers.
Machine learning is another field that benefits from small floating point numbers. Half-precision floats are well suited for it. Coefficients of a neural network require a rather large dynamic range, but low precision. Once a model has been trained, its precision can be reduced even further to 8 bits or 4 bits.
The digital telephone network has been around since the 1970s. It uses 8-bit audio samples at a sampling rate of 8000 samples per second. Therefore it is 64kilobits/s. The samples are not linear PCM, but a format with a higher dynamic range with less precision for higher values and more precision for lower values. This is called A-law or µ-law curve. This is very similar to an 8-bit floating point format. The dynamic range of the raw samples is 13 bits (A-law) or 14 bits (µ-law), but they are compressed into 8 bits. Both A-law and µ-law curves are in use.
Come to think about it: a four-bit floating point number can have 16 distinct values. Typically they are -6.0, -4.0, -3.0, -2.0, -1.5, -1.0, -0.5, -0.0, +0.0,. 0.5, 1.,0, 1.5. 2.0. 3.0, 4.0 and 6.0. The ratio between the largest value (6.0) and the lowest non-zero value (0.5) is just 12. For 4-bit integers this would be 7. Four-bit floats skip the infinity and NaN values (but is is possible to replace -0.0 with a catch-all for infinity and NaN).. Four-bit floats are used in Large-Language models, not for training purposes but to store them once they are trained. They are often stored in groups, with an additional scaling factor per group.
8-bit and 4-bit floating point numbers are also sometimes used for education, to illustrate how IEEE-754 works. A four-bit floating point format is the minimum that has all the important elements. The 16 values would now be. -NaN, -Inf, -3.0, -2.0, -1.5. -1.0, -0.5, -0.0, +0.0, 0.5, 1.0, 1.5, 2.0. 3.0, +Inf and NaN. We can show the results of operations like addition, subtraction, multiplication and division in single-page tables, for each of the four rounding modes. The 4-bit float has one sign bit, two exponent bits and 1 mantissa bit, giving 2 bits of precision using an implicit leading 1.
Leave a Reply