View Single Post
 
Old 06-09-2010, 07:38 PM
Kimberly Kimberly is offline Windows 7 Office 2010 (Version 14.0)
Expert
 
Join Date: May 2010
Posts: 517
Kimberly is on a distinguished road
Default

15 digits of precision... without regard to the position of the decimal point.

Excel, like most computer programs, uses a number storage standard named IEEE Double Precision Floating Points. Numbers stored in this format are limited to 15 digits of precision, beyond which the numbers are rounded.

Institute of Electrical and Electronic Engineers (IEEE) is an international organization that determines standards for computer software and hardware. The 754 specification is a very widely adopted specification that describes how floating-point numbers can be stored in a binary computer.

There is, for example, no way to represent 0.1 in a finite number of digits in binary.

I would look at the many rounding functions, such as ROUND(), if I were you, instead of setting the 'precision as displayed'.
Reply With Quote