Microsoft Office Forums

Go Back   Microsoft Office Forums > >

 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #9  
Old 06-09-2010, 07:38 PM
Kimberly Kimberly is offline Formula problem? Windows 7 Formula problem? Office 2010 (Version 14.0)
Expert
 
Join Date: May 2010
Posts: 517
Kimberly is on a distinguished road
Default

15 digits of precision... without regard to the position of the decimal point.

Excel, like most computer programs, uses a number storage standard named IEEE Double Precision Floating Points. Numbers stored in this format are limited to 15 digits of precision, beyond which the numbers are rounded.

Institute of Electrical and Electronic Engineers (IEEE) is an international organization that determines standards for computer software and hardware. The 754 specification is a very widely adopted specification that describes how floating-point numbers can be stored in a binary computer.

There is, for example, no way to represent 0.1 in a finite number of digits in binary.

I would look at the many rounding functions, such as ROUND(), if I were you, instead of setting the 'precision as displayed'.
Reply With Quote
 



Similar Threads
Thread Thread Starter Forum Replies Last Post
excel help Formula sellspeed Excel 15 03-10-2010 10:57 AM
Formula problem? Help with Formula Corca Excel 6 02-22-2010 09:40 PM
Formula problem? If formula sixhobbits Excel 1 10-02-2009 08:02 AM
Help for formula dehann Excel 5 05-01-2009 10:44 AM
Formula problem? Need help with a formula tinkertron Excel 11 04-16-2009 11:43 PM

Other Forums: Access Forums

All times are GMT -7. The time now is 04:51 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2025, vBulletin Solutions Inc.
Search Engine Optimisation provided by DragonByte SEO (Lite) - vBulletin Mods & Addons Copyright © 2025 DragonByte Technologies Ltd.
MSOfficeForums.com is not affiliated with Microsoft