Excel Seems to be Rounding Twice Rather Than Once

joeu2004 · #4 08-22-2016, 12:15 PM

Quote:

Originally Posted by xor

When I do your calculation in Excel I get:
=41048/1312319*895.33 = 28.0050093308

Quote:

Originally Posted by melissapete24

I pulled out the calculator and did the math again, and it is still saying 28.004937. But, I pulled out my iPhone and used that calculator, and it came out the same as what you got, as did the calculator on the computer. [....] How the heck is a CALCULATOR wrong?

It is not "wrong" per se. Ironically, it is the calculator, not Excel that is "rounding twice".

More accurately, the calculator is simply rounding (or truncating) to a different number of decimal places (or significant digits).

[EDIT] And it might be a decimal calculator; that is, it might use decimal instead of binary representation internally.

We can simulate the calculator result by doing the following in Excel:

A1: 41048
A2: 1312319
A3: 895.33
A4: =ROUNDDOWN(A1/A2, 7)
A5: =A4*A3

A4 displays 0.0312789, and A5 displays 28.004937537.

Presumably, the calculator displays only 6 decimal places, truncating.

What calculator did you use? Perhaps the number of decimal places (or significant digits) is configurable.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Time Calc with rounding.	billreed7	Excel	5	07-03-2016 06:43 PM
Rounding to nearest 0.25	Guloluseus	Excel	5	10-27-2015 11:28 PM
Rounding values inconsistent	required_username	Excel	10	06-05-2014 05:35 AM
Standard Dev. Rounding - Excel	judicial85	Excel	1	08-28-2011 05:41 PM
Federal financial rounding?	markg2	Excel	6	05-26-2010 09:54 AM