Excel can't count??

E

eeyrjmr

[H]ardness Supreme
Joined
Apr 23, 2002
Messages
4,363
could someone doublecheck this? done with Office-XP and Office-2003

in a cell: =29513736*92842033

The answer should be: 2740115251665288
Done with
1) windows calc
2) python -c 'print 29513736*92842033'
3) matlab

BUT for some reason excel is giving: 2740115251665290
 
Odd, I just checked on Office 2008 for mac and im getting exactly what you describe, i also checked =PRODUCT() and it also returns the improper value. Calculator for mac does return the correct number.
 
but that is an int/long NOT a float

IF I did jrb@FluidMotion ~ $ python -c 'print 29513736.0*92842033.0'
2.74011525167e+15

as opose to
jrb@FluidMotion ~ $ python -c 'print 29513736*92842033'
2740115251665288

ie forcing fpu maths in python, it to gets the same thing, since python adheres to the same IEEE (it has a Decimal module for precise floats)

IEEE-754 precision has nothing todo with integer mathematics
 
I gave you the answer why you see 15 signigicant digits. Try reading the link. Horse, water, your choice.

There are add-ons for Excel that extend the precision, or you can write your own in VBA.
 
You must log in or register to reply here.
Back
Top