[Skywise]

(12-30-2013, 10:34 PM)Skywise Wrote:

(12-30-2013, 04:26 PM)Roger Hunter Wrote: TB just told me that int((6.3-6.2)*10) = 0.

I understand why but I thought TB was better than that.

Roger

Thanks, Roger. You just sent MY cpu into a tail spin...

Well, we all know that 1 + 1 = 10, right?

Brian

To expand, for those who may be reading, it all has to do with the way non-integer numbers are stored in computer memory. It's a sort of rounding error. Computers only work in binary numbers - 1's or 0's.

It's similar to how you write out 1 divided by 3. The answer is .3333, approximately. Approximately because, in reality those 3's go on forever, a repeating decimal. In base 10 numbers, the numbers we use everyday, 1/3rd cannot be expressed precisely in decimal.

In binary (base 2) certain numbers also cannot be expressed precisely. For example, the numbers 6.3 and 6.2 that roger gives in his example cannot be stored precisely, and when the difference is taken the result is not precisely .1. In fact, it turns out to be just a hair under .1. Therefore, when it is multiplied by 10, it turns out to be just a hair under 1, instead of one precisely.

The INT function in programming means to take the integer of the number, chopping off the decimals. Since the result is not precisely 1, but something just less than 1, all the decimals get chopped off and we are left with zero.

Now, something I discovered in C++ is that I can get the right answer if I program the equation differently.

int((6.3-6.2)*10) gives the answer 0, whereas
int((6.3f-6.2f)*10) gives the expected answer of 1.

That extra 'f' in there tells the compiler to treat the number as an 80bit extended floating point literal instead of the usual standard 64bit double precision floating point. Basically, I'm just using more decimals of precision even though it's still not exactly precise, so that when the answer is rounded off to 'only' 64bit, it rounds off to the exact value.

When you learn to program computers, you have to learn numbers and counting all over again.

Brian