-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
new Decimal(2).pow(3) is 7.999999999999999 #173
Comments
I'm not sure if this one is fixable. Floating point numbers are naturally going to be somewhat imprecise - it's like how 0.1 + 0.2 = 0.3000000000000001. |
wouldnt a valid fix be to just round the values if both the exponent and base are integers (and final val < 9e15). Not the cleanest solution, sure, but works for most necessary cases |
The main problem there is that then pow might not quite be an increasing function anymore (random example I found: Decimal.pow(403.0000000000001, 3) is 65450826.99999999, which is less than the rounded 403^3 = 65450827), which could cause problems with things like logarithms. I'd rather have slight imprecision than give up the property that the function is increasing. |
that only ever occurs when log(n) = log(floor(n)) and n is a integer very slightly larger than floor(n), so that n != floor(n) |
That doesn't make sense. floor(n) is always equal to n when n is an integer. |
correct. Your point was that my solution would break non-integers to which i pointed out only affected a very small, unsubstantial amount of numbers. Additionally, my fix wouldnt even do that since you could test for being integers and less than 9e15 after the log not before |
Math.pow(2,3)
is 8.The text was updated successfully, but these errors were encountered: