Skip to content
This repository has been archived by the owner on Apr 6, 2019. It is now read-only.

fixed_point compare uses too great a range #257

Open
johnmcfarlane opened this issue Jul 1, 2017 · 0 comments
Open

fixed_point compare uses too great a range #257

johnmcfarlane opened this issue Jul 1, 2017 · 0 comments

Comments

@johnmcfarlane
Copy link
Owner

In

auto a = fixed_point<int8, 32>(0x100000000LL);
auto b = fixed_point<int8, 0>(1);
ASSERT_GT(a, b);

expression overflows. But only 16 (maybe 15) bits of range are required to scale operands such that comparison can be performed. See issue 123 for a discussion.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant