Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify how string conversion functions in signature REAL treat the sign of a NaN value #33

Open
pclayton opened this issue Jul 19, 2022 · 0 comments

Comments

@pclayton
Copy link

In the signature REAL, the conversion functions fromDecimal and toDecimal ensure that the sign field of the IEEEReal.decimal_approx value matches the sign bit of the real value even if the value is NaN.

The description of the functions scan and fromString does not specify how the sign bit is derived in the case of an argument giving the NaN value, so an implementation may arbitrarily choose a value for the sign bit. Is this underspecified behavior intended? If the intention was to preserve the sign bit through conversion of NaN, could we say:

  • fromString is equivalent to Option.composePartial (fromDecimal, IEEEReal.fromString)?
  • scan is equivalent to Option.composePartial (mapPartialFirst fromDecimal, IEEEReal.scan)
    where fun mapPartialFirst f (a, b) = case f a of SOME a' => SOME (a', b) | NONE => NONE?

The description of the function toString says that a NaN value is converted to the string "nan". This ignores the sign bit of the NaN value. Therefore toString is not equivalent to IEEEReal.toString o Real.toDecimal. Is this intended?

This issue is raised as suggested in smlnj/legacy#73 (comment).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant