Avoid unnecessary casts to double as this my mistakenly alter NaN bits.

R=ahaas@chromium.org, joransiu@ca.ibm.com, jyan@ca.ibm.com, michael_dawson@ca.ibm.com, dstence@us.ibm.com
BUG=

Review URL: https://codereview.chromium.org/1445603002

Cr-Commit-Position: refs/heads/master@{#31991}