Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GA-1525] NCL Entry for Precision Differences #67

Open
cyschneck opened this issue Jul 1, 2024 · 1 comment
Open

[GA-1525] NCL Entry for Precision Differences #67

cyschneck opened this issue Jul 1, 2024 · 1 comment
Labels
ncl Related to ncl specific content

Comments

@cyschneck
Copy link
Contributor

cyschneck commented Jul 1, 2024

NCL uses a different floating type which impacts precision. Would be useful to make a fully entry about the origin of the precision differences to be able to link to in future entries

For example:

asin: 
	python (math):	0.5235987755982989
	python (numpy):	0.5235987755982989
	ncl:		0.5235988

There are some diffs in the definitions depending upon the context.

In NCL the float type is 32 bits and double 64 bits, whereas in Python the float type is generally a 64-bit floating point number.

This is also slightly different from numpy which has its own types (e.g. np.float64, np.float32).

Generally the 64-bit types are the defaults in Python though, which is different from NCL and something that's come up in the past.

(Originally from #40)

@cyschneck cyschneck added the ncl Related to ncl specific content label Jul 1, 2024
@cyschneck cyschneck changed the title NCL Entry for Precession Differenes NCL Entry for Precision Differenes Jul 16, 2024
@anissa111 anissa111 changed the title NCL Entry for Precision Differenes [GA-1525] NCL Entry for Precision Differences Sep 16, 2024
@kafitzgerald
Copy link
Collaborator

We might also think a bit more about some good defaults and/or guidance for setting tolerances (i.e. in math.isclose() or numpy.isclose()) in our receipts testing.

This could be it's own chunk of work, but I think it aligns here thematically.

This came up most recently in Andy's climatology PR. Conversation here: #80 (comment)

Some relevant docs / reading:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ncl Related to ncl specific content
Projects
None yet
Development

No branches or pull requests

2 participants