Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support PrecisionTimestamp and PrecisionTimestampTZ literals #283

Merged
merged 2 commits into from
Jul 19, 2024

Conversation

Blizzara
Copy link
Contributor

No description provided.

@Blizzara
Copy link
Contributor Author

cc @vbarua

@vbarua
Copy link
Member

vbarua commented Jul 19, 2024

Will look at this today.

Copy link
Member

@vbarua vbarua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good. I have two questions around the ExpressionCreator constructors.

var epochMicro =
TimeUnit.SECONDS.toMicros(value.toEpochSecond(ZoneOffset.UTC))
+ TimeUnit.NANOSECONDS.toMicros(value.toLocalTime().getNano());
return precisionTimestamp(nullable, epochMicro, 6);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs for LocalDateTime indicate

Time is represented to nanosecond precision.

Should we be using nanoseconds (9) here instead of microseconds (6)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no real preference here - just kept it like this so that it aligns as closely as possible to the existing timestamp creators.

Given the LocalDate (and also Instant) contain more data than just a int64, (they have seconds and then the subseconds separately), I guess there's some dates that cannot be represented as only nanonseconds but can in microseconds. Quick search suggests that's about +- 292 years. So while we can switch this to nanoseconds, it could overflow for some inputs (if those inputs are like year 3000).

Frankly, I think I'd rather keep this as is, to make switching from the earlier timestamp behavior as easy as possible. If people want more control, they can always use the other creator func that lets them decide the precision. Alternatively, we could even consider not adding this second constructor. Thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we could even consider not adding this second constructor.

Let's leave the new constructors out for now. It sounds like we may need to consider their behaviours more carefully now that we're not eliding to microseconds by default, but we don't need to wait to figure that out before merging the rest of this in.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! f255e6c

var epochMicro =
TimeUnit.SECONDS.toMicros(value.getEpochSecond())
+ TimeUnit.NANOSECONDS.toMicros(value.getNano());
return precisionTimestampTZ(nullable, epochMicro, 6);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs for Instant indicate that

the class stores a long representing epoch-seconds and an int representing nanosecond-of-second

Should we be using nanoseconds (9) here instead of microseconds (6)?

Relatedly, are all possible Instant values representable as PrecistionTimestampTz literals?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the other answer. Not all instants are representable as PTTz, I believe, since instant stores in total 96 bits of data, while PTTz is only 64 bits.

@vbarua vbarua merged commit 94996f9 into substrait-io:main Jul 19, 2024
12 checks passed
@Blizzara Blizzara deleted the avo/precision-timestamp branch July 19, 2024 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants