-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[java] Fix avro logical-types conversions for BQ storage #33422
base: master
Are you sure you want to change the base?
Conversation
Most of the avro logical-type to BQ are broken. Add support for both joda and java time to ensure compatibility with older avro versions
@@ -115,34 +120,90 @@ static String convertUUID(Object value) { | |||
} | |||
|
|||
static Long convertTimestamp(Object value, boolean micros) { | |||
if (value instanceof ReadableInstant) { | |||
return ((ReadableInstant) value).getMillis() * (micros ? 1000 : 1); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was wrong. BQ always expects epoch microseconds. Conversion should be applied on the raw type depending if it represents millis or micros
.setScale(type.getScale(), RoundingMode.DOWN) | ||
.round(new MathContext(type.getPrecision(), RoundingMode.DOWN)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this seems legit to round ? We might also fail if the BigDecimal
precision and scale do not match with expected logical type
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
assign set of reviewers |
Assigning reviewers. If you would like to opt out of this review, comment R: @kennknowles for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Most of the avro logical-type to BQ are broken.
Add support for both joda and java time to ensure compatibility with older avro versions