Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(avro-schema): logical type support #19607

Merged

Conversation

joscha
Copy link
Contributor

@joscha joscha commented Sep 17, 2024

This implements logical type support for:

for the OpenAPI built-in string formats:

  • date-time
  • date

The change is backwards compatible by allowing to opt-in to the behaviour via a new property useLogicalTypes.
Avro's support for timestamp fields includes the definition which timescale they are represented in (milliseconds, microseconds or nanoseconds). This can be controlled via a new property logicalTypeTimeQuantifier.

PR checklist

  • Read the contribution guidelines.
  • Pull Request title clearly describes the work in the pull request and Pull Request description provides details about how to validate the work. Missing information here may result in delayed response from the community.
  • Run the following to build the project and update samples:
    ./mvnw clean package 
    ./bin/generate-samples.sh ./bin/configs/*.yaml
    ./bin/utils/export_docs_generators.sh
    
    (For Windows users, please run the script in Git BASH)
    Commit all changed files.
    This is important, as CI jobs will verify all generator outputs of your HEAD commit as it would merge with master.
    These must match the expectations made by your contribution.
    You may regenerate an individual generator by passing the relevant config(s) as an argument to the script, for example ./bin/generate-samples.sh bin/configs/java*.
    IMPORTANT: Do NOT purge/delete any folders/files (e.g. tests) when regenerating the samples as manually written tests may be removed.
  • File the PR against the correct branch: master (upcoming 7.6.0 minor release - breaking changes with fallbacks), 8.0.x (breaking changes without fallbacks)
  • If your PR is targeting a particular programming language, @mention the technical committee members, so they are more likely to review the pull request. There is no committee for this one, as it's a schema transformer, thus: @sgadouar or @wing328 please?

bin/configs/avro-schema.yaml Outdated Show resolved Hide resolved
@@ -0,0 +1 @@
{{#useLogicalTypes}}{{#isDate}}{ "type": "int", "logicalType": "date" }{{/isDate}}{{#isDateTime}}{ "type": "long", "logicalType": "timestamp-{{{logicalTypeTimeQuantifier}}}" }{{/isDateTime}}{{^isDate}}{{^isDateTime}}"{{{dataType}}}"{{/isDateTime}}{{/isDate}}{{/useLogicalTypes}}{{^useLogicalTypes}}"{{{dataType}}}"{{/useLogicalTypes}}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would be great to format this better; if there is a way to use whitespace in the template and then make it collapse in the result, please let me know.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, that's changing the template operator, right? I basically wish I could change the string in this template to:

Suggested change
{{#useLogicalTypes}}{{#isDate}}{ "type": "int", "logicalType": "date" }{{/isDate}}{{#isDateTime}}{ "type": "long", "logicalType": "timestamp-{{{logicalTypeTimeQuantifier}}}" }{{/isDateTime}}{{^isDate}}{{^isDateTime}}"{{{dataType}}}"{{/isDateTime}}{{/isDate}}{{/useLogicalTypes}}{{^useLogicalTypes}}"{{{dataType}}}"{{/useLogicalTypes}}
{{#useLogicalTypes}}
{{#isDate}}
{
"type": "int",
"logicalType": "date"
}
{{/isDate}}
{{#isDateTime}}
{
"type": "long",
"logicalType": "timestamp-{{{logicalTypeTimeQuantifier}}}"
}
{{/isDateTime}}
{{^isDate}}
{{^isDateTime}}
"{{{dataType}}}"
{{/isDateTime}}
{{/isDate}}
{{/useLogicalTypes}}
{{^useLogicalTypes}}
"{{{dataType}}}"
{{/useLogicalTypes}}

and still have well-spaced output in the resulting schema. Something like a {{#collapseWhitespace}}...{{/collapseWhitespace}} operator that takes anything in its body and reduces all \w+ to

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope. i don't think mustache comes with anything to do that

looks like avsc file are just JSON file so users can use a linter to auto format the auto-generated output as they do with other generators

@wing328 wing328 added Enhancement: Feature Schema: Avro Generation of Avro schemas labels Sep 18, 2024
@wing328 wing328 added this to the 7.9.0 milestone Sep 18, 2024
@wing328 wing328 merged commit 171804e into OpenAPITools:master Sep 19, 2024
15 checks passed
@joscha joscha deleted the joscha/avro-schema-logical-type-support branch September 19, 2024 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement: Feature Schema: Avro Generation of Avro schemas
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants