[Bug Fix] Allow Partition data to be nullable in ManifestEntry #509
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fokko
reviewed
Mar 10, 2024
pyiceberg/manifest.py
Outdated
| @@ -308,6 +308,7 @@ def data_file_with_partition(partition_type: StructType, format_version: Literal | |||
| field_id=field.field_id, | |||
| name=field.name, | |||
| field_type=partition_field_to_data_file_partition_field(field.field_type), | |||
| required=False, | |||
There was a problem hiding this comment.
What do you think of carrying this over from the field?
Suggested change
| required=False, | |
| required=field.required, |
I would prefer this because Avro schema's are translated to Iceberg, and then used in reading the files. In Avro reading an optional field is different than reading a required field (in the case of an optional field, it will first read a boolean checking if the value is there).
I checked locally in tests, and there the fields are not required.
There was a problem hiding this comment.
Sounds good @Fokko . I've made the corresponding change to partition_type to carry this over from the partition field as suggested 👍
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Similar to how the partition fields are already initialized with
required=Falseinpartition_type, we should setrequired=Falsein the manifest_entry schema within the ManifestFile.This PR introduces this fix first from @jqin61 's WIP PR, so other PRs that add partition values to manifest files can be unblocked.
Without this change, we cannot serialize None partition values into the ManifestEntry in the avro file.
Below WIP PRs require this change
Partitioned Write: #353
Add Files: #506