Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot parse missing long: current-snapshot-id #1164

Closed
kevinjqliu opened this issue Sep 11, 2024 · 1 comment
Closed

Cannot parse missing long: current-snapshot-id #1164

kevinjqliu opened this issue Sep 11, 2024 · 1 comment

Comments

@kevinjqliu
Copy link
Contributor

Apache Iceberg version

None

Please describe the bug 🐞

From slack,

Hi! We’ve noticed that after creating an empty table in the Glue catalog with PyIceberg, initially there are no snapshots. Then if we run a glue job with a merge statement to upsert some CDC records in this table, the job fails with an error Error Category: UNCLASSIFIED_ERROR; IllegalArgumentException: Cannot parse missing long: current-snapshot-id. A workaround that resolved this was to insert and delete a dummy row after creating the table, but that doesn’t seem right. Is this a Glue / Spark bug or did we miss something?

Error Category: UNCLASSIFIED_ERROR; IllegalArgumentException: Cannot parse missing long: current-snapshot-id
@kevinjqliu
Copy link
Contributor Author

kevinjqliu commented Sep 11, 2024

Also from slack,

Sung:
This is actually an issue with some of the older Java applications making the incorrect assumption that the current_snapshot_id is a required field. As @Kevin Liu noted, this is an optional attribute in the Spec.
In PyIceberg, we do have an env variable you can set on your application that will force your application to write metadata files with '-1' as the current_snapshot_id to circumvent this issue. It's a hack, but it works in creating files that are compatible with these older Java applications. Please let me know if that's helpful!
#473

The official documentation on this property is listed here: https://py.iceberg.apache.org/configuration/#backward-compatibility

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant