Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent metadata schema in association catalogs #314

Open
3 tasks done
camposandro opened this issue Jul 25, 2024 · 0 comments
Open
3 tasks done

Inconsistent metadata schema in association catalogs #314

camposandro opened this issue Jul 25, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@camposandro
Copy link
Contributor

Bug report

The generate_data.ipynb notebook generates an association catalog with the following schema:

pa.schema(
        [
            pa.field("Norder", pa.int64()),
            pa.field("Npix", pa.int64()),
            pa.field("join_Norder", pa.int64()),
            pa.field("join_Npix", pa.int64())
        ]
)

It seems like the column data types are all being inferred to int64, when *_Norder should be of type uint8 and *_Npix of type uint64. Specifying the schema when creating the record batches from arrays should solve this issue:
https://github.com/astronomy-commons/hipscat/blob/047600e667191af000337b446f7c24fd37a6b0eb/src/hipscat/catalog/association_catalog/partition_join_info.py#L93-L101

Before submitting
Please check the following:

  • I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
  • I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
  • If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
@camposandro camposandro added the bug Something isn't working label Jul 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: No status
Development

No branches or pull requests

1 participant