Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better error messages when using lazy mode #19377

Open
EtienneT opened this issue Oct 22, 2024 · 0 comments
Open

Better error messages when using lazy mode #19377

EtienneT opened this issue Oct 22, 2024 · 0 comments
Labels
enhancement New feature or an improvement of an existing feature

Comments

@EtienneT
Copy link

Description

When doing a very large query using lazy mode, if there's any error in the query, you will get it when you call collect which is normal, but we are receiving errors like this in production:

polars.exceptions.ComputeError: cannot compare string with numeric type (i32)

Which column name, compared to which other column produced this error? Impossible to know. My query expression tree is constructed from several methods adding filters dynamically with a lot of columns, it is very difficult to investigate which column specifically has the wrong type in production when this doesn't happen locally.

But given that we are in a lazy query, we have all the query expression tree, so we could potentially at least return in the error message which column name are having a problem.

The only solution right now is to transform this big query in several chunk in eager mode and see where it will crash in production which is not ideal.

@EtienneT EtienneT added the enhancement New feature or an improvement of an existing feature label Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature
Projects
None yet
Development

No branches or pull requests

1 participant