BigQuery driver seems does not cast all numerical values equally to strings #1835
Labels
bug
Something isn't working
driver:bigquery
Issues related to the BigQuery driver
Roadmap: 2021 Q1
Cube.js team roadmap for Q1 of 2021.
EDIT: I realize that I am two minor versions behind, so I will update to 0.25 to check if the issue is solved
EDIT 2: I confirm the bug is still present on version 0.25.21.
Describe the bug
Six days ago, we upgraded our version of the
@cubejs-backend/bigquery-driver
from "0.23.8" to "0.23.14" and yesterday we stumbled on a type cast error for the results of the query which caused our following code to fail due to a type assumption.Following a discussion on slack with @paveltiunov, turns out that Cube is transitioning towards returning all types as string for simplicity and for supporting large numbers.
The problem here is twofolds:
sum
typed measures resulted in different types - one a string and one a float.To Reproduce
Steps to reproduce the behavior:
Given a BigQuery view with two
sum
typed measures:the result returned is like the following:
where "schedule_track_facts.charges_amount" should be a float or "schedule_track_objective_calendar.charges_amount_objective" a string, but not both different types.
Schema:
Expected behavior
All FLOAT64, INT64, NUMERIC types should be returned as the same type in the resulting records, be it string or number.
It is very possible that the problem is that amount_cad is NUMERIC NULLABLE whereas
may be implicitly returned as FLOAT64 by the BQ engine due to
_open_time_slot_hours_value
being implicitly FLOAT NULLABLE.Learning on our side
Moving forward we will cast explicitly all of our types coming out of Cube.js (or any other service) to the expected type so that we can handle them properly in pandas, and catch unexpected type conversions earlier in the code.
The text was updated successfully, but these errors were encountered: