You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to improve specific types detection it's important to use as many documents as possible for schema inference and in the case of arrays, unlike streams, we have a luxury to access entire dataset.
Currently for both streams and arrays inference depth is equal to batch size.
So as arrays are loaded into memory we can process entire array twice, first time for schema inference and second time for importing data.
The text was updated successfully, but these errors were encountered:
In order to improve specific types detection it's important to use as many documents as possible for schema inference and in the case of arrays, unlike streams, we have a luxury to access entire dataset.
Currently for both streams and arrays inference depth is equal to batch size.
So as arrays are loaded into memory we can process entire array twice, first time for schema inference and second time for importing data.
The text was updated successfully, but these errors were encountered: