You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Am trying to import a CSV file to Databricks (9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12) and even with a very simple 2-column example and tiny integer input, it fails with Failed to merge incompatible data types LongType and IntegerType
Here are my test commands:
dbsqlcli -e "create or replace table `datatype_test`(`bigint_col` bigint, `int_col` integer)"
dbsqlcli -e "COPY INTO datatype_test FROM '/tmp/datatype_test.csv' FILEFORMAT = CSV FORMAT_OPTIONS ('header'='true', 'inferSchema'='true') COPY_OPTIONS ('mergeSchema'='true')"
where /tmp/datatype_test.csv contains:
bigint_col,int_col
,123
123,
123,123
Any suggestions?
The error response header is:
org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.sql.AnalysisException: Failed to merge fields 'bigint_col' and 'bigint_col'. Failed to merge incompatible data types LongType and IntegerType
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:1019)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:759)
...
The text was updated successfully, but these errors were encountered:
Hi,
Am trying to import a CSV file to Databricks (9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12) and even with a very simple 2-column example and tiny integer input, it fails with
Failed to merge incompatible data types LongType and IntegerType
Here are my test commands:
where /tmp/datatype_test.csv contains:
Any suggestions?
The error response header is:
The text was updated successfully, but these errors were encountered: