Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hive connector can write numeric NaN and ±infinity with JSON format, but can't read the value #20395

Open
ebyhr opened this issue Jan 17, 2024 · 3 comments · May be fixed by #24558
Open

Hive connector can write numeric NaN and ±infinity with JSON format, but can't read the value #20395

ebyhr opened this issue Jan 17, 2024 · 3 comments · May be fixed by #24558
Assignees
@sug-ghosh
Labels
hive Hive connector

Comments

@ebyhr
Copy link
Member

ebyhr commented Jan 17, 2024 

create table test_nan with (format = 'json') as select nan() a;
table test_nan;

Failed to read file at /var/folders/9s/_zwn4r_n2_9bp0krllp1pl3c0000gp/T/TrinoTest16541592175967501555/hive_data/tpch/test_nan/20240117_001008_00024_7aya8_e9ee5153-2b74-4446-a143-49470ee8dd3b.gz
io.trino.spi.TrinoException: Failed to read file at /var/folders/9s/_zwn4r_n2_9bp0krllp1pl3c0000gp/T/TrinoTest16541592175967501555/hive_data/tpch/test_nan/20240117_001008_00024_7aya8_e9ee5153-2b74-4446-a143-49470ee8dd3b.gz
	at io.trino.plugin.hive.line.LinePageSource.getNextPage(LinePageSource.java:75)
	at io.trino.plugin.hive.HivePageSource.getNextPage(HivePageSource.java:166)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:266)
	at io.trino.operator.Driver.processInternal(Driver.java:398)
	at io.trino.operator.Driver.lambda$process$8(Driver.java:301)
	at io.trino.operator.Driver.tryWithLock(Driver.java:704)
	at io.trino.operator.Driver.process(Driver.java:293)
	at io.trino.operator.Driver.processForDuration(Driver.java:264)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:887)
	at io.trino.execution.executor.timesharing.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:187)
	at io.trino.execution.executor.timesharing.TimeSharingTaskExecutor$TaskRunner.run(TimeSharingTaskExecutor.java:565)
	at io.trino.$gen.Trino_testversion____20240117_000919_1.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.fasterxml.jackson.core.JsonParseException: Current token (VALUE_STRING) not numeric, can not use numeric value accessors
 at [Source: (byte[])"{"a":"NaN"}"; line: 1, column: 7]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2477)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:755)
	at com.fasterxml.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:936)
	at com.fasterxml.jackson.core.base.ParserBase.getDoubleValue(ParserBase.java:830)
	at io.trino.hive.formats.line.json.JsonDeserializer$DoubleDecoder.decodeValue(JsonDeserializer.java:403)
	at io.trino.hive.formats.line.json.JsonDeserializer$Decoder.decode(JsonDeserializer.java:243)
	at io.trino.hive.formats.line.json.JsonDeserializer$RowDecoder.decodeValue(JsonDeserializer.java:718)
	at io.trino.hive.formats.line.json.JsonDeserializer$RowDecoder.decode(JsonDeserializer.java:688)
	at io.trino.hive.formats.line.json.JsonDeserializer.deserialize(JsonDeserializer.java:157)
	at io.trino.plugin.hive.line.LinePageSource.getNextPage(LinePageSource.java:62)

create table test_inf with (format = 'json') as select infinity() a;
table test_inf;

Failed to read file at /var/folders/9s/_zwn4r_n2_9bp0krllp1pl3c0000gp/T/TrinoTest16541592175967501555/hive_data/tpch/test_inf/20240117_001337_00026_7aya8_7924f951-0c9a-4914-8512-ef47571a42f0.gz
io.trino.spi.TrinoException: Failed to read file at /var/folders/9s/_zwn4r_n2_9bp0krllp1pl3c0000gp/T/TrinoTest16541592175967501555/hive_data/tpch/test_inf/20240117_001337_00026_7aya8_7924f951-0c9a-4914-8512-ef47571a42f0.gz
	at io.trino.plugin.hive.line.LinePageSource.getNextPage(LinePageSource.java:75)
	at io.trino.plugin.hive.HivePageSource.getNextPage(HivePageSource.java:166)
	at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:266)
	at io.trino.operator.Driver.processInternal(Driver.java:398)
	at io.trino.operator.Driver.lambda$process$8(Driver.java:301)
	at io.trino.operator.Driver.tryWithLock(Driver.java:704)
	at io.trino.operator.Driver.process(Driver.java:293)
	at io.trino.operator.Driver.processForDuration(Driver.java:264)
	at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:887)
	at io.trino.execution.executor.timesharing.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:187)
	at io.trino.execution.executor.timesharing.TimeSharingTaskExecutor$TaskRunner.run(TimeSharingTaskExecutor.java:565)
	at io.trino.$gen.Trino_testversion____20240117_000919_1.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.fasterxml.jackson.core.JsonParseException: Current token (VALUE_STRING) not numeric, can not use numeric value accessors
 at [Source: (byte[])"{"a":"Infinity"}"; line: 1, column: 7]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2477)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:755)
	at com.fasterxml.jackson.core.base.ParserBase._parseNumericValue(ParserBase.java:936)
	at com.fasterxml.jackson.core.base.ParserBase.getDoubleValue(ParserBase.java:830)
	at io.trino.hive.formats.line.json.JsonDeserializer$DoubleDecoder.decodeValue(JsonDeserializer.java:403)
	at io.trino.hive.formats.line.json.JsonDeserializer$Decoder.decode(JsonDeserializer.java:243)
	at io.trino.hive.formats.line.json.JsonDeserializer$RowDecoder.decodeValue(JsonDeserializer.java:718)
	at io.trino.hive.formats.line.json.JsonDeserializer$RowDecoder.decode(JsonDeserializer.java:688)
	at io.trino.hive.formats.line.json.JsonDeserializer.deserialize(JsonDeserializer.java:157)
	at io.trino.plugin.hive.line.LinePageSource.getNextPage(LinePageSource.java:62)
@ebyhr ebyhr added the hive Hive connector label Jan 17, 2024
@ebyhr ebyhr changed the title Hive connector can write numeric NaN and ±inifinity with JSON format, but can't read the value Hive connector can write numeric NaN and ±infinity with JSON format, but can't read the value Jan 17, 2024
@ebyhr
Copy link
Member Author

ebyhr commented Jan 17, 2024

Probably, we should throw an exception when writing such values.

@hashhar
Copy link
Member

hashhar commented Jan 17, 2024

Indeed, NaN and Infinity are not valid in JSON. https://www.json.org/json-en.html

@sug-ghosh
Copy link

Can I pick this one?

@sug-ghosh sug-ghosh self-assigned this Dec 9, 2024
@sug-ghosh sug-ghosh linked a pull request Dec 22, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sug-ghosh sug-ghosh

Labels
hive Hive connector
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Avoid writing NaN and Infinity with json format table sug-ghosh/trino
3 participants
@ebyhr @hashhar @sug-ghosh