DataFrame.parse() performance issue for wide DF

[Jolanrensen]

We noticed this issue especially when parsing DataFrames with lots of String columns, such as a wide CSV file.

If you run DataFrame.parse(), each column is getting parsed one at a time.

If a column has type String, then tryParse goes over each parser in Parsers and if any of the values of the column cannot be parsed it will try the next parser.

These parsers are ordered like Int -> Long -> Instant -> LocalDateTime -> ... -> Json -> String. Which means that for "normal" string columns that need no parsing, at least one cell in the column is attempted to be parsed 17 times. Many of these attempts are achieved by catchSilent {} blocks, which catch any exception thrown an return null if they do. And they are heavy: https://www.baeldung.com/java-exceptions-performance.

This is easily measurable by creating two wide dataframes, one with columns that can be parsed to ints and another with cols that cannot be parsed and must remain strings:

We can see that parsing the wide string DF takes a considerable amount of time more.

And this is mostly due to Instant.parse, toLocalDateTimeOrNull, etc., and, most importantly, all fillInStackTrace calls at the top of the graph, a.k.a the exceptions of the parsers. We might be able to improve this :)

Looking at the parsers there are some interesting observations and possible solutions:

• Kotlin's Instant.parse() is a lot slower than Java's. We should use Java's and .toKotlin it.
• Many parsers are duplicates, like Java's LocalDateTime and Kotlin's. If a String can be parsed as date time, it will pick Kotlin's every time. If it cannot, it will fail both the Kotlin and Java one, creating a useless exception. We should drop the java duplicates.
• Exceptions are heavy. toIntOrNull and toLongOrNull are so fast, the time isn't even shown. If a library offers a canParse() function, we should use it.
• We should try to parallelize the parsing. Columns don't depend on each other. Parsing is built on convert to which is built on replace with, so that's where the parallelization should occur. Relevant issue: Parallel computations #723