v0.15: Experimental new CSV-, and Geographic integrations and many other fixes

[Jolanrensen]

This release contains several new features, tons of fixes and two new exciting experimental new integrations:

• Experimental new CSV parser based on Deephaven-CSV. See below for more information.
• Experimental new GeoDataFrame class for working with geographical data (from GeoJson/Shapefile) and plotting it with Kandy. See below for more information.
• Full BigInteger support:
  Just like we support the BigDecimal numbers, DataFrame now also supports BigInteger in parsing, converting, statistics, column arithmetics, etc.
• Custom SQL DataBase registration
• Improved parsing:
  Parsing and converting String columns to other types is now faster. We also introduce the new experimental ParserOptions.useFastDoubleParser setting which uses FastDoubleParser for faster and more flexible Double parsing.
• We continue improving our Compiler Plugin with every release. See below for more information.
• See this notebook for some more information about the changes.

New Experimental CSV integration

DataFrame's CSV parsing has been based on Apache Commons CSV from the beginning. While this has been sufficient for most applications, it had some issues like running out of memory, performance, and our API lacking in clarity, documentation, and completeness.

For DataFrame 0.15, we introduce a new separate package org.jetbrains.kotlinx:dataframe-csv which tries to solve all these issues at once. It's based on Deephaven-CSV which makes it faster and more memory efficient. And since we built it from the ground up, we made sure the API was complete, predictable, and documented carefully.

To try it yourself, explicitly add the dependency org.jetbrains.kotlinx:dataframe-csv to your project. In notebooks you can add enableExperimentalCsv=true to the %use-magic, like %use dataframe(enableExperimentalCsv=true).
Use the new DataFrame.readCsv()/DataFrame.readTsv()/DataFrame.readDelim() functions over the old DataFrame.readCSV() ones.

We happily await your feedback!

New Experimental Geo integration

Kandy v0.8 introduces geo-plotting which allows you to visualize geospatial/geographical data using the awesome Kandy DSL. To make working with this geographical data (from GeoJson/Shapefile) easier, we happily accepted the GeoDataFrame PR from the Kandy team.

To try it yourself, explicitly add the dependency org.jetbrains.kotlinx:dataframe-geo to your project or notebook (with the repository maven("https://repo.osgeo.org/repository/release")) and use GeoDataFrame.readGeoJson() or GeoDataFrame.readShapeFile() to get started!

Features

• New CSV implementation by @Jolanrensen in New CSV implementation #903
• GeoDataFrame init by @AndreiKingsley in GeoDataFrame init #909
• Change default flatten parent-child separator to "_" by @Jolanrensen in Change default flatten parent-child separator to "_" #920
• Split OpenAPI in module needed for user projects and module needed for code-generation by @koperagen in Split OpenAPI in module needed for user projects and module needed for code-generation #916
• Support read unstructured excel file by @khm0651 in Support read unstructured excel file #901
• Fast double parser by @Jolanrensen in Fast double parser #935
• Implemented custom SQL DB registration by @zaleslaw in Implemented custom SQL DB registration #917
• Render FormattedFrame stored inside columns as HTML by @koperagen in Render FormattedFrame stored inside columns as HTML #944
• Adding some missing converters by @Jolanrensen in Adding some missing converters #958
• Full BigInteger support by @Jolanrensen in Full BigInteger support #972

Compiler Plugin

• [Compiler plugin] Lower frontend generated implicit receivers by @koperagen in [Compiler plugin] Lower frontend generated implicit receivers #869
• Generate valid code in transform(call) when interpret(call) fails by @koperagen in Generate valid code in transform(call) when interpret(call) fails #907
• [Compiler plugin] Support dataFrameOf(Pair<String, List) by @koperagen in [Compiler plugin] Support dataFrameOf(Pair<String, List<T>) #908
• [Compiler plugin] Add a mechanism to handle function calls to stdlib that can appear as df api arguments by @koperagen in [Compiler plugin] Add a mechanism to handle function calls to stdlib that can appear as df api arguments #914
• [Compiler plugin] Generate ColumnName annotations on frontend for all names that contain illegal characters by @koperagen in [Compiler plugin] Generate ColumnName annotations on frontend for all names that contain illegal characters #913
• Revert insertGenericTreeImpl by @koperagen in Revert insertGenericTreeImpl #923
• [Compiler plugin] Propagate nullability in toDataFrame tree conversion by @koperagen in [Compiler plugin] Propagate nullability in toDataFrame tree conversion #942
• Add castTo(Function) overload for workflows that use compiler plugin by @koperagen in Add castTo(Function) overload for workflows that use compiler plugin  #948
• [Compiler plugin] Setup call transformer pipeline to handle (...) -> DataRow functions by @koperagen in [Compiler plugin] Setup call transformer pipeline to handle (...) -> DataRow functions #918
• Compiler plugin read improvements by @koperagen in Compiler plugin read improvements #949
• [Compiler plugin] Support valueCounts by @koperagen in [Compiler plugin] Support valueCounts #951

Fixes

• Adding contracts for Anycol.isValueColumn etc. for smart-casting by @Jolanrensen in Adding contracts for Anycol.isValueColumn etc. for smart-casting #882
• Fix publish indexes docs by @Jolanrensen in Fix publish indexes docs #885
• Update algolia index builder by @koperagen in Update algolia index builder #895
• Find KSP Configurations that are Added Later by @mgroth0 in Find KSP Configurations that are Added Later #881
• Partially inline AnyFrame typealias in return type position by @koperagen in Partially inline AnyFrame typealias in return type position #888
• Deprecating DataFrame.read("", delimiter =) by @Jolanrensen in Deprecating DataFrame.read("", delimiter =) #902
• Parsing improvements by @Jolanrensen in Parsing improvements #874
• Fixed local classes being inferred as Any by changing visibility check by @Jolanrensen in Fixed local classes being inferred as Any by changing visibility check #929
• Open File in readExcel in read-only mode by @koperagen in Open File in readExcel in read-only mode #931
• Adds the binary compatibility validator plugin by @Jolanrensen in Adds the binary compatibility validator plugin #938
• Fixes nulls in framecols and improves column creation situation by @Jolanrensen in Fixes nulls in framecols and improves column creation situation #925
• Specify slf4j-api instead of slf4j-simple. by @erikogenvik in Specify slf4j-api instead of slf4j-simple. #934
• [Important Fix!] Parse started to removed unselected columns by @Jolanrensen in [Important Fix!] Parse started to removed unselected columns #947
• Fixed error message for ColumnAccessor by @zaleslaw in Fixed error message for ColumnAccessor #953
• fix crs by @AndreiKingsley in fix crs #955
• Added inferNullability test for other databases by @zaleslaw in Added inferNullability test for other databases #954
• Fix: disabled FastDoubleParser debug logs overload in the tests by @Jolanrensen in Fix: disabled FastDoubleParser debug logs overload in the tests #956
• describe() fixes by @Jolanrensen in describe() fixes #937
• Fix IO closing & add new useful extensions by @AndreiKingsley in Fix IO closing & add new useful extensions #960
• Remove dependency on fuel by @koperagen in Remove dependency on fuel #969
• [fix] Parser should be skipped after it fails to parse value by @koperagen in [fix] Parser should be skipped after it fails to parse value #975
• small fix for file clash at impl/io/readDelim.kt by @Jolanrensen in small fix for file clash at impl/io/readDelim.kt #976
• Fixed csv dependencies by @Jolanrensen in Fixed csv dependencies #977
• Bumped deprecations of startsWith and endsWith in CS DSL to Error by @Jolanrensen in Bumped deprecations of startsWith and endsWith in CS DSL to Error #978
• Version bumps for 0.15 by @Jolanrensen in Version bumps for 0.15 #980

Docs and Examples

• READMEs by @Jolanrensen in READMEs #900
• Make it clear that dataframes are immutable by @dmcg in Make it clear that dataframes are immutable #924
• Add examples for rename by @koperagen in Add examples for rename #952

New Contributors

• @khm0651 made their first contribution in Support read unstructured excel file #901
• @erikogenvik made their first contribution in Specify slf4j-api instead of slf4j-simple. #934
• @dmcg made their first contribution in Make it clear that dataframes are immutable #924
• @AndreiKingsley made their first contribution in GeoDataFrame init #909

Full Changelog: v0.14.2...v0.15.0

---

This discussion was created from the release v0.15: Experimental new CSV-, and Geographic integrations and many other fixes.