18.0.0.md

18.0.0 (2023-02-10)

Full Changelog

Breaking changes:

• Use DataFusionError instead of ArrowError in SendableRecordBatchStream #5101 (comphead)
• Update to arrow 32 and Switch to RawDecoder for JSON #5056 [sql] (tustvold)

Implemented enhancements:

• DiskManager to create a spill folder if doesn't exist #5186
• cast expression may cause duplicate column name error #5174
• Add type coercion from Dictionary to string for regular expressions #5154
• Unnecessary Filter on Parquet datasources #5149
• [sqllogictest] Support pg_typeof for Postgres compatibility tests #5147
• Supporting Grafana global variables #5144
• add example for standalone DataFusion server which supports Arrow Flight SQL JDBC driver #5139
• Support for InList in datafusion-substrait #5134
• Pipeline file opening in FileStream #5129
• Make parse_physical_expr public #5107
• Use DataFusionError in SendableRecordBatchStream #5039
• Interval coercion:date_bin('1 hour',...) does not work but date_bin(interval '1 hour', ... does #4853
• Explain <query> should not fail if meeting errors when optimizing the query #4766
• Add option to determine whether to convert identifiers #4551
• Replace &Option<T> with Option<&T>. #4424
• Error type in RecordBatchStream #4172
• Support non-equi join (e.g. ON clause) in Dataframe API #1254
• Allow ParquetExec to parallelize work based on row groups #137

Fixed bugs:

• Confusing schema errors when using window partition #5229
• Propagating empty_relation generate an illegal plan #5218
• The test in_list_types_struct_literal fails when setting skip_failed_rules as false #5217
• Placeholder values are not replaced in ScalarSubqueries #5215
• Querying against delta lake table does not seem to work #5202
• Arithmetic operation doesn't work with DictionaryArray #5193
• simplify_expr() invoke nullable() exist bug #5191
• CI is currently broken on git diff: Not a git repository #5180
• write_csv/json/parquet isn't cancel safe #5178
• no hyperlink to blaze-rs [doc: README-"Use Cases"] #5175
• Arithmetic scalar operation doesn't work with DictionaryArray #5150
• Sort operator disappear in physical_plan #5100
• Window function error: InvalidArgumentError("number of columns(27) must match number of fields(35) in schema" #5090
• INSERT statements without target column list are not working #5078
• fix file stream time scanning metrics bug #5019
• Date before 1678 causes panic #4875
• Can not ORDER BY an aliased group column #4854
• The filters expressions in TableScan may contain fields not included in schema. #4793
• Comparing a Timestamp to a Date32 fails #4644
• String --> TableReference parsing does not properly handle " and . #4532
• can't compare NULL type with NULL type #4335
• Add ambiguous check when generate selection plan #4196
• Internal error in CAST from Timestamp[us] #3922
• Run median expr on parquet file column got error #3805
• aliasing a field renders it missing in the order by clause #669
• Querying datetime data in DataFusion with an embedded timezone always fails #153

Documentation updates:

• Update README.md fix [welcoming community] links #5232 (jiangzhx)
• Update README.md update blaze-rs link to https://github.com/blaze-init/blaze #5190 (jiangzhx)
• Typo of greptimedb #5103 (fengjiachun)
• chore: change DataBend to Databend #5096 (xudong963)

Closed issues:

• Change coerced type for comparison between timestamp with date to timestamp #4761

Merged pull requests:

• fix: correct expected error in test #5224 (jackwener)
• bugfix: fix propagating empty_relation generates an illegal plan #5219 (yukkit)
• Replace placeholders in ScalarSubqueries #5216 [sql] (avantgardnerio)
• Dataframe join_on method #5210 [sql] (Jefffrey)
• bugfix: fix eval nullalbe() in simplify_exprs #5208 (jackwener)
• minor: remove unnecessary clone #5207 (Ted-Jiang)
• minor: extract merge_schema() function. #5203 (jackwener)
• minor: remove unnecessary continue #5200 (xiaoyong-z)
• fix(MemTable): make it cancel-safe and fix parallelism #5197 (DDtKey)
• fix: make write_csv/json/parquet cancel-safe #5196 (DDtKey)
• Support arithmetic operation on DictionaryArray #5194 (viirya)
• sqllogicaltest: add cleanup and use rowsort. #5189 (jackwener)
• bugfix: fix TableScan may contain fields not included in schema #5188 (jackwener)
• Create disk manager spill folder if doesn't exist #5185 (comphead)
• Parse identifiers properly for TableReferences #5183 [sql] (Jefffrey)
• Fix decimal scalar dyn kernels #5179 (viirya)
• Patch git Safe Paths in CI #5177 (tustvold)
• Add initial support for serializing physical plans with Substrait #5176 (andygrove)
• Bump tokio from 1.24.1 to 1.24.2 in /datafusion-cli #5172 (dependabot[bot])
• Make EnforceSorting global sort aware, fix sort mis-optimizations involving unions, support parallel sort + merge transformations #5171 (mustafasrepo)
• Update substrait README.md #5168 (jiangzhx)
• Switch to use sum kernel from arrow-rs for Decimal128 #5167 (sunchao)
• FileStream: Open next file in parallel while decoding #5161 (thinkharderdev)
• Fix FairSpillPool try_grow for non-spillable consumers #5160 (tustvold)
• fix: treat unsupported SQL plans as "not implemented" #5159 (crepererum)
• Compare NULL types #5158 (melgenek)
• chore: add object_name_to_table_reference in SqlToRel #5155 [sql] (jiacai2050)
• Ambiguity check for where selection #5153 [sql] (Jefffrey)
• feat: Type coercion for Dictionary(_, _) to Utf8 for regex conditions #5152 (stuartcarnie)
• Support arithmetic scalar operation with DictionaryArray #5151 (viirya)
• [sqllogictest] Support pg_typeof #5148 (melgenek)
• Date to Timestamp cast #5140 (comphead)
• add example for Flight SQL server that supports JDBC driver #5138 (kmitchener)
• Add in-list test #5135 (nseekhao)
• Bug fix: Empty Record Batch handling #5131 (mustafasrepo)
• Add option to control whether to normalize ident #5124 [sql] (jiacai2050)
• Make parse_physical_expr public #5118 (comphead)
• Support coercing utf8 to interval and timestamp (including arguments to date_bin) #5117 (alamb)
• Fix release issues #5116 [sql] (andygrove)
• minor: port date_bin tests to sqllogictests #5115 (alamb)
• Minor: reduce code duplication using rewrite_expr #5114 [sql] (alamb)
• Replace &Option<T> with Option<&T> #5113 (gaoxinge)
• Improve get_meet_of_orderings to check for common prefixes #5111 (ozankabak)
• [sqllogictest] Apply rowsort when there is no explicit order by #5110 (melgenek)
• Add unnest_column to DataFrame #5106 (vincev)
• Minor: reduce indent level in page filter pruning code #5105 (alamb)
• Replace &Option<T> with Option<&T> #5102 (gaoxinge)
• Minor: remove unused methods in datafusion/optimizer/src/utils.rs #5098 (ygf11)
• ci: don't trigger rust ci for doc changes #5097 (xudong963)
• sqllogicaltest: fix unstable slt case. #5095 (jackwener)
• chore: update cranelift-module #5094 (jackwener)
• refactor: Add rewrite_expr convenience method for rewriting Exprs #5092 (alamb)
• Minor: extract sort col rewrite into its own module, add unit tests #5088 (alamb)
• [sqllogictest] Move decimal.rs tests #5086 (melgenek)
• Insert target columns empty fix #5079 [sql] (gruuya)
• sqllogicaltest: move union.rs #5075 (jackwener)
• Support ORDER BY an aliased column #5067 [sql] (alamb)
• Parquet parallel scan #5057 (korowa)
• [BugFix] fix file stream time scanning metrics bug #5020 (xiaoyong-z)
• Show optimization errors in explain #4819 [sql] (Jefffrey)