29.0.0.md

29.0.0 (2023-08-11)

Full Changelog

Breaking changes:

• change the input_type parameter of the create_udaf function from DataType to Vec #7096 (jiangzhx)
• Implement array_slice and array_element, remove array_trim #6936 (izveigor)
• improve the ergonomics of creating field and list array accesses #7215 (izveigor)
• Update Arrow 45.0.0 And Datum Arithmetic, change Decimal Division semantics #6832 (tustvold)

Implemented enhancements:

• feat: support SQL array replacement and removement functions #7057 (izveigor)
• feat: array containment operator @> and <@ #6885 (izveigor)
• feat: add sqllogictests crate #7134 (tshauck)
• feat: allow datafusion-cli to accept multiple statements #7138 (NiwakaDev)
• feat: Add linear regression aggregate functions #7211 (2010YOUY01)

Fixed bugs:

• fix: disallow interval - timestamp #7086 (jackwener)
• fix: Projection columns_map remove name search #7099 (mustafasrepo)
• fix: fix index bug and add test to check it #7124 (mustafasrepo)
• fix: Fix panic in filter predicate #7126 (alamb)
• fix: correct count(*) alias #7081 (jackwener)
• fix: skip compression tests on --no-default-features #7172 (not-my-profile)
• fix: typo in substrait #7224 (waynexia)

Documentation updates:

• Add additional links to main README #7102 (alamb)
• docs: fix broken link #7177 (SteveLauC)

Merged pull requests:

• [Minor] Speedup to_array_of_size for Decimal128 #7055 (Dandandan)
• Replace array_contains with SQL array functions: array_has, array_has_any, array_has_all #6990 (jayzhan211)
• Add more Decimal256 type coercion #7047 (viirya)
• Create dfbench, split up tpch benchmark runner into modules #7054 (alamb)
• chore(deps): update sqlparser requirement from 0.35 to 0.36.1 #7051 (alamb)
• use ObjectStore for dataframe writes #6987 (devinjdangelo)
• Prepare 28.0.0 Release #7056 (andygrove)
• refactor: with_inputs() can use original schema to avoid recompute schema. #7069 (jackwener)
• Fix cli tests #7083 (mustafasrepo)
• Ignore blank lines and comments at the end of query files for datafusion-cli #7076 (sarutak)
• Support case sensitive column for with_column_renamed #7063 (comphead)
• Add Decimal256 to ScalarValue #7048 (viirya)
• Enrich CSV reader config: quote & escape #6927 (parkma99)
• [Refactor] PipelineFixer physical optimizer rule removal #7059 (metesynnada)
• fix: disallow interval - timestamp #7086 (jackwener)
• Add Utf8->Binary type coercion for comparison #7080 (jonahgao)
• Refactor Replace Repartition rule #7090 (mustafasrepo)
• change the input_type parameter of the create_udaf function from DataType to Vec #7096 (jiangzhx)
• fix: Projection columns_map remove name search #7099 (mustafasrepo)
• Minor: Refine doc comments for BuiltinScalarFunction::return_dimension #7045 (alamb)
• Relax check during aggregate partial mode. #7101 (mustafasrepo)
• refactor byte_to_string and string_to_byte #7091 (parkma99)
• Minor: add test + docs for 2 argument trunc with columns #7042 (alamb)
• Move inactive projects to a different section #7104 (alamb)
• Port remaining information_schema rust tests to sqllogictests #7050 (palash25)
• Change rust-version in Cargo.toml to comply with MSRV #7107 (sarutak)
• create all needed folders in advance for benchmarks #7105 (smiklos)
• Initial support for functional dependencies handling primary key and unique constraints #7040 (mustafasrepo)
• Add ClickBench queries to DataFusion benchmark runner #7060 (alamb)
• feat: support SQL array replacement and removement functions #7057 (izveigor)
• [doc], [minor]. Update docstring of group by rewrite. #7111 (mustafasrepo)
• Add additional links to main README #7102 (alamb)
• fix: fix index bug and add test to check it #7124 (mustafasrepo)
• fix: Fix panic in filter predicate #7126 (alamb)
• Add MSRV check as a GA job #7123 (sarutak)
• Minor: move AnalysisContext out of physical_expr and into its own module #7127 (alamb)
• fix: correct count(*) alias #7081 (jackwener)
• make_array with column of list #7137 (jayzhan211)
• feat: array containment operator @> and <@ #6885 (izveigor)
• [MINOR]: Make memory exec partition number =1, in test utils #7148 (mustafasrepo)
• Substrait union/union all #7117 (nseekhao)
• minor: Remove mac m1 compilation for size_of_scalar test #7151 (mustafasrepo)
• chore: add config option for allowing bounded use of sort-preserving operators #7164 (wolffcm)
• chore: edition use workspace #7140 (jackwener)
• [bug]: Fix multi partition wrong column requirement bug #7129 (mustafasrepo)
• Refactor memory_limit tests to make them easier to extend #7131 (alamb)
• Minor: show output ordering in MemoryExec #7169 (alamb)
• Move ordering equivalence, and output ordering for joins to util functions #7167 (mustafasrepo)
• Add regr_slope() aggregate function #7135 (2010YOUY01)
• Add expression for array_agg #7159 (willrnch)
• fix: skip compression tests on --no-default-features #7172 (not-my-profile)
• HashJoin order fixing #7155 (metesynnada)
• tweak: demote heading levels in PR template #7176 (not-my-profile)
• feat: add sqllogictests crate #7134 (tshauck)
• docs: fix broken link #7177 (SteveLauC)
• Add nanvl builtin function #7171 (sarutak)
• chore(deps): update apache-avro requirement from 0.14 to 0.15 #7174 (jackwener)
• make dataframe.task_ctx public #7183 (milenkovicm)
• feat: allow datafusion-cli to accept multiple statements #7138 (NiwakaDev)
• Add plan_err! error macro #7115 (comphead)
• refactor: add ExecutionPlan::file_scan_config to avoid downcasting #7175 (not-my-profile)
• Minor: Add documentation + diagrams for ExternalSorter #7179 (alamb)
• Support simplifying expressions such as ~ ^(ba_r|foo)$ , where the string includes underline #7186 (tanruixiang)
• Add MemoryReservation::{split_off, take, new_empty} #7184 (alamb)
• Update bench.sh to only run 5 iterations #7189 (alamb)
• Implement array_slice and array_element, remove array_trim #6936 (izveigor)
• Unify DataFrame and SQL (Insert Into) Write Methods #7141 (devinjdangelo)
• Minor: Further Increase stack_size to prevent roundtrip_deeply_nested test stack overflow #7208 (devinjdangelo)
• Don't track files generated by regen.sh #7204 (sarutak)
• Update some docs/scripts to reflect the removed/added packages. #7202 (sarutak)
• Implement array_repeat, remove array_fill #7199 (izveigor)
• Use tokio only if running from a multi-thread tokio context #7205 (viirya)
• Remove Outdated NY Taxi benchmark #7210 (alamb)
• improve the ergonomics of creating field and list array accesses #7215 (izveigor)
• [MINOR] Document refactor on NestedLoopJoin #7217 (metesynnada)
• Docs: Add GlareDB to list of DataFusion users #7223 (alamb)
• fix: typo in substrait #7224 (waynexia)
• Minor: Add constructors to GetFieldAccessExpr and add docs #7219 (alamb)
• chore: required at least 1 approve before merge #7226 (jackwener)
• feat: Add linear regression aggregate functions #7211 (2010YOUY01)
• Add Expr::field, Expr::index, and Expr::slice, add docs #7218 (alamb)
• Extend insert into support to include Json backed tables #7212 (devinjdangelo)
• Minor: rename GetFieldAccessCharacteristic and add docs #7220 (alamb)
• Minor: Remove unecessary clone_with_replacement #7232 (alamb)
• Update Arrow 45.0.0 And Datum Arithmetic, change Decimal Division semantics #6832 (tustvold)
• Support make_array null handling in nested version #7207 (jayzhan211)
• [Minor], Bug Fix: Add empty ordering check at the source. #7230 (mustafasrepo)
• Minor: with preserve order now receives argument #7231 (mustafasrepo)
• Minor: Remove [[example]] table from datafusion-examples/Cargo.toml #7235 (sarutak)
• Remove additional cast from TPCH q8 #7233 (viirya)
• Minor: Move project_schema to datafusion_common #7237 (alamb)
• Minor: Extract ExecutionPlanVisitor to its own module #7236 (alamb)
• Minor: Move streams out of physical_plan module #7234 (alamb)
• doc: Add link to contributor's guide for new functions within the src #7240 (2010YOUY01)
• Account for memory usage in SortPreservingMerge (#5885) #7130 (alamb)
• Deprecate batch_byte_size #7245 (alamb)
• Minor: Move Partitioning andDistribution to physical_expr #7238 (alamb)
• Minor: remove duplication in create_writer #7229 (alamb)
• Support array flatten sql function #7239 (jayzhan211)
• Minor: fix clippy for memory_limit test #7248 (yjshen)
• Update physical_plan tests to not use SessionContext #7243 (alamb)
• Add API to make unnest consistent with DuckDB/ClickHouse, add option for preserve_nulls, update docs #7168 (alamb)
• chore(sqllogictests-doc): add testing set up #7258 (appletreeisyellow)
• Avoid to use TempDir::into_path for temporary dirs expected to be deleted automatically #7252 (sarutak)
• [MINOR]: update benefits_from_input_partitioning implementation for projection and repartition #7246 (mustafasrepo)
• Adding order equivalence support on MemoryExec #7259 (metesynnada)
• chore(functions): fix function names typo #7269 (appletreeisyellow)