28.0.0.md

28.0.0 (2023-07-21)

Full Changelog

Breaking changes:

• Cleanup type coercion (#3419) #6778 (tustvold)
• refactor: encapsulate Alias as a struct #6795 (jackwener)
• Set DisplayAs to be a supertrait of ExecutionPlan #6835 (qrilka)
• [MINOR] Remove unnecessary api from MemTable #6861 (metesynnada)
• refactor: Merge Expr::Like and Expr::ILike #7007 (waynexia)

Implemented enhancements:

• feat: array_contains #6618 (izveigor)
• feat: support NULL in array functions #6662 (izveigor)
• feat: implement posgres style encode/decode #6821 (ozgrakkurt)
• feat: column support for array_append, array_prepend, array_position and array_positions #6805 (izveigor)
• feat: preserve metadata for Field and Schema in proto #6865 (jonahgao)
• feat: Add graphviz display format for execution plan. #6726 (liurenjie1024)
• feat: implement substrait join filter support #6868 (nseekhao)
• feat: column support for array_dims, array_ndims, cardinality and array_length #6864 (izveigor)
• feat: support for NestedLoopJoinExec in datafusion-proto #6902 (r4ntix)
• feat: add round trip test of physical plan in tpch unit tests #6918 (r4ntix)
• feat: implement substrait for LIKE/ILIKE expr #6840 (waynexia)
• feat: array functions treat an array as an element #6986 (izveigor)

Fixed bugs:

• fix: incorrect nullability of between expr #6786 (jonahgao)
• fix: incorrect nullability of InList expr #6799 (jonahgao)
• fix: from_plan generate Agg can be with different schema. #6820 (jackwener)
• fix: incorrect nullability of Like expressions #6829 (jonahgao)
• fix: incorrect simplification of case expr #7006 (jonahgao)
• fix: array_concat with arrays with different dimensions, add _list* aliases for _array* functions #7008 (izveigor)

Documentation updates:

• docs: Add encode and decode to the user guide #6856 (alamb)

Merged pull requests:

• chore(deps): update indexmap requirement from 1.9.2 to 2.0.0 #6766 (dependabot[bot])
• Support IsDistinctFrom and IsNotDistinctFrom on interval types #6776 (joroKr21)
• Protect main branch #6775 (tustvold)
• Prepare 27.0.0 release #6773 (andygrove)
• Support hex string literal #6767 (ShiKaiWi)
• feat: array_contains #6618 (izveigor)
• Make 'date_trunc' returns the same type as its input #6654 (Weijun-H)
• Fix inserting into a table with non-nullable columns #6722 (jonahgao)
• Cleanup type coercion (#3419) #6778 (tustvold)
• Properly project grouping set expressions #6777 (fsdvh)
• Minor: Simplify date_trunc code and add comments #6783 (alamb)
• Minor: Add array / array sqllogic tests for array_contains #6771 (alamb)
• Minor: Make date_trunc code easier to understand #6789 (alamb)
• feat: support NULL in array functions #6662 (izveigor)
• fix: incorrect nullability of between expr #6786 (jonahgao)
• Use checked division kernel #6792 (tustvold)
• Minor: add sqllogictests for binary data type #6770 (alamb)
• refactor: encapsulate Alias as a struct #6795 (jackwener)
• chore(deps): bump actions/labeler from 4.1.0 to 4.2.0 #6803 (dependabot[bot])
• Consistently coerce dictionaries for arithmetic #6785 (tustvold)
• Implement serialization for UDWF and UDAF in plan protobuf #6769 (parkma99)
• fix: incorrect nullability of InList expr #6799 (jonahgao)
• Fix timestamp_add_interval_months to pass any date #6815 (jayzhan211)
• Minor: Log TPCH benchmark results #6813 (alamb)
• Refactor Decimal128 averaging code to be vectorizable (and easier to read) #6810 (alamb)
• Minor: Encapsulate return_type and signature in AggregateFunction and WindowFunction #6748 (alamb)
• fix: from_plan generate Agg can be with different schema. #6820 (jackwener)
• [MINOR] Improve performance of create_hashes #6816 (Dandandan)
• Add fetch to SortPreservingMergeExec and SortPreservingMergeStream #6811 (Dandandan)
• chore(deps): update substrait requirement from 0.11.0 to 0.12.0 #6825 (dependabot[bot])
• Upgrade arrow 43 #6812 (tustvold)
• Fix cargo build warning #6831 (viirya)
• Simplify IsUnkown and IsNotUnkown expression #6830 (jonahgao)
• fix: incorrect nullability of Like expressions #6829 (jonahgao)
• Minor: Add one more assert to hash_array_primitive #6834 (alamb)
• revert #6595 #6820 #6827 (jackwener)
• Add Duration to ScalarValue #6838 (tustvold)
• Replace AbortOnDrop / AbortDropOnMany with tokio JoinSet #6750 (aprimadi)
• Add clickbench queries to sqllogictest coverage #6836 (alamb)
• feat: implement posgres style encode/decode #6821 (ozgrakkurt)
• chore(deps): update rstest requirement from 0.17.0 to 0.18.0 #6847 (dependabot[bot])
• [minior] support serde for some function #6846 (liukun4515)
• Support fixed_size_list for make_array #6759 (jayzhan211)
• Improve median performance. #6837 (vincev)
• Mismatch in MemTable of Select Into when projecting on aggregate window functions #6566 (berkaysynnada)
• feat: column support for array_append, array_prepend, array_position and array_positions #6805 (izveigor)
• MINOR: Fix ordering of the aggregate_source_with_order table #6852 (mustafasrepo)
• Return error when internal multiplication overflowing in decimal division kernel #6833 (viirya)
• Deprecate ScalarValue::and, ScalarValue::or (#6842) #6844 (tustvold)
• chore(deps): update bigdecimal requirement from 0.3.0 to 0.4.0 #6848 (dependabot[bot])
• feat: preserve metadata for Field and Schema in proto #6865 (jonahgao)
• Set DisplayAs to be a supertrait of ExecutionPlan #6835 (qrilka)
• [MINOR] Remove unnecessary api from MemTable #6861 (metesynnada)
• Adjustment of HashJoinExec APIs to Preserve Probe Side Order #6858 (metesynnada)
• [MINOR] Adding order into StreamingTableExec #6860 (metesynnada)
• Docs: try and clarify what PartitionEvaluator functions are called #6869 (alamb)
• docs: Add encode and decode to the user guide #6856 (alamb)
• Fix build on main due to logical conflict #6875 (alamb)
• feat: Add graphviz display format for execution plan. #6726 (liurenjie1024)
• Fix (another) logical conflict #6882 (alamb)
• Minor: Consolidate display related traits #6883 (alamb)
• test: parquet use the byte array as the physical type to store decimal #6851 (smallzhongfeng)
• Make streaming_merge public #6874 (kazuyukitanimura)
• Performance: Use a specialized sum accumulator for retractable aggregregates #6888 (alamb)
• Support array concatenation for arrays with different dimensions #6872 (jayzhan211)
• feat: implement substrait join filter support #6868 (nseekhao)
• feat: column support for array_dims, array_ndims, cardinality and array_length #6864 (izveigor)
• Add FixedSizeBinary support to binary_op_dyn_scalar #6891 (maxburke)
• Minor: deleted duplicated substrait integration test #6894 (alamb)
• Minor: add test cases with columns for math expressions #6787 (izveigor)
• Minor: reduce redundant code #6901 (smallzhongfeng)
• Minor: Add some more doc comments to BoundedAggregateStream #6881 (alamb)
• feat: support for NestedLoopJoinExec in datafusion-proto #6902 (r4ntix)
• Fix make_array null handling, update tests #6900 (alamb)
• chore(deps): bump actions/labeler from 4.2.0 to 4.3.0 #6911 (dependabot[bot])
• Minor: Add TPCH scale factor 10 to bench.sh, use 10 iteration #6893 (alamb)
• Minor: Add output to aggregrate_fuzz.rs on failure #6905 (alamb)
• allow window UDF to return null #6915 (mhilton)
• Minor: Add factory method to PartitionedFile to create File Scan #6909 (comphead)
• [minor]fix doc to remove duplicate content #6923 (liukun4515)
• Revert "chore(deps): update bigdecimal requirement from 0.3.0 to 0.4.0 (#6848)" #6896 (alamb)
• [Minor] Make FileScanConfig::project pub #6931 (Dandandan)
• feat: add round trip test of physical plan in tpch unit tests #6918 (r4ntix)
• Minor: Use thiserror to implement the From trait for DFSqlLogicTestError #6924 (jonahgao)
• parallel csv scan #6801 (2010YOUY01)
• Add additional test coverage for aggregaes using dates/times/timestamps/decimals #6939 (alamb)
• Replace repartition execs with sort preserving repartition execs #6921 (mertak)
• Vectorized hash grouping #6904 (alamb)
• Fix incorrect results in BitAnd GroupsAccumulator #6957 (alamb)
• Fixes for clippy 1.71 #6959 (alamb)
• Improve unnest_column performance #6903 (vincev)
• Pass schema_infer_max_records to JsonFormat. #6945 (vincev)
• deps: bump sqllogictest to 0.15.0 #6941 (jonahgao)
• Preserve field metadata across expressions in logical plans #6920 (dexterduck)
• Support equality and comparison between interval arrays and scalars #6948 (joroKr21)
• chore(deps): update bigdecimal requirement from 0.3.0 to 0.4.1 #6946 (dependabot[bot])
• feat: implement substrait for LIKE/ILIKE expr #6840 (waynexia)
• Minor: Add comments about initial value for BitAnd accumulator #6964 (alamb)
• [Functions] Support Arithmetic function COT() #6925 (Syleechan)
• Minor: remove duplication in Min/Max accumulator #6960 (alamb)
• [MINOR]Add new tests #6953 (mustafasrepo)
• Column support for array concat #6879 (jayzhan211)
• Minor: Add FixedSizeBinaryTest #6895 (alamb)
• [MINOR] Remove update state api from PartitionEvaluator #6966 (mustafasrepo)
• Fix required partitioning of Single aggregation mode #6950 (Dandandan)
• [MINOR] Remove global sort rule from planner #6965 (mustafasrepo)
• Column support for array_to_string #6940 (jayzhan211)
• chore: fix format #6991 (Weijun-H)
• Extend Ordering Equivalence Support #6956 (mustafasrepo)
• chore: break earlier in macro contains! #6989 (Weijun-H)
• fix: incorrect simplification of case expr #7006 (jonahgao)
• Minor: Add String/Binary aggregate tests #6962 (alamb)
• [MINOR] Supporting repartition joins conf in SHJ #6998 (metesynnada)
• [MINOR] Code refactor on hash join utils #6999 (metesynnada)
• feat: array functions treat an array as an element #6986 (izveigor)
• [MINOR] Moving some test utils from EnsureSorting to test_utils #7009 (metesynnada)
• MINOR: Bug fix, Use correct ordering equivalence when window expr contains partition by #7011 (mustafasrepo)
• refactor: Merge Expr::Like and Expr::ILike #7007 (waynexia)
• Docs: Add docs to RepartitionExec and architecture guide #7003 (alamb)
• Consolidate BoundedAggregateStream #6932 (alamb)
• Minor: Improve aggregate test coverage more #6952 (alamb)
• Don't store hashes in GroupOrdering #7029 (tustvold)
• Extract GroupValues (#6969) #7016 (tustvold)
• Refactor AnalysisContext and statistics() of FilterExec #6982 (berkaysynnada)
• Fix datafusion-cli/Dockerfile to build successfully #7031 (sarutak)
• functions: support trunc() function with one or two args #6942 (Syleechan)
• Move the column aliases below the SubqueryAlias #7035 (jonahgao)
• fix: array_concat with arrays with different dimensions, add _list* aliases for _array* functions #7008 (izveigor)
• Add support for ClickBench in bench.sh #7005 (alamb)
• Remove RowAccumulators and datafusion-row #6968 (alamb)
• Decimal256 coercion #7034 (jdye64)
• Double RawTable on grow instead of triple #7041 (tustvold)
• Specialize single column primitive group values #7043 (tustvold)