32.0.0.md

32.0.0 (2023-10-07)

Full Changelog

Breaking changes:

• Remove implicit interval type coercion from ScalarValue comparison #7514 (tustvold)
• Remove get_scan_files and ExecutionPlan::file_scan_config (#7357) #7487 (tustvold)
• Move FileCompressionType out of common and into core #7596 (haohuaijin)
• Update arrow 47.0.0 in DataFusion #7587 (tustvold)
• Rename bounded_order_preserving_variants config to prefer_exising_sort and update docs #7723 (alamb)

Implemented enhancements:

• Parallelize Stateless (CSV/JSON) File Write Serialization #7452 (devinjdangelo)
• Create a Priority Queue based Aggregation with limit #7192 (avantgardnerio)
• feat: add guarantees to simplification #7467 (wjones127)
• [Minor]: Produce better plan when group by contains all of the ordering requirements #7542 (mustafasrepo)
• Make AvroArrowArrayReader possible to scan Avro backed table which contains nested records #7525 (sarutak)
• feat: Support spilling for hash aggregation #7400 (kazuyukitanimura)
• Parallelize Parquet Serialization #7562 (devinjdangelo)
• feat: natively support more data types for the abs function. #7568 (jonahgao)
• feat: Parallel collecting parquet files statistics #7573 #7595 (hengfeiyang)
• Support hashing List columns #7616 (jonmmease)
• feat: Better large output display in datafusion-cli with --maxrows option #7617 (2010YOUY01)
• feat: make parse_float_as_decimal work on negative numbers #7648 (jonahgao)
• Update Default Parquet Write Compression #7692 (devinjdangelo)
• Support all the codecs supported by Avro #7718 (sarutak)
• Optimize "ORDER BY + LIMIT" queries for speed / memory with special TopK operator #7721 (Dandandan)

Fixed bugs:

• fix: inconsistent behaviors when dividing floating numbers by zero #7503 (jonahgao)
• fix: skip EliminateCrossJoin rule if inner join with filter is found #7529 (epsio-banay)
• fix: check for precision overflow when parsing float as decimal #7627 (jonahgao)
• fix: substrait limit when fetch is None #7669 (waynexia)
• fix: coerce text to timestamps with timezones #7720 (mhilton)
• fix: avro_to_arrow: Handle avro nested nullable struct (union) #7663 (Samrose-Ahmed)

Documentation updates:

• Documentation Updates for New Write Related Features #7520 (devinjdangelo)
• Create 2023 Q4 roadmap #7551 (graydenshand)
• docs: add section on supports_filters_pushdown #7680 (tshauck)
• Add LanceDB to the list of Known Users #7716 (alamb)
• Document crate feature flags #7713 (alamb)

Merged pull requests:

• Prepare 31.0.0 release #7508 (andygrove)
• Minor(proto): Implement TryFrom<&DFSchema> for protobuf::DfSchema #7505 (jonahgao)
• fix: inconsistent behaviors when dividing floating numbers by zero #7503 (jonahgao)
• Parallelize Stateless (CSV/JSON) File Write Serialization #7452 (devinjdangelo)
• Minor: Remove stray comment markings from encoding error message #7512 (devinjdangelo)
• Remove implicit interval type coercion from ScalarValue comparison #7514 (tustvold)
• Minor: deprecate ScalarValue::get_datatype() #7507 (Weijun-H)
• Propagate error from spawned task reading spills #7510 (viirya)
• Refactor the EnforceDistribution Rule #7488 (mustafasrepo)
• Remove get_scan_files and ExecutionPlan::file_scan_config (#7357) #7487 (tustvold)
• Simplify ScalarValue::distance (#7517) #7519 (tustvold)
• typo: change delimeter to delimiter #7521 (Weijun-H)
• Fix some simplification rules for floating-point arithmetic operations #7515 (jonahgao)
• Documentation Updates for New Write Related Features #7520 (devinjdangelo)
• [MINOR]: Move tests from repartition to enforce_distribution file #7539 (mustafasrepo)
• Update the async-trait crate to resolve clippy bug #7541 (metesynnada)
• Fix flaky test_sort_fetch_memory_calculation test #7534 (viirya)
• Move common code to utils #7545 (mustafasrepo)
• Minor: Add comments and clearer constructors to Interval #7526 (alamb)
• fix: skip EliminateCrossJoin rule if inner join with filter is found #7529 (epsio-banay)
• Create a Priority Queue based Aggregation with limit #7192 (avantgardnerio)
• feat: add guarantees to simplification #7467 (wjones127)
• [Minor]: Produce better plan when group by contains all of the ordering requirements #7542 (mustafasrepo)
• Minor: beautify interval display #7554 (Weijun-H)
• replace ptree with termtree #7560 (avantgardnerio)
• Make AvroArrowArrayReader possible to scan Avro backed table which contains nested records #7525 (sarutak)
• Fix a race condition issue on reading spilled file #7538 (sarutak)
• [MINOR]: Add is single method #7558 (mustafasrepo)
• Fix describe <table> to work without SessionContext #7441 (alamb)
• Make the tests in SHJ faster #7543 (metesynnada)
• feat: Support spilling for hash aggregation #7400 (kazuyukitanimura)
• Make backtrace as a cargo feature #7527 (comphead)
• Minor: Fix clippy by switching to timestamp_nanos_opt instead of (deprecated) timestamp_nanos #7572 (alamb)
• Update sqllogictest requirement from 0.15.0 to 0.16.0 #7569 (dependabot[bot])
• extract datafusion-physical-plan to its own crate #7432 (alamb)
• First and Last Accumulators should update with state row excluding is_set flag #7565 (viirya)
• refactor: simplify code of eliminate_cross_join.rs #7561 (jackwener)
• Update release instructions for datafusion-physical-plan crate #7576 (alamb)
• Minor: Update chrono pin to 0.4.31 #7575 (alamb)
• [feat] Introduce cacheManager in session ctx and make StatisticsCache share in session #7570 (Ted-Jiang)
• Enhance/Refactor Ordering Equivalence Properties #7566 (mustafasrepo)
• fix misplaced statements in sqllogictest #7586 (jonahgao)
• Update substrait requirement from 0.13.1 to 0.14.0 #7585 (dependabot[bot])
• chore: use the create_udwf function in simple_udwf, consistent with simple_udf and simple_udaf #7579 (tanruixiang)
• Implement protobuf serialization for AnalyzeExec #7574 (adhish20)
• chore: fix catalog's usage docs error and add docs about CatalogList trait #7582 (tanruixiang)
• Implement CardinalityAwareRowConverter while doing streaming merge #7401 (JayjeetAtGithub)
• Parallelize Parquet Serialization #7562 (devinjdangelo)
• feat: natively support more data types for the abs function. #7568 (jonahgao)
• implement string_to_array #7577 (casperhart)
• Create 2023 Q4 roadmap #7551 (graydenshand)
• chore: reduce physical-plan dependencies #7599 (crepererum)
• Minor: add githubs start/fork buttons to documentation page #7588 (alamb)
• Minor: add more examples for CREATE EXTERNAL TABLE doc #7594 (comphead)
• Update nix requirement from 0.26.1 to 0.27.1 #7438 (dependabot[bot])
• Update sqllogictest requirement from 0.16.0 to 0.17.0 #7606 (dependabot[bot])
• Fix panic in TopK #7609 (avantgardnerio)
• Move FileCompressionType out of common and into core #7596 (haohuaijin)
• Expose contents of Constraints #7603 (tv42)
• Change the unbounded_output API default #7605 (metesynnada)
• feat: Parallel collecting parquet files statistics #7573 #7595 (hengfeiyang)
• Support hashing List columns #7616 (jonmmease)
• [MINOR] Make the sink input aware of its plan #7610 (metesynnada)
• [MINOR] Reduce complexity on SHJ #7607 (metesynnada)
• feat: Better large output display in datafusion-cli with --maxrows option #7617 (2010YOUY01)
• Minor: add examples for arrow_cast and arrow_typeof to user guide #7615 (alamb)
• [MINOR]: Fix stack overflow bug for get field access expr #7623 (mustafasrepo)
• Group By All #7622 (berkaysynnada)
• Implement protobuf serialization for (Bounded)WindowAggExec. #7557 (vrongmeal)
• Make it possible to compile datafusion-common without default features #7625 (jonmmease)
• Minor: Adding backtrace documentation #7628 (comphead)
• fix(5975/5976): timezone handling for timestamps and date_trunc, date_part and date_bin #7614 (wiedld)
• Minor: remove unecessary Arcs in datetime_expressions #7630 (alamb)
• fix: check for precision overflow when parsing float as decimal #7627 (jonahgao)
• Update arrow 47.0.0 in DataFusion #7587 (tustvold)
• Add test crate to compile DataFusion with wasm-pack #7633 (jonmmease)
• Minor: Update documentation of case expression #7646 (ongchi)
• Minor: improve docstrings on SessionState #7654 (alamb)
• Update example in the DataFrame documentation. #7650 (jsimpson-gro)
• Add HTTP object store example #7602 (pka)
• feat: make parse_float_as_decimal work on negative numbers #7648 (jonahgao)
• Minor: add doc comments to ExtractEquijoinPredicate #7658 (alamb)
• [MINOR]: Do not add unnecessary hash repartition to the physical plan #7667 (mustafasrepo)
• Minor: add ticket references to parallel parquet writing code #7592 (alamb)
• Minor: Add ticket reference and add test comment #7593 (alamb)
• Support Avro's Enum type and Fixed type #7635 (sarutak)
• Minor: Migrate datafusion-proto tests into it own binary #7668 (ongchi)
• Upgrade apache-avro to 0.16 #7674 (sarutak)
• Move window analysis to the window method #7672 (mustafasrepo)
• Don't add filters to projection in TableScan #7670 (Dandandan)
• Minor: Improve TableProviderFilterPushDown docs #7685 (alamb)
• FIX: Test timestamp with table #7701 (jayzhan211)
• Fix bug in SimplifyExpressions #7699 (Dandandan)
• Enhance Enforce Dist capabilities to fix, sub optimal bad plans #7671 (mustafasrepo)
• docs: add section on supports_filters_pushdown #7680 (tshauck)
• Improve cache usage in CI #7678 (sarutak)
• fix: substrait limit when fetch is None #7669 (waynexia)
• minor: revert parsing precedence between Aggr and UDAF #7682 (waynexia)
• Minor: Move hash utils to common #7684 (jayzhan211)
• Update Default Parquet Write Compression #7692 (devinjdangelo)
• Stop using cache for the benchmark job #7706 (sarutak)
• Change rust.yml to run benchmark #7708 (sarutak)
• Extend infer_placeholder_types to support BETWEEN predicates #7703 (andrelmartins)
• Minor: Add comment explaining why verify benchmark results uses release mode #7712 (alamb)
• Support all the codecs supported by Avro #7718 (sarutak)
• Update substrait requirement from 0.14.0 to 0.15.0 #7719 (dependabot[bot])
• fix: coerce text to timestamps with timezones #7720 (mhilton)
• Add LanceDB to the list of Known Users #7716 (alamb)
• Enable avro reading/writing in datafusion-cli #7715 (alamb)
• Document crate feature flags #7713 (alamb)
• Minor: Consolidate UDF tests #7704 (alamb)
• Minor: fix CI failure due to Cargo.lock in datafusioncli #7733 (yjshen)
• MINOR: change file to column index in page_filter trace log #7730 (mapleFU)
• preserve array type / timezone in date_bin and date_trunc functions #7729 (mhilton)
• Remove redundant is_numeric for DataType #7734 (qrilka)
• fix: avro_to_arrow: Handle avro nested nullable struct (union) #7663 (Samrose-Ahmed)
• Rename SessionContext::with_config_rt to SessionContext::new_with_config_from_rt, etc #7631 (alamb)
• Rename bounded_order_preserving_variants config to prefer_exising_sort and update docs #7723 (alamb)
• Optimize "ORDER BY + LIMIT" queries for speed / memory with special TopK operator #7721 (Dandandan)
• Minor: Improve crate docs #7740 (alamb)
• [MINOR]: Resolve linter errors in the main #7753 (mustafasrepo)
• Minor: Build concat_internal() with ListArray construction instead of ArrayData #7748 (jayzhan211)
• Minor: Add comment on input_schema from AggregateExec #7727 (viirya)
• Fix column name for COUNT(*) set by AggregateStatistics #7757 (qrilka)
• Add documentation about type signatures, and export TIMEZONE_WILDCARD #7726 (alamb)
• [feat] Support cache ListFiles result cache in session level #7620 (Ted-Jiang)
• Support SHOW ALL VERBOSE to show settings description #7735 (comphead)