1.4.0

Core Library

• feat: add incremental lag (attribution window) for datetime, int, and float cursors by @donotpush in #1957
• LanceDB - (1) support merge key to merge chunked documents correctly - removes orphaned chunks (2) huge performance upgrade by loading data via arrow by @Pipboyguy in #1620
• Move exclude_keys() to dlt.common.utils by @burnash in #1966
• Fix BigQueryLoadJob hiding root cause exception by @xneg in #1992
• loads secrets from colab userdata and steamlit + bugfixes by @rudolfix in #1994
• Fix pagination issue in JSONResponseCursorPaginator with empty string cursor value by @kang8 in #2016
• fix: if name of distribution is None by @senickel in #2024
• allows to pass default values when writing specs by @rudolfix in #2018
• enable delta partitioning on arrow normalizer load id by @jorritsandbrink in #2022
• add session token to duckdb s3 secret by @jorritsandbrink in #2007
• Add user agent for Databricks by @VioletM in #1987
• Fix an incorrect missing dependency error by @burnash in #2001
• fix resource level max_table_nesting and normalizer performance tuning by @sh-rp in #2026
• move default pipelines of cores sources into source folders by @sh-rp in #1888
• duckdb filesystem custom secrets by @sh-rp in #2017
• allows for empty dataset clickhouse by @rudolfix in #2045
• add GCP default credential handling for delta table format by @jorritsandbrink in #2048
• enables merges for bigquery autodetect schema by @sh-rp in #2035
• logs warning if deduplication state is large by @willi-mueller in #1877
• Add core sources extras to requirements in dlt init by @burnash in #2028
• Fix merge write disposition for pyarrow and ClickHouse by @burnash in #2042

Experimental interfaces

dlt dataset public interface and docs coming next week.

• 1990 - dataset columns select and limit by @sh-rp in #2000

Docs

• Updated databricks destination documentation by @dat-a-man in #1984
• Docs: fix capitalization of some terms, fix typos by @burnash in #1988
• fix typo by @mariarice15 in #1995
• Fix Zendesk example: make test resilient to data changes by @burnash in #1999
• fix s3 credentials environment variable names by @seunggs in #2010
• remove ga add tm by @alexanderfifefd in #2008
• Super fast snippet linting & type checking by @sh-rp in #2019
• Fix the deprecation warning in .common.configuration.container by @burnash in #2025
• Added deploy with modal. by @dat-a-man in #1805
• Updated google cloud function documentation by @dat-a-man in #2034
• add warning for large delta memory footprint on filesystem docs page by @sh-rp in #2036
• simplify advanced section by @kning in #2037
• Added docs on how to deploy a pipeline using Google Cloud run by @dat-a-man in #2038
• Format Delta table section in the filesystem destination by @burnash in #2057
• Docs: add table formats to the sidebar by @burnash in #2060

New Contributors

• @xneg made their first contribution in #1992
• @seunggs made their first contribution in #2010
• @alexanderfifefd made their first contribution in #2008
• @kang8 made their first contribution in #2016
• @senickel made their first contribution in #2024
• @kning made their first contribution in #2037

Full Changelog: 1.3.0...1.4.0