-
Notifications
You must be signed in to change notification settings - Fork 900
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add index creation on orderby columns #7436
Conversation
@mkindahl, @akuzm: please review this pull request.
|
Lets make sure we don't run segmentwise recompression on chunks without segmentby because currently its suboptimal vs doing a full recompress. |
cf5d71c
to
b68e994
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets disable segmentwise recompression for these chunks for now, until we can optimize it in the near future.
b68e994
to
36f796f
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #7436 +/- ##
==========================================
+ Coverage 80.06% 82.30% +2.23%
==========================================
Files 190 238 +48
Lines 37181 43794 +6613
Branches 9450 10995 +1545
==========================================
+ Hits 29770 36043 +6273
- Misses 2997 3415 +418
+ Partials 4414 4336 -78 ☔ View full report in Codecov by Sentry. |
36d03d6
to
0e14a8a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved but I think we need to add a comment on why we are disabling segmentwise recompression for these certain chunks (even if it is temporary).
0e14a8a
to
f8e4504
Compare
Previously, for chunks without a segmentby column specified in the compression settings, no indexes were created. This change allows indexes to be created on orderby columns only in this scenario. Since there is no segmentby in the compression settings for these chunks we also disable segmentwise recompression for them as it is more optimal to do a full recompression in this case.
f8e4504
to
8d743d3
Compare
This release contains performance improvements and bug fixes since the 2.17.2 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#6901: Add hypertable support for transition tables. * timescale#7104: Hypercore table access method. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column * timescale#7390: Disable custom `hashagg` planner code. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7412: Add GUC for `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. * timescale#7433 Add support for merging chunks * timescale#7436 Add index creation on orderby columns * timescale#7443: Add hypercore function and view aliases. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7486: Prevent building against postgres versions with broken ABI. * timescale#7521 Add optional `force` argument to `refresh_continuous_aggregate` * timescale#7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565 Add hint when hypertable creation fails * timescale#7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fixes the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517 Fixes performance regression on `cagg_migrate` procedure * timescale#7527 Restart scheduler on error * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566 Improve transaction check in CAgg refresh * timescale#7584 Fix NaN-handling for vectorized aggregation **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @k-rus for suggesting the improvement * @pgloader for reporting the issue in an internal background job. * @staticlibs for sending PR to improve transaction check in CAgg refresh * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release contains performance improvements and bug fixes since the 2.17.2 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#6901: Add hypertable support for transition tables. * timescale#7104: Hypercore table access method. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column * timescale#7390: Disable custom `hashagg` planner code. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7412: Add GUC for `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. * timescale#7433 Add support for merging chunks * timescale#7436 Add index creation on orderby columns * timescale#7443: Add hypercore function and view aliases. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7486: Prevent building against postgres versions with broken ABI. * timescale#7521 Add optional `force` argument to `refresh_continuous_aggregate` * timescale#7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565 Add hint when hypertable creation fails * timescale#7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fixes the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517 Fixes performance regression on `cagg_migrate` procedure * timescale#7527 Restart scheduler on error * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566 Improve transaction check in CAgg refresh * timescale#7584 Fix NaN-handling for vectorized aggregation **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @k-rus for suggesting the improvement * @pgloader for reporting the issue in an internal background job. * @staticlibs for sending PR to improve transaction check in CAgg refresh * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release contains performance improvements and bug fixes since the 2.17.2 release. We recommend that you upgrade at the next available opportunity. **Features** * #6901: Add hypertable support for transition tables. * #7104: Hypercore table access method. * #7271: Push down `order by` in real-time continuous aggregate queries. * #7295: Support `alter table set access method` on hypertable. * #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column * #7390: Disable custom `hashagg` planner code. * #7411: Change parameter name to enable hypercore table access method. * #7412: Add GUC for `hypercore_use_access_method` default. * #7413: Add GUC for segmentwise recompression. * #7433 Add support for merging chunks * #7436 Add index creation on orderby columns * #7443: Add hypercore function and view aliases. * #7455: Support `drop not null` on compressed hypertables. * #7458: Support vecorized aggregation with aggregate `filter` clauses that are also vectorizable. * #7482: Optimize recompression of partially compressed chunks. * #7486: Prevent building against postgres versions with broken ABI. * #7521 Add optional `force` argument to `refresh_continuous_aggregate` * #7528 Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * #7565 Add hint when hypertable creation fails * #7587 Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API **Bugfixes** * #7378: Remove obsolete job referencing `policy_job_error_retention`. * #7409: Update `bgw_job` table when altering procedure. * #7410: Fix the `aggregated compressed column not found` error on aggregation query. * #7426: Fix `datetime` parsing error in chunk constraint creation. * #7432: Verify that the heap tuple is valid before using. * #7434: Fixes the segfault when internally setting the replica identity for a given chunk. * #7488: Emit error for transition table trigger on chunks. * #7514: Fix the error: `invalid child of chunk append`. * #7517 Fixes performance regression on `cagg_migrate` procedure * #7527 Restart scheduler on error * #7557: Fix null handling for in-memory tuple filtering. * #7566 Improve transaction check in CAgg refresh * #7584 Fix NaN-handling for vectorized aggregation **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @k-rus for suggesting the improvement * @pgloader for reporting the issue in an internal background job. * @staticlibs for sending PR to improve transaction check in CAgg refresh * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release improves `group by` performance of compressing data and filters, introduces the ability to add secondary indexes, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in this release** * TAM * Significant performance improvements for aggregations using a `group by` with one column and/or using a filter clause. SIMD * Transition tables **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. **Features** * #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * #7104: Hypercore table access method. * #6901: Add hypertable support for transition tables. * #7482: Optimize recompression of partially compressed chunks. * #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * #7433: Add support for merging chunks. * #7271: Push down `order by` in real-time continuous aggregate queries. * #7455: Support `drop not null` on compressed hypertables. * #7295: Support `alter table set access method` on hypertable. * #7411: Change parameter name to enable hypercore table access method. * #7436: Add index creation on `order by` columns. * #7443: Add hypercore function and view aliases. * #7521: Add optional `force` argument to `refresh_continuous_aggregate`. * #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * #7565: Add hint when hypertable creation fails. * #7390: Disable custom `hashagg` planner code. * #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * #7486: Prevent building against PostgreSQL versions with broken ABI. * #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * #7413: Add GUC for segmentwise recompression. **Bugfixes** * #7378: Remove obsolete job referencing `policy_job_error_retention`. * #7409: Update `bgw_job` table when altering procedure. * #7410: Fix the `aggregated compressed column not found` error on aggregation query. * #7426: Fix `datetime` parsing error in chunk constraint creation. * #7432: Verify that the heap tuple is valid before using. * #7434: Fix the segfault when internally setting the replica identity for a given chunk. * #7488: Emit error for transition table trigger on chunks. * #7514: Fix the error: `invalid child of chunk append`. * #7517: Fix the performance regression on the `cagg_migrate` procedure. * #7527: Restart scheduler on error. * #7557: Fix null handling for in-memory tuple filtering. * #7566: Improve transaction check in CAGG refresh. * #7584: Fix NaN-handling for vectorized aggregation. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @pgloader for reporting the issue in an internal background job. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.17.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * #7104: Hypercore table access method. * #6901: Add hypertable support for transition tables. * #7482: Optimize recompression of partially compressed chunks. * #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * #7433: Add support for merging chunks. * #7271: Push down `order by` in real-time continuous aggregate queries. * #7455: Support `drop not null` on compressed hypertables. * #7295: Support `alter table set access method` on hypertable. * #7411: Change parameter name to enable hypercore table access method. * #7436: Add index creation on `order by` columns. * #7443: Add hypercore function and view aliases. * #7521: Add optional `force` argument to `refresh_continuous_aggregate`. * #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * #7565: Add hint when hypertable creation fails. * #7390: Disable custom `hashagg` planner code. * #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * #7486: Prevent building against PostgreSQL versions with broken ABI. * #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * #7413: Add GUC for segmentwise recompression. **Bugfixes** * #7378: Remove obsolete job referencing `policy_job_error_retention`. * #7409: Update `bgw_job` table when altering procedure. * #7410: Fix the `aggregated compressed column not found` error on aggregation query. * #7426: Fix `datetime` parsing error in chunk constraint creation. * #7432: Verify that the heap tuple is valid before using. * #7434: Fix the segfault when internally setting the replica identity for a given chunk. * #7488: Emit error for transition table trigger on chunks. * #7514: Fix the error: `invalid child of chunk append`. * #7517: Fix the performance regression on the `cagg_migrate` procedure. * #7527: Restart scheduler on error. * #7557: Fix null handling for in-memory tuple filtering. * #7566: Improve transaction check in CAGG refresh. * #7584: Fix NaN-handling for vectorized aggregation. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @pgloader for reporting the issue in an internal background job. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * timescale#7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * timescale#7104: Hypercore table access method. * timescale#6901: Add hypertable support for transition tables. * timescale#7482: Optimize recompression of partially compressed chunks. * timescale#7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * timescale#7433: Add support for merging chunks. * timescale#7271: Push down `order by` in real-time continuous aggregate queries. * timescale#7455: Support `drop not null` on compressed hypertables. * timescale#7295: Support `alter table set access method` on hypertable. * timescale#7411: Change parameter name to enable hypercore table access method. * timescale#7436: Add index creation on `order by` columns. * timescale#7443: Add hypercore function and view aliases. * timescale#7521: Add optional `force` argument to `refresh_continuous_aggregate`. * timescale#7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * timescale#7565: Add hint when hypertable creation fails. * timescale#7390: Disable custom `hashagg` planner code. * timescale#7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * timescale#7486: Prevent building against PostgreSQL versions with broken ABI. * timescale#7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * timescale#7413: Add GUC for segmentwise recompression. **Bugfixes** * timescale#7378: Remove obsolete job referencing `policy_job_error_retention`. * timescale#7409: Update `bgw_job` table when altering procedure. * timescale#7410: Fix the `aggregated compressed column not found` error on aggregation query. * timescale#7426: Fix `datetime` parsing error in chunk constraint creation. * timescale#7432: Verify that the heap tuple is valid before using. * timescale#7434: Fix the segfault when internally setting the replica identity for a given chunk. * timescale#7488: Emit error for transition table trigger on chunks. * timescale#7514: Fix the error: `invalid child of chunk append`. * timescale#7517: Fix the performance regression on the `cagg_migrate` procedure. * timescale#7527: Restart scheduler on error. * timescale#7557: Fix null handling for in-memory tuple filtering. * timescale#7566: Improve transaction check in CAGG refresh. * timescale#7584: Fix NaN-handling for vectorized aggregation. * timescale#7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
This release introduces the ability to add secondary indexes to the columnstore, improves group by and filtering performance through columnstore vectorization, and contains the highly upvoted community request of transition table support. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.18.0** * The ability to add secondary indexes to the columnstore through the new hypercore table access method. * Significant performance improvements through vectorization (`SIMD`) for aggregations using a group by with one column and/or using a filter clause when querying the columnstore. * Hypertables support triggers for transition tables, which is one of the most upvoted community feature requests. * Updated methods to manage Timescale's hybrid row-columnar store (hypercore) that highlight the usage of the columnstore which includes both an optimized columnar format as well as compression. **Dropping support for Bitnami images** After the recent change in Bitnami’s [LTS support policy](bitnami/containers#75671), we are no longer building Bitnami images for TimescaleDB. We recommend using the [official TimescaleDB Docker image](https://hub.docker.com/r/timescale/timescaledb-ha) **Deprecation Notice** We are deprecating the following parameters, functions, procedures and views. They will be removed with the next major release of TimescaleDB. Please find the replacements in the table below: | Deprecated | Replacement | Type | | --- | --- | --- | | decompress_chunk | convert_to_rowstore | Procedure | | compress_chunk | convert_to_columnstore | Procedure | | add_compression_policy | add_columnstore_policy | Function | | remove_compression_policy | remove_columnstore_policy | Function | | hypertable_compression_stats | hypertable_columnstore_stats | Function | | chunk_compression_stats | chunk_columnstore_stats | Function | | hypertable_compression_settings | hypertable_columnstore_settings | View | | chunk_compression_settings | chunk_columnstore_settings | View | | compression_settings | columnstore_settings | View | | timescaledb.compress | timescaledb.enable_columnstore | Parameter | | timescaledb.compress_segmentby | timescaledb.segmentby | Parameter | | timescaledb.compress_orderby | timescaledb.orderby | Parameter | **Features** * #7341: Vectorized aggregation with grouping by one fixed-size by-value compressed column (such as arithmetic types). * #7104: Hypercore table access method. * #6901: Add hypertable support for transition tables. * #7482: Optimize recompression of partially compressed chunks. * #7458: Support vectorized aggregation with aggregate `filter` clauses that are also vectorizable. * #7433: Add support for merging chunks. * #7271: Push down `order by` in real-time continuous aggregate queries. * #7455: Support `drop not null` on compressed hypertables. * #7295: Support `alter table set access method` on hypertable. * #7411: Change parameter name to enable hypercore table access method. * #7436: Add index creation on `order by` columns. * #7443: Add hypercore function and view aliases. * #7521: Add optional `force` argument to `refresh_continuous_aggregate`. * #7528: Transform sorting on `time_bucket` to sorting on time for compressed chunks in some cases. * #7565: Add hint when hypertable creation fails. * #7390: Disable custom `hashagg` planner code. * #7587: Add `include_tiered_data` parameter to `add_continuous_aggregate_policy` API. * #7486: Prevent building against PostgreSQL versions with broken ABI. * #7412: Add [GUC](https://www.postgresql.org/docs/current/acronyms.html#:~:text=GUC) for the `hypercore_use_access_method` default. * #7413: Add GUC for segmentwise recompression. **Bugfixes** * #7378: Remove obsolete job referencing `policy_job_error_retention`. * #7409: Update `bgw_job` table when altering procedure. * #7410: Fix the `aggregated compressed column not found` error on aggregation query. * #7426: Fix `datetime` parsing error in chunk constraint creation. * #7432: Verify that the heap tuple is valid before using. * #7434: Fix the segfault when internally setting the replica identity for a given chunk. * #7488: Emit error for transition table trigger on chunks. * #7514: Fix the error: `invalid child of chunk append`. * #7517: Fix the performance regression on the `cagg_migrate` procedure. * #7527: Restart scheduler on error. * #7557: Fix null handling for in-memory tuple filtering. * #7566: Improve transaction check in CAGG refresh. * #7584: Fix NaN-handling for vectorized aggregation. * #7598: Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. **Thanks** * @bharrisau for reporting the segfault when creating chunks. * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * @k-rus for suggesting that we add a hint when hypertable creation fails. * @staticlibs for sending the pull request that improves the transaction check in CAGG refresh. * @uasiddiqi for reporting the `aggregated compressed column not found` error.
Previously, for chunks without a segmentby column specified in the compression settings, no indexes were created. This change allows indexes to be created on orderby columns only in this scenario.