Automatically run ANALYZE after babelfish extension create/upgrade #1922

sumitj824 · 2023-10-16T05:19:31Z

Description

Earlier, when having less than 50 schemas on the fresh instance causes the poor performance for problematic query when having large number of objects in database. In case of less than 50 schemas, sys.babelfish_namespace_ext catalog which stores the name of schema will have less than 50 rows which means autovacuum ANALYZE has never been run by PostgreSQL autovacuum daemon as the number of inserted rows doesn't exceeds the threshold criteria. Since the sys.babelfish_namespace_ext catalog has never been analyzed it leads bad query planning for large query. This commit add changes to automatically run ANALYZE after babelfish extension create/upgrade for all babelfish catalogs to avoid such issues in future.

Signed-off-by: Sumit Jaiswal [email protected]

Issues Resolved

Task: BABEL-4487

Test Scenarios Covered

Performance tests

Execution time for Problematic query:

select count(*) from sys.tables AS tbl 
inner join sys.all_columns as clmns on clmns.object_id = tbl.object_id 
inner join sys.types as usrt on usrt.user_type_id = clmns.user_type_id;

Before ANALYZE: 29 min query plan
After ANALYZE: 2 sec query plan

USE CASE BASED

Manually tested with creating table for which analyze will fail

jdbc_testdb=# call sys.analyze_babelfish_catalogs();
WARNING:  ANALYZE for babelfish catalog sys.test_table failed with error: data out of range for datetimes
CALL

Logs:

WARNING:  ANALYZE for babelfish catalog sys.test_table failed with error: data out of range for datetimes
CONTEXT:  PL/pgSQL function sys.analyze_babelfish_catalogs() line 17 at RAISE

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sumit Jaiswal <[email protected]>

contrib/babelfishpg_tsql/sql/ownership.sql

Signed-off-by: Sumit Jaiswal <[email protected]>

contrib/babelfishpg_tsql/sql/ownership.sql

test/python/expected/sql_validation_framework/expected_create.out

forestkeeper · 2023-10-23T18:23:45Z

contrib/babelfishpg_tsql/sql/upgrades/babelfishpg_tsql--3.3.0--3.4.0.sql

+			EXECUTE format('ANALYZE %I.%I', schema_name, babelfish_catalog.name);
+		EXCEPTION WHEN OTHERS THEN
+			GET STACKED DIAGNOSTICS error_msg = MESSAGE_TEXT;
+			RAISE WARNING 'ANALYZE for babelfish catalog %.% failed with error: %s', schema_name, babelfish_catalog.name, error_msg;


When we have this warning in the upgrade, what's our action item ?

Warning will help us to debug any performance issue after upgrade due the failure. I will add the same in Jira also.

how will it lead to action on customer or our side? A warning will go to upgrade scripts and might be lost if no one takes a look.

yes, warning will not lead to any action but it will be helpful for any performance issue observed just after the upgrade and logs are present.

Let us track this item via a Jira. Talk to Sharu about this. We should consider how we track successful MVUs and corresponding logs in long term to resolve issues down the line.

contrib/babelfishpg_tsql/sql/upgrades/babelfishpg_tsql--3.3.0--3.4.0.sql

surendravishnoi · 2023-10-24T08:56:45Z

contrib/babelfishpg_tsql/sql/upgrades/babelfishpg_tsql--3.3.0--3.4.0.sql

+			EXECUTE format('ANALYZE %I.%I', schema_name, babelfish_catalog.name);
+		EXCEPTION WHEN OTHERS THEN
+			GET STACKED DIAGNOSTICS error_msg = MESSAGE_TEXT;
+			RAISE WARNING 'ANALYZE for babelfish catalog %.% failed with error: %s', schema_name, babelfish_catalog.name, error_msg;


how will it lead to action on customer or our side? A warning will go to upgrade scripts and might be lost if no one takes a look.

Signed-off-by: Sumit Jaiswal <[email protected]>

… during source latest to target latest Signed-off-by: Sumit Jaiswal <[email protected]>

Signed-off-by: Sumit Jaiswal <[email protected]>

test/JDBC/upgrade/latest/schedule

test/JDBC/expected/AUTO_ANALYZE-vu-prepare.out

test/JDBC/input/AUTO_ANALYZE-before-15-5-or-14-10-vu-prepare.sql

Signed-off-by: Sumit Jaiswal <[email protected]>

…abelfish-for-postgresql#1922) Earlier, when having less than 50 schemas on the fresh instance causes the poor performance for problematic query when having large number of objects in database. In case of less than 50 schemas, sys.babelfish_namespace_ext catalog which stores the name of schema will have less than 50 rows which means autovacuum ANALYZE has never been run by PostgreSQL autovacuum daemon as the number of inserted rows doesn't exceeds the threshold criteria. Since the sys.babelfish_namespace_ext catalog has never been analyzed it leads bad query planning for large query. This commit add changes to automatically run ANALYZE after babelfish extension create/upgrade for all babelfish catalogs to avoid such issues in future. Issues Resolved Task: BABEL-4487 Signed-off-by: Sumit Jaiswal [email protected]

…1922) (#1958) * Automatically run ANALYZE after babelfish extension create/upgrade (#1922) Earlier, when having less than 50 schemas on the fresh instance causes the poor performance for problematic query when having large number of objects in database. In case of less than 50 schemas, sys.babelfish_namespace_ext catalog which stores the name of schema will have less than 50 rows which means autovacuum ANALYZE has never been run by PostgreSQL autovacuum daemon as the number of inserted rows doesn't exceeds the threshold criteria. Since the sys.babelfish_namespace_ext catalog has never been analyzed it leads bad query planning for large query. This commit add changes to automatically run ANALYZE after babelfish extension create/upgrade for all babelfish catalogs to avoid such issues in future. Issues Resolved Task: BABEL-4487 Signed-off-by: Sumit Jaiswal <[email protected]>

Run analyze after creating the extension

af58cf6

Signed-off-by: Sumit Jaiswal <[email protected]>

sumitj824 changed the title ~~Run analyze after creating the extension~~ Automatically run ANALYZE after babelfish extension create/upgrade Oct 16, 2023

Sumit Jaiswal added 5 commits October 16, 2023 08:07

run analyze for babelfish catalogs only

0f510b8

Signed-off-by: Sumit Jaiswal <[email protected]>

Update test file

f19f5f7

Signed-off-by: Sumit Jaiswal <[email protected]>

Retrigger Test

5d6f1fa

Run analyze for all babelfish catalogs during upgrade

f1260fd

Signed-off-by: Sumit Jaiswal <[email protected]>

Correct Typo

58061d3

Signed-off-by: Sumit Jaiswal <[email protected]>

sumitj824 mentioned this pull request Oct 19, 2023

Performance improvements for SSMS query to enumerate tables and views #1897

Merged

1 task

KushaalShroff reviewed Oct 20, 2023

View reviewed changes

contrib/babelfishpg_tsql/sql/ownership.sql Outdated Show resolved Hide resolved

Sumit Jaiswal added 4 commits October 20, 2023 10:40

make sure it runs for babelfish catalogs only

6e2f7ed

Signed-off-by: Sumit Jaiswal <[email protected]>

Merge branch 'BABEL_3_X_DEV' into sumiji-dev-analyze

e833c65

throw error in case of exception

d52e98f

Signed-off-by: Sumit Jaiswal <[email protected]>

updated upgrade scripts

10d5f82

Signed-off-by: Sumit Jaiswal <[email protected]>

sumitj824 mentioned this pull request Oct 23, 2023

Performance improvements for SSMS query to enumerate tables and views #1944

Merged

1 task

make changes to raise warning in case of failure

c9bdba3

Signed-off-by: Sumit Jaiswal <[email protected]>

Deepesh125 reviewed Oct 23, 2023

View reviewed changes

contrib/babelfishpg_tsql/sql/ownership.sql Show resolved Hide resolved

Deepesh125 previously approved these changes Oct 23, 2023

View reviewed changes

Deepesh125 requested a review from surendravishnoi October 23, 2023 14:00

forestkeeper reviewed Oct 23, 2023

View reviewed changes

test/python/expected/sql_validation_framework/expected_create.out Show resolved Hide resolved

forestkeeper previously approved these changes Oct 23, 2023

View reviewed changes

forestkeeper reviewed Oct 23, 2023

View reviewed changes

KushaalShroff previously approved these changes Oct 24, 2023

View reviewed changes

surendravishnoi reviewed Oct 24, 2023

View reviewed changes

Add upgrade test make sure analyze is working

b5fb8d0

Signed-off-by: Sumit Jaiswal <[email protected]>

sumitj824 dismissed stale reviews from KushaalShroff, forestkeeper, and Deepesh125 via b5fb8d0 October 24, 2023 10:48

Sumit Jaiswal added 3 commits October 24, 2023 11:55

comment test from latest as we don't run upgrade script for extension…

e14e679

… during source latest to target latest Signed-off-by: Sumit Jaiswal <[email protected]>

Retrigger test

b14a792

Signed-off-by: Sumit Jaiswal <[email protected]>

Retrigger test

77c9e13

Signed-off-by: Sumit Jaiswal <[email protected]>

surendravishnoi reviewed Oct 24, 2023

View reviewed changes

test/JDBC/upgrade/latest/schedule Show resolved Hide resolved

KushaalShroff reviewed Oct 25, 2023

View reviewed changes

test/JDBC/expected/AUTO_ANALYZE-vu-prepare.out Outdated Show resolved Hide resolved

Deepesh125 reviewed Oct 25, 2023

View reviewed changes

test/JDBC/input/AUTO_ANALYZE-before-15-5-or-14-10-vu-prepare.sql Outdated Show resolved Hide resolved

Deepesh125 previously approved these changes Oct 25, 2023

View reviewed changes

Remove functions from test file

03cd14a

Signed-off-by: Sumit Jaiswal <[email protected]>

sumitj824 dismissed Deepesh125’s stale review via 03cd14a October 25, 2023 10:49

only list in case of failure

fbd6b49

Signed-off-by: Sumit Jaiswal <[email protected]>

KushaalShroff approved these changes Oct 25, 2023

View reviewed changes

surendravishnoi approved these changes Oct 25, 2023

View reviewed changes

KushaalShroff merged commit 354feb0 into babelfish-for-postgresql:BABEL_3_X_DEV Oct 25, 2023
27 checks passed

KushaalShroff deleted the sumiji-dev-analyze branch October 25, 2023 16:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatically run ANALYZE after babelfish extension create/upgrade #1922

Automatically run ANALYZE after babelfish extension create/upgrade #1922

sumitj824 commented Oct 16, 2023 •

edited

Loading

forestkeeper Oct 23, 2023

sumitj824 Oct 24, 2023

surendravishnoi Oct 24, 2023

sumitj824 Oct 24, 2023 •

edited

Loading

surendravishnoi Oct 24, 2023

surendravishnoi Oct 24, 2023

Automatically run ANALYZE after babelfish extension create/upgrade #1922

Automatically run ANALYZE after babelfish extension create/upgrade #1922

Conversation

sumitj824 commented Oct 16, 2023 • edited Loading

Description

Issues Resolved

Test Scenarios Covered

Performance tests

USE CASE BASED

Check List

forestkeeper Oct 23, 2023

Choose a reason for hiding this comment

sumitj824 Oct 24, 2023

Choose a reason for hiding this comment

surendravishnoi Oct 24, 2023

Choose a reason for hiding this comment

sumitj824 Oct 24, 2023 • edited Loading

Choose a reason for hiding this comment

surendravishnoi Oct 24, 2023

Choose a reason for hiding this comment

surendravishnoi Oct 24, 2023

Choose a reason for hiding this comment

sumitj824 commented Oct 16, 2023 •

edited

Loading

sumitj824 Oct 24, 2023 •

edited

Loading