Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically run ANALYZE after babelfish extension create/upgrade #1922

Merged
merged 17 commits into from
Oct 25, 2023
Merged

Automatically run ANALYZE after babelfish extension create/upgrade #1922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
af58cf6
Run analyze after creating the extension
Oct 16, 2023
0f510b8
run analyze for babelfish catalogs only
Oct 16, 2023
f19f5f7
Update test file
Oct 16, 2023
5d6f1fa
Retrigger Test
Oct 16, 2023
f1260fd
Run analyze for all babelfish catalogs during upgrade
Oct 18, 2023
58061d3
Correct Typo
Oct 18, 2023
6e2f7ed
make sure it runs for babelfish catalogs only
Oct 20, 2023
e833c65
Merge branch 'BABEL_3_X_DEV' into sumiji-dev-analyze
Oct 20, 2023
d52e98f
throw error in case of exception
Oct 23, 2023
10d5f82
updated upgrade scripts
Oct 23, 2023
c9bdba3
make changes to raise warning in case of failure
Oct 23, 2023
b5fb8d0
Add upgrade test make sure analyze is working
Oct 24, 2023
e14e679
comment test from latest as we don't run upgrade script for extension…
Oct 24, 2023
b14a792
Retrigger test
Oct 24, 2023
77c9e13
Retrigger test
Oct 24, 2023
03cd14a
Remove functions from test file
Oct 25, 2023
fbd6b49
only list in case of failure
Oct 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions contrib/babelfishpg_tsql/sql/ownership.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,30 @@ BEGIN
END
$$;

CREATE OR REPLACE PROCEDURE sys.analyze_babelfish_catalogs()
sumitj824 marked this conversation as resolved.
Show resolved Hide resolved
LANGUAGE plpgsql
AS $$
DECLARE
babelfish_catalog RECORD;
schema_name varchar = 'sys';
error_msg text;
BEGIN
FOR babelfish_catalog IN (
SELECT relname as name from pg_class t
INNER JOIN pg_namespace n on n.oid = t.relnamespace
WHERE t.relkind = 'r' and n.nspname = schema_name
)
LOOP
BEGIN
EXECUTE format('ANALYZE %I.%I', schema_name, babelfish_catalog.name);
EXCEPTION WHEN OTHERS THEN
GET STACKED DIAGNOSTICS error_msg = MESSAGE_TEXT;
RAISE WARNING 'ANALYZE for babelfish catalog %.% failed with error: %s', schema_name, babelfish_catalog.name, error_msg;
END;
END LOOP;
END;
$$;

CREATE OR REPLACE PROCEDURE initialize_babelfish ( sa_name VARCHAR(128) )
LANGUAGE plpgsql
AS $$
Expand Down Expand Up @@ -266,6 +290,8 @@ BEGIN
CALL sys.babel_initialize_logins('sysadmin');
CALL sys.babel_create_builtin_dbs(sa_name);
CALL sys.initialize_babel_extras();
-- run analyze for all babelfish catalog
CALL sys.analyze_babelfish_catalogs();
END
$$;

Expand Down
28 changes: 28 additions & 0 deletions contrib/babelfishpg_tsql/sql/upgrades/babelfishpg_tsql--3.3.0--3.4.0.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2543,6 +2543,30 @@ GRANT EXECUTE ON FUNCTION sys.has_perms_by_name(
permission sys.SYSNAME,
sub_securable sys.SYSNAME,
sub_securable_class sys.nvarchar(60)) TO PUBLIC;

CREATE OR REPLACE PROCEDURE sys.analyze_babelfish_catalogs()
LANGUAGE plpgsql
AS $$
DECLARE
babelfish_catalog RECORD;
schema_name varchar = 'sys';
error_msg text;
BEGIN
FOR babelfish_catalog IN (
SELECT relname as name from pg_class t
INNER JOIN pg_namespace n on n.oid = t.relnamespace
WHERE t.relkind = 'r' and n.nspname = schema_name
)
LOOP
BEGIN
EXECUTE format('ANALYZE %I.%I', schema_name, babelfish_catalog.name);
EXCEPTION WHEN OTHERS THEN
GET STACKED DIAGNOSTICS error_msg = MESSAGE_TEXT;
RAISE WARNING 'ANALYZE for babelfish catalog %.% failed with error: %s', schema_name, babelfish_catalog.name, error_msg;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we have this warning in the upgrade, what's our action item ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning will help us to debug any performance issue after upgrade due the failure. I will add the same in Jira also.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how will it lead to action on customer or our side? A warning will go to upgrade scripts and might be lost if no one takes a look.

Copy link
Contributor Author

@sumitj824 sumitj824 Oct 24, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, warning will not lead to any action but it will be helpful for any performance issue observed just after the upgrade and logs are present.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let us track this item via a Jira. Talk to Sharu about this. We should consider how we track successful MVUs and corresponding logs in long term to resolve issues down the line.

END;
END LOOP;
END;
$$;

-- This is a temporary procedure which is called during upgrade to update guest schema
-- for the guest users in the already existing databases
Expand All @@ -2560,6 +2584,10 @@ DROP PROCEDURE sys.babelfish_update_user_catalog_for_guest_schema();
-- Please have this be one of the last statements executed in this upgrade script.
DROP PROCEDURE sys.babelfish_drop_deprecated_object(varchar, varchar, varchar);

-- After upgrade, always run analyze for all babelfish catalogs.
CALL sys.analyze_babelfish_catalogs();
sumitj824 marked this conversation as resolved.
Show resolved Hide resolved

-- Reset search_path to not affect any subsequent scripts
SELECT set_config('search_path', trim(leading 'sys, ' from current_setting('search_path')), false);


9 changes: 9 additions & 0 deletions test/JDBC/expected/AUTO_ANALYZE-before-15-5-or-14-10-vu-prepare.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

-- list all the babel catalogs that has been analyzed manually during extension create
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NOT NULL order by relname
Go
~~START~~
varchar
~~END~~

9 changes: 9 additions & 0 deletions test/JDBC/expected/AUTO_ANALYZE-before-15-5-or-14-10-vu-verify.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

-- list all the babel catalogs that has not been analyzed manually during extension upgrade
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NULL order by relname
GO
~~START~~
varchar
~~END~~

34 changes: 34 additions & 0 deletions test/JDBC/expected/AUTO_ANALYZE-vu-prepare.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@

-- list all the babel catalogs that has not been analyzed manually during extension create
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NULL order by relname
Go
~~START~~
varchar
~~END~~


-- list all the babel catalogs that has been analyzed manually during extension create
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NOT NULL order by relname
Go
~~START~~
varchar
assemblies
babelfish_authid_login_ext
babelfish_authid_user_ext
babelfish_configurations
babelfish_domain_mapping
babelfish_extended_properties
babelfish_function_ext
babelfish_helpcollation
babelfish_namespace_ext
babelfish_schema_permissions
babelfish_server_options
babelfish_sysdatabases
babelfish_syslanguages
babelfish_view_def
service_settings
spt_datatype_info_table
versions
~~END~~

9 changes: 9 additions & 0 deletions test/JDBC/expected/AUTO_ANALYZE-vu-verify.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

-- list all the babel catalogs that has not been analyzed manually during extension upgrade
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NULL order by relname
GO
~~START~~
varchar
~~END~~

4 changes: 2 additions & 2 deletions test/JDBC/expected/openquery_upgrd_before_15_4-vu-prepare.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,13 +177,13 @@ EXEC sp_serveroption @server='bbf_server_2', @optname='connect timeout', @optval
GO

-- check if the linked servers added above are reflected in babelfish_server_options catalog
SELECT servername, query_timeout, connect_timeout FROM babelfish_server_options WHERE servername = 'bbf_server' OR servername = 'bbf_server_1' OR servername = 'bbf_server_2'
SELECT servername, query_timeout, connect_timeout FROM babelfish_server_options WHERE servername = 'bbf_server' OR servername = 'bbf_server_1' OR servername = 'bbf_server_2' order by servername
GO
~~START~~
varchar#!#int#!#int
bbf_server#!#0#!#0
bbf_server_2#!#0#!#1
bbf_server_1#!#1#!#5
bbf_server_2#!#0#!#1
~~END~~


Expand Down
5 changes: 5 additions & 0 deletions test/JDBC/input/AUTO_ANALYZE-before-15-5-or-14-10-vu-prepare.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

-- list all the babel catalogs that has been analyzed manually during extension create
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NOT NULL order by relname
Go
5 changes: 5 additions & 0 deletions test/JDBC/input/AUTO_ANALYZE-before-15-5-or-14-10-vu-verify.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

-- list all the babel catalogs that has not been analyzed manually during extension upgrade
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NULL order by relname
GO
9 changes: 9 additions & 0 deletions test/JDBC/input/AUTO_ANALYZE-vu-prepare.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

-- list all the babel catalogs that has not been analyzed manually during extension create
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NULL order by relname
Go

-- list all the babel catalogs that has been analyzed manually during extension create
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NOT NULL order by relname
Go
5 changes: 5 additions & 0 deletions test/JDBC/input/AUTO_ANALYZE-vu-verify.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

-- list all the babel catalogs that has not been analyzed manually during extension upgrade
-- will return NULL in this case
SELECT relname FROM pg_stat_all_tables WHERE schemaname = 'sys' and last_analyze IS NULL order by relname
GO
2 changes: 1 addition & 1 deletion test/JDBC/input/openquery_upgrd_before_15_4-vu-prepare.mix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ EXEC sp_serveroption @server='bbf_server_2', @optname='connect timeout', @optval
GO

-- check if the linked servers added above are reflected in babelfish_server_options catalog
SELECT servername, query_timeout, connect_timeout FROM babelfish_server_options WHERE servername = 'bbf_server' OR servername = 'bbf_server_1' OR servername = 'bbf_server_2'
SELECT servername, query_timeout, connect_timeout FROM babelfish_server_options WHERE servername = 'bbf_server' OR servername = 'bbf_server_1' OR servername = 'bbf_server_2' order by servername
GO

-- Create a view dependent on OPENQUERY
Expand Down
2 changes: 2 additions & 0 deletions test/JDBC/jdbc_schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ ignore#!#forjson-subquery-before-14_10-or-15_5-vu-cleanup
ignore#!#forjson-datatypes-before-14_10-or-15_5-vu-prepare
ignore#!#forjson-datatypes-before-14_10-or-15_5-vu-verify
ignore#!#forjson-datatypes-before-14_10-or-15_5-vu-cleanup
ignore#!#AUTO_ANALYZE-before-15-5-or-14-10-vu-prepare
ignore#!#AUTO_ANALYZE-before-15-5-or-14-10-vu-verify
ignore#!#orderby-before-15_3-vu-prepare
ignore#!#orderby-before-15_3-vu-verify
ignore#!#orderby-before-15_3-vu-cleanup
Expand Down
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/13_4/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,4 +215,5 @@ TestXML
timefromparts
triggers_with_transaction
BABEL-4046
getdate
getdate
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/13_5/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,4 +268,5 @@ TestXML
timefromparts
triggers_with_transaction
BABEL-4046
getdate
getdate
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/13_6/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,3 +325,4 @@ BABEL-4046
getdate
BABEL-4410
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/13_7/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,4 +316,5 @@ TestXML
timefromparts
triggers_with_transaction
BABEL-4046
getdate
getdate
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/13_8/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,4 +316,5 @@ TestXML
timefromparts
triggers_with_transaction
BABEL-4046
getdate
getdate
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/13_9/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,3 +321,4 @@ BABEL-4046
getdate
BABEL-4410
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/14_3/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,4 +338,5 @@ BABEL-4046
BABEL_4330
BABEL-2619
BABEL-4410
GRANT_SCHEMA
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/14_5/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -354,3 +354,4 @@ BABEL_4330
BABEL-2619
BABEL-4410
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/14_6/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -388,3 +388,4 @@ BABEL_4330
BABEL-2619
BABEL-4410
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/14_7/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -407,4 +407,5 @@ datetimeoffset-timezone-before-15_3
BABEL-4046
getdate
BABEL_4330
BABEL-2619
BABEL-2619
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/14_8/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,4 +405,5 @@ datetimeoffset-timezone-before-15_3
BABEL-4046
getdate
BABEL_4330
BABEL-2619
BABEL-2619
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/14_9/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,3 +408,4 @@ getdate
BABEL_4330
BABEL-2619
BABEL-4410
AUTO_ANALYZE-before-15-5-or-14-10
3 changes: 2 additions & 1 deletion test/JDBC/upgrade/15_1/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,4 +384,5 @@ timefromparts
triggers_with_transaction
BABEL-4046
getdate
BABEL_4330
BABEL_4330
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/15_2/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,3 +416,4 @@ getdate
BABEL_4330
BABEL-4410
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/15_3/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,3 +437,4 @@ datetimeoffset-timezone
BABEL-4046
getdate
BABEL_4330
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/15_4/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,3 +450,4 @@ sp_who
BABEL_4330
BABEL-4410
GRANT_SCHEMA
AUTO_ANALYZE-before-15-5-or-14-10
1 change: 1 addition & 0 deletions test/JDBC/upgrade/latest/schedule
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -474,3 +474,4 @@ sys_certificates
sys_database_permissions
BABEL-4279
BABEL-4484
#AUTO_ANALYZE #uncomment this test when preparing for new minor version
sumitj824 marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions test/python/expected/sql_validation_framework/expected_create.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ Could not find tests for procedure babel_datatype_precedence_initializer
Could not find tests for procedure initialize_babel_extras
Could not find tests for procedure initialize_babelfish
Could not find tests for procedure remove_babelfish
Could not find tests for procedure sys.analyze_babelfish_catalogs
Could not find tests for procedure sys.babel_drop_all_dbs
Could not find tests for procedure sys.babel_drop_all_logins
Could not find tests for procedure sys.babel_initialize_logins
Expand Down Expand Up @@ -186,6 +187,7 @@ Could not find upgrade tests for procedure babel_datatype_precedence_initializer
Could not find upgrade tests for procedure initialize_babel_extras
Could not find upgrade tests for procedure initialize_babelfish
Could not find upgrade tests for procedure remove_babelfish
Could not find upgrade tests for procedure sys.analyze_babelfish_catalogs
sumitj824 marked this conversation as resolved.
Show resolved Hide resolved
Could not find upgrade tests for procedure sys.babel_drop_all_dbs
Could not find upgrade tests for procedure sys.babel_drop_all_logins
Could not find upgrade tests for procedure sys.babel_initialize_logins
Expand Down
Loading