PT-2340: support MySQL 8.4 #871

svetasmirnova · 2024-10-04T16:35:46Z

This huge PR adds MySQL 8.4 support to the Percona Toolkit.

The contributed code is licensed under GPL v2.0
Contributor Licence Agreement (CLA) is signed
util/update-modules has been ran
Documentation updated
Test suite update

- Adjusted sandbox scripts, so they can start MySQL 8.4 - Added MySQL 8.4 configuration file - Removed empty file sandbox/5.6 - Removed unused files sandbox/set-mysql, sandbox/slave_channels_t.sql, sandbox/jenkins-test - Removed offensive terminology from the sandbox scripts wherever it is possible

- Fixed status check for 8.4 sandbox - Removed option "reset" from the usage output, because option itself was removed by 92a3671 - Fixed sandbox/start-sandbox script, so it can start custom setups with 8.4

- Removed offensive terminology from library files and their tests - Removed unused sandbox/prove2junit.pl - Added option mysql_ssl to DSN and possibility to have DSN of multiple letters

- Removed all unneeded occuriences of the word "master" in lib

- Removed lib/Percona/Test.pm, lib/Safeguards.pm, t/lib/Safeguards.t, because they are not used anymore - Removed word "slave" from lib

- Removed runtime.txt after discussion with Anastasia Alexandrova - Added "use VersionParser" into tests in t/lib when needed - Removed word master from tests for pt-archiver, pt-config-diff, pt-deadlock-logger, pt-duplicate-key-checker, pt-find, pt-fk-error-logger, pt-heartbeat, pt-index-usage, pt-ioprofile, pt-kill, pt-mysql-summary - Removed word slave from tests for pt-archiver, pt-config-diff, pt-deadlock-logger, pt-duplicate-key-checker, pt-find, pt-fk-error-logger, pt-heartbeat, pt-index-usage, pt-ioprofile, pt-kill, pt-mysql-summary - Updated modules for pt-archiver, pt-config-diff, pt-deadlock-logger, pt-duplicate-key-checker, pt-find, pt-fk-error-logger, pt-heartbeat, pt-index-usage, pt-ioprofile, pt-kill, pt-mysql-summary - Changed mysql_ssl patch, so it is now short option s - Added a check for existing zombies in t/pt-kill/execute_command.t - Added bin/pt-galera-log-explainer to .gitignore

- Updated modules and tests for pt-online-schema-change - Removed typo from lib/MasterSlave.pm

- Updated modules and tests for pt-query-digest, pt-show-grants, pt-slave-delay, pt-slave-find, pt-slave-restart, pt-stalk, pt-summary

- Updated modules and tests for pt-query-digest, pt-show-grants, pt-slave-delay, pt-slave-find, pt-slave-restart, pt-stalk, pt-summary, pt-table-checksum

…a-toolkit into PT-2340_Support_MySQL_8.4

- Updated modules and tests for pt-table-sync, pt-table-usage, pt-upgrade, pt-variable-advisor

- Fixed staff I broke for 8.0

- Fixed 5.7 tests

- Added deprecation warning to pt-slave-delay - Re-enabled tests for pt-slave-delay to check the warning - Disabled pt-slave-delay for MySQL 8.1+

- Created pt-replica-restart. pt-slave-restart kept as symbolic link

- Changed write-user-docs script, so it creates special documentation pages for symbolic links.

- Fixed comparison operators for VersionCheck calls

- Created pt-replica-find. pt-slave-find kept as symbolic link

- Updated documentation

- Fixed typos - Added tests for MyRocks and TokuDB clustered indexes into t/pt-duplicate-key-checker/clustered_keys.t - Updated modules

- Updated tests for pt-variable-advisor and lib/VariableAdvisorRules

- Moved data collection for THP from lib/bash/report_system_info.sh to lib/bash/collect_system_info.sh, so pt-summary reports THP status on the machine where samples were collected, not on the machine where an engineer examines samples.

- Adjusted test results, broken due to mysql.component table in PS

- Tests for the minimal SSL support - Updated util/update-modules, so they don't skip tools with modules, not defined in lib

- Added tests for legacy style heartbeat and checksum tables

- Added test for replica lag check for pt-archiver - Re-added deprecated slave- options - Added tests for deprecation warnings and for legacy options for pt-archiver - Removed practically not supported options --replica-user and --replica-password from pt-archiver, pt-kill, pt-query-digest - Added legacy source/replica options (master/slave) variants and tests for pt-heartbeat, pt-online-schema-change, pt-replica-find, pt-replica-restart, pt-table-checksum, pt-table-sync - Updated modules after lib/MasterSlave.pm change

- Adjusted t/pt-deadlock-logger/check_schema_exists.t so it makes sense

- Adjusted pt-variable-advisor, so it reflects current defaults

- Updated modules

- Fixed typo in bin/pt-table-sync

- Fixed typos

t/lib/bash/collect_mysql_info.sh

t/pt-archiver/archive_using_channels.t

ylacancellera · 2024-10-07T09:49:29Z

bin/pt-archiver

@@ -7179,14 +7292,14 @@ sub main {
            my $lag_dbh = $lag_server->{'dbh'};
            my $id      = $lag_server->{'id'};
            if ( $lag_dbh ) {
-               my $lag = $ms->get_slave_lag($lag_dbh);
+               my $lag = $ms->get_replica_lag($lag_dbh);
               while ( !defined $lag || $lag > $o->get('max-lag') ) {
                  PTDEBUG && _d("Sleeping: slave lag for server '$id' is", $lag);


Since pt-heartbeat had his schema changed to reflect the source/replica change, does this one should be changed for coherence ?
Or, is it left as is so to keep the current output ?

Hi! I don't understand your comment. Modified code uses get_replica_lag. What needs to be changed?

It is for _d("Sleeping: slave lag for server '$id' is", $lag);, on row 7297 and 7299 (on new branch).
7297 is with PTDEBUG, so really not a problem, but the other one is a log to users
"replica lag" would make it more coherent, but that's only a nitpick

Fixed, please review again

lgtm for the bin, the tests will need some fixing as well

$ grep -rn 'Sleeping:' t/pt-archiver/replica_lag.t:73: qr/Sleeping: slave lag for server/, t/pt-archiver/replica_lag.t:101: qr/Sleeping: slave lag for server/, t/pt-archiver/replica_lag.t:129: qr/Sleeping: slave lag for server/, t/pt-archiver/replica_lag.t:156: qr/Sleeping: slave lag for server/, t/pt-archiver/replica_lag.t:184: qr/Sleeping: slave lag for server/, t/pt-archiver/replica_lag.t:212: qr/Sleeping: slave lag for server/, t/pt-archiver/replica_lag.t:240: qr/Sleeping: slave lag for server/, bin/pt-archiver:7297: PTDEBUG && _d("Sleeping: replica lag for server '$id' is", $lag); bin/pt-archiver:7299: _d("Sleeping: replica lag for server '$id' is", $lag);

t/pt-heartbeat/multi_update_mode.t

- s/slave lag/replica lag/ in reports

ylacancellera · 2024-10-10T09:37:28Z

lib/MySQLProtocolParser.pm

@@ -75,7 +75,7 @@ use constant {
   COM_BINLOG_DUMP         => '12',
   COM_TABLE_DUMP          => '13',
   COM_CONNECT_OUT         => '14',
-   COM_REGISTER_SLAVE      => '15',
+   COM_REGISTER_REPLICA    => '15',


The packet exists on 8.0 and 8.4, but not 5.7
I don't find any code usage of it in any lib/bin though

ylacancellera · 2024-10-10T09:48:04Z

lib/PerconaTest.pm

+our $source_reset = 'binary logs and gtids';
+our $source_change = 'replication source';
+our $replica_name = 'replica';
+if ( $sandbox_version lt '8.1' || ( $ENV{FORK} || "" eq 'mariadb' ) ) {


works in an unintended way (PT-2390). I listed this code path in PT-2390 for review. I'm still commenting since it's a different condition
10.x will return true on $sandbox_version lt '8.1', so the second part on the fork is not used.

$ echo $sandbox_version $FORK 10.4 mysql $ perl -e "if ( $sandbox_version lt '8.1' || ( $ENV{FORK} || '' eq 'mariadb' ) ) { print('works') }" works

As long as mysql 10 does not exist it still does what we want

ylacancellera · 2024-10-11T09:46:15Z

bin/pt-replica-find

+# Check at the end of this package for the call to main() which actually runs
+# the program.
+# ###########################################################################
+package pt_slave_find;


can this impact maintenance ? The names are mismatched between the toolname and the package name
If it's safe, good to me. I don't find any direct usage

ylacancellera · 2024-10-11T09:55:24Z

bin/pt-replica-find

+#    +- 1
+#       +- 2
+#    +- 1
+sub print_slaves {


Since many internal variables were renamed in lib, I'm commenting to be thorough
This tool package did not receive the s/slave/replica
However I don't see any misusage

Unless this was a goal to remove all of them in this PR specifically, I'm ok to keep it and maybe flag it for later

ylacancellera · 2024-10-11T09:57:26Z

bin/pt-replica-find

+   }
+   push @lines, [
+      'Replication',
+      ($ss ? "Is a slave, " : "Is not a slave, ")


Unaligned with rest, I can flag it for later if needed

ylacancellera · 2024-10-11T09:58:29Z

bin/pt-replica-find

+     Version         5.1.34-log
+     Server ID       12346
+     Uptime          04:54 (started 2010-06-17T11:21:24)
+     Replication     Is a slave, has 1 slaves connected


Unaligned with the rest, but coherent with the current states of things
Again, I can flag it for later, or skip

ylacancellera

I have commented what I found already, it's only nitpicks on some missed "slave" wordings. (except the version comparisons, but PT-2390 have been created so I exclude it).

Looks good to me

svetasmirnova added 30 commits July 23, 2024 16:07

PT-2340 - Support MySQL 8.4

a530c6f

- Fixed status check for 8.4 sandbox - Removed option "reset" from the usage output, because option itself was removed by 92a3671 - Fixed sandbox/start-sandbox script, so it can start custom setups with 8.4

PT-2340 - Support MySQL 8.4

e2207ea

- Removed offensive terminology from library files and their tests - Removed unused sandbox/prove2junit.pl - Added option mysql_ssl to DSN and possibility to have DSN of multiple letters

PT-2340 - Support MySQL 8.4

b9a5231

- Removed all unneeded occuriences of the word "master" in lib

PT-2340 - Support MySQL 8.4

8cbb5a0

- Removed lib/Percona/Test.pm, lib/Safeguards.pm, t/lib/Safeguards.t, because they are not used anymore - Removed word "slave" from lib

PT-2340 - Support MySQL 8.4

76c1202

- Updated modules and tests for pt-online-schema-change - Removed typo from lib/MasterSlave.pm

PT-2340 - Support MySQL 8.4

067391b

- Updated modules and tests for pt-query-digest, pt-show-grants, pt-slave-delay, pt-slave-find, pt-slave-restart, pt-stalk, pt-summary

PT-2340 - Support MySQL 8.4

d02355f

- Updated modules and tests for pt-query-digest, pt-show-grants, pt-slave-delay, pt-slave-find, pt-slave-restart, pt-stalk, pt-summary, pt-table-checksum

Merge branch 'PT-2340_Support_MySQL_8.4' of github.com:percona/percon…

56599f0

…a-toolkit into PT-2340_Support_MySQL_8.4

PT-2340 - Support MySQL 8.4

a38bee6

- Updated modules and tests for pt-table-sync, pt-table-usage, pt-upgrade, pt-variable-advisor

PT-2340 - Support MySQL 8.4

b332537

- Fixed staff I broke for 8.0

PT-2340 - Support MySQL 8.4

194bb99

- Fixed 5.7 tests

PT-2340 - Support MySQL 8.4

1006101

- Added deprecation warning to pt-slave-delay - Re-enabled tests for pt-slave-delay to check the warning - Disabled pt-slave-delay for MySQL 8.1+

PT-2340 - Support MySQL 8.4

8f87bad

- Created pt-replica-restart. pt-slave-restart kept as symbolic link

PT-2340 - Support MySQL 8.4

42486a8

- Changed write-user-docs script, so it creates special documentation pages for symbolic links.

PT-2340 - Support MySQL 8.4

e3e352d

- Fixed comparison operators for VersionCheck calls

PT-2340 - Support MySQL 8.4

2ca8156

- Created pt-replica-find. pt-slave-find kept as symbolic link

PT-2340 - Support MySQL 8.4

94bcfd2

- Updated documentation

PT-2340 - Support MySQL 8.4

d03bd60

- Fixed typos - Added tests for MyRocks and TokuDB clustered indexes into t/pt-duplicate-key-checker/clustered_keys.t - Updated modules

Merge branch '3.x' into PT-2340_Support_MySQL_8.4

d737288

PT-2340 - Support MySQL 8.4

c3f327d

- Updated tests for pt-variable-advisor and lib/VariableAdvisorRules

PT-2340 - Support MySQL 8.4

ea6bd77

- Moved data collection for THP from lib/bash/report_system_info.sh to lib/bash/collect_system_info.sh, so pt-summary reports THP status on the machine where samples were collected, not on the machine where an engineer examines samples.

PT-2340 - Support MySQL 8.4

045b2fe

- Adjusted test results, broken due to mysql.component table in PS

PT-2340 - Support MySQL 8.4

5d55904

- Tests for the minimal SSL support - Updated util/update-modules, so they don't skip tools with modules, not defined in lib

Merge branch '3.x' into PT-2340_Support_MySQL_8.4

e481396

PT-2340 - Support MySQL 8.4

1de1491

- Added tests for legacy style heartbeat and checksum tables

PT-2340 - Support MySQL 8.4

e4281dc

- Adjusted t/pt-deadlock-logger/check_schema_exists.t so it makes sense

PT-2340 - Support MySQL 8.4

fcb2d54

- Adjusted pt-variable-advisor, so it reflects current defaults

svetasmirnova added 3 commits October 4, 2024 19:06

PT-2340 - Support MySQL 8.4

86fe021

- Updated modules

PT-2340 - Support MySQL 8.4

287a1c0

- Fixed typo in bin/pt-table-sync

Merge branch '3.x' into PT-2340_Support_MySQL_8.4

ec677f4

it-percona deployed to PT-2340_Support_MySQL_8.4 - percona-toolkit PR #871 October 4, 2024 16:35 — with Render View deployment

svetasmirnova requested review from ihanick, EvgeniyPatlan, ylacancellera and saikumar-vs October 4, 2024 16:36

PT-2340 - Support MySQL 8.4

aab0b0c

- Fixed typos

it-percona deployed to PT-2340_Support_MySQL_8.4 - percona-toolkit PR #871 October 6, 2024 19:56 — with Render View deployment

ylacancellera reviewed Oct 7, 2024

View reviewed changes

t/lib/bash/collect_mysql_info.sh Show resolved Hide resolved

ylacancellera reviewed Oct 7, 2024

View reviewed changes

t/pt-archiver/archive_using_channels.t Show resolved Hide resolved

ylacancellera reviewed Oct 7, 2024

View reviewed changes

t/pt-heartbeat/multi_update_mode.t Show resolved Hide resolved

svetasmirnova requested a review from ylacancellera October 7, 2024 12:11

PT-2340 - Support MySQL 8.4

a2102ef

- s/slave lag/replica lag/ in reports

it-percona deployed to PT-2340_Support_MySQL_8.4 - percona-toolkit PR #871 October 7, 2024 12:53 — with Render View deployment

ylacancellera reviewed Oct 10, 2024

View reviewed changes

ylacancellera reviewed Oct 11, 2024

View reviewed changes

ylacancellera approved these changes Oct 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PT-2340: support MySQL 8.4 #871

PT-2340: support MySQL 8.4 #871

svetasmirnova commented Oct 4, 2024 •

edited

Loading

ylacancellera Oct 7, 2024

svetasmirnova Oct 7, 2024

ylacancellera Oct 7, 2024

svetasmirnova Oct 7, 2024

ylacancellera Oct 10, 2024

ylacancellera Oct 10, 2024

ylacancellera Oct 10, 2024 •

edited

Loading

ylacancellera Oct 11, 2024 •

edited

Loading

ylacancellera Oct 11, 2024

ylacancellera Oct 11, 2024

ylacancellera Oct 11, 2024

ylacancellera left a comment

PT-2340: support MySQL 8.4 #871

Are you sure you want to change the base?

PT-2340: support MySQL 8.4 #871

Conversation

svetasmirnova commented Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylacancellera Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

ylacancellera Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylacancellera left a comment

Choose a reason for hiding this comment

svetasmirnova commented Oct 4, 2024 •

edited

Loading

ylacancellera Oct 10, 2024 •

edited

Loading

ylacancellera Oct 11, 2024 •

edited

Loading