Change order of deploy-fast command

[maikschneider]

Hey,
I wanted to ask if it would be possible to change one order of the commands called by deploy-fast.

I would like to see that the commands typo3cms:database:updateschema and db:truncate are switched. This way, the truncate command does not fail with an empty database and deploy-fast can be executed on newly provisioned hosts that do not have a database structure yet.

It's a very common task for us to create new hosts from scratch that shall run TYPO3 installations. Unfortunately, this is not as easy as it could be. We have to run the deployment commands multiple times, wait for the next error, fix them by ssh-ing to the host, re-run and again.. That's why I created the command deploy:prepare:typo3 that does all the stuff, we would need to do manually. An issue, I always run into and cannot fix without checking out the concrete project repository in advance, is the missing database schema. It is not possible to setup a fresh host and run a (fast) deployment without triggering some sort of database access that throws an exception, because of missing tables.

I know, my command is highly customized to our workflow and needs improvements (fixed key paths etc.). I just wanted to avoid to override the deploy-fast task right away and talk to you first. What do you think about my plan? Maybe I'm missing something..

Best,

Maik

[kszymukowicz]

hey,

well - there are few things to consider:

1. regular deploy and deploy-fast should not differ in order of task execution. Otherwise we would need to explain in documentation why its differnt and that explanation would be cumbersome.
2. order of db:truncate before typo3cms:database:updateschema was there mainly to clear caching tables BEFORE first call of new TYPO3 code. If you switch order of them some things may broke.

I think the best solution could be to modify db:truncate command so it does not break when the table does not exists.
What do you think?

[maikschneider]

Hey,

thanks for your response.

1. Good point, didn't think about consistency of these two commands.
2. As far as I know, typo3cms database:updateschema only reads the ext_tables.sql files and does the database update stuff. I think there are very litte options to intercept this with any hook, certainly no code that uses cache tables. But I'm not 100% sure and like your suggestion of modifying db:truncate even better.

I think this could even solve an issue where you cannot import / push a database to a fresh host that does not have the structure yet.

I just looked into the command and think --force could be the way to go. In this thread they discussed how to ignore errors. Since it's just a truncate command for unimportant tables, there should no be much space for dangerous errors. An additional check would add unnecessary complexity to 99% of the use cases.

Have you already thought about a solution?

[kszymukowicz]

Hmm. I checked the code now and I am not sure why it fails for you actually.
If database has no tables then $truncateTables should be also empty?
https://github.com/sourcebroker/deployer-extended-database/blob/0d9ac121a087233ee2c08832ff41b2acf29a34d5/deployer/db/task/db_truncate.php#L18

[maikschneider]

Ah, right. I remember. It's actually not the mysql TRUNCATE command, that fails. It's the typo3cms configuration:showactive that is called before, when using the new driver_typo3cms. Very annoying behavior of the typo3cms command.

Maybe surrounding it with try catch?

EDIT: This wouldn't help. Then the database connection isn't available and another exception will occur.. Hm..

Here is the exception trace:

➤ Executing task db:truncate
[testing] > if [ -L /home/www/p604424/html/typo3-test/release ]; then echo 'true'; fi
[testing] < true
[testing] > if [ -L /home/www/p604424/html/typo3-test/release ]; then echo 'true'; fi
[testing] < true
[testing] > if [ -e '/home/www/p604424/html/typo3-test/release/vendor/bin/dep' ]; then echo 'true'; fi
[testing] < true
[testing] > cd /home/www/p604424/html/typo3-test/release && /usr/local/bin/php /home/www/p604424/html/typo3-test/release/vendor/bin/dep db:truncate  -vvv
[testing] < ➤ Executing task db:truncate
[testing] <
[testing] <   [ Doctrine\DBAL\Exception\TableNotFoundException ]
[testing] <   An exception occurred while executing 'SELECT * FROM `be_users` WHERE (`use
[testing] <   rname` = ?) AND ((`be_users`.`deleted` = 0) AND (`be_users`.`disable` = 0)
[testing] <   AND (`be_users`.`starttime` <= 1638650820) AND ((`be_users`.`endtime` = 0)
[testing] <   OR (`be_users`.`endtime` > 1638650820)) AND (`be_users`.`pid` = 0))' with p
[testing] <   arams ["_cli_"]:
[testing] <
[testing] <   Table 'usr_p604424_1.be_users' doesn't exist
[testing] < Exception trace:
[testing] < #0 ()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php:61
[testing] < #1 Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php:182
[testing] < #2 Doctrine\DBAL\DBALException::wrapException()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php:159
[testing] < #3 Doctrine\DBAL\DBALException::driverExceptionDuringQuery()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:2225
[testing] < #4 Doctrine\DBAL\Connection->handleExceptionDuringQuery()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1312
[testing] < #5 Doctrine\DBAL\Connection->executeQuery()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:212
[testing] < #6 Doctrine\DBAL\Query\QueryBuilder->execute()
[testing] <    typo3/sysext/core/Classes/Database/Query/QueryBuilder.php:220
[testing] < #7 TYPO3\CMS\Core\Database\Query\QueryBuilder->execute()
[testing] <    typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:1333
[testing] < #8 TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->getRawUserByName()
[testing] <    typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:1296
[testing] < #9 TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->setBeUserByName()
[testing] <    typo3/sysext/core/Classes/Authentication/CommandLineUserAuthentication.php:95
[testing] < #10 TYPO3\CMS\Core\Authentication\CommandLineUserAuthentication->authenticate()
[testing] <    typo3/sysext/core/Classes/Authentication/CommandLineUserAuthentication.php:120
[testing] < #11 TYPO3\CMS\Core\Authentication\CommandLineUserAuthentication->backendCheckLogin()
[testing] <    typo3/sysext/core/Classes/Core/Bootstrap.php:590
[testing] < #12 TYPO3\CMS\Core\Core\Bootstrap::initializeBackendAuthentication()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Scripts.php:81
[testing] < #13 Helhum\Typo3Console\Core\Booting\Scripts::initializeAuthenticatedOperations()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Step.php:49
[testing] < #14 Helhum\Typo3Console\Core\Booting\Step->__invoke()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Sequence.php:123
[testing] < #15 Helhum\Typo3Console\Core\Booting\Sequence->invokeStep()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Sequence.php:129
[testing] < #16 Helhum\Typo3Console\Core\Booting\Sequence->invokeStep()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Sequence.php:106
[testing] < #17 Helhum\Typo3Console\Core\Booting\Sequence->invoke()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/RunLevel.php:99
[testing] < #18 Helhum\Typo3Console\Core\Booting\RunLevel->runSequenceForCommand()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Kernel.php:103
[testing] < #19 Helhum\Typo3Console\Core\Kernel->handle()
[testing] <    vendor/helhum/typo3-console/Scripts/typo3-console.php:31
[testing] < #20 {closure}()
[testing] <    vendor/helhum/typo3-console/Scripts/typo3-console.php:33
[testing] < #21 require()
[testing] <    vendor/helhum/typo3-console/typo3cms:3
[testing] < Caused by:
[testing] <
[testing] <   [ Doctrine\DBAL\Driver\Mysqli\Exception\ConnectionError ]
[testing] <   Table 'usr_p604424_1.be_users' doesn't exist
[testing] <
[testing] < Exception trace:
[testing] < #0 ()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/Mysqli/Exception/ConnectionError.php:21
[testing] < #1 Doctrine\DBAL\Driver\Mysqli\Exception\ConnectionError::new()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/Mysqli/MysqliStatement.php:105
[testing] < #2 Doctrine\DBAL\Driver\Mysqli\MysqliStatement->__construct()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/Mysqli/MysqliConnection.php:152
[testing] < #3 Doctrine\DBAL\Driver\Mysqli\MysqliConnection->prepare()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php:1300
[testing] < #4 Doctrine\DBAL\Connection->executeQuery()
[testing] <    vendor/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php:212
[testing] < #5 Doctrine\DBAL\Query\QueryBuilder->execute()
[testing] <    typo3/sysext/core/Classes/Database/Query/QueryBuilder.php:220
[testing] < #6 TYPO3\CMS\Core\Database\Query\QueryBuilder->execute()
[testing] <    typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:1333
[testing] < #7 TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->getRawUserByName()
[testing] <    typo3/sysext/core/Classes/Authentication/AbstractUserAuthentication.php:1296
[testing] < #8 TYPO3\CMS\Core\Authentication\AbstractUserAuthentication->setBeUserByName()
[testing] <    typo3/sysext/core/Classes/Authentication/CommandLineUserAuthentication.php:95
[testing] < #9 TYPO3\CMS\Core\Authentication\CommandLineUserAuthentication->authenticate()
[testing] <    typo3/sysext/core/Classes/Authentication/CommandLineUserAuthentication.php:120
[testing] < #10 TYPO3\CMS\Core\Authentication\CommandLineUserAuthentication->backendCheckLogin()
[testing] <    typo3/sysext/core/Classes/Core/Bootstrap.php:590
[testing] < #11 TYPO3\CMS\Core\Core\Bootstrap::initializeBackendAuthentication()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Scripts.php:81
[testing] < #12 Helhum\Typo3Console\Core\Booting\Scripts::initializeAuthenticatedOperations()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Step.php:49
[testing] < #13 Helhum\Typo3Console\Core\Booting\Step->__invoke()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Sequence.php:123
[testing] < #14 Helhum\Typo3Console\Core\Booting\Sequence->invokeStep()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Sequence.php:129
[testing] < #15 Helhum\Typo3Console\Core\Booting\Sequence->invokeStep()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/Sequence.php:106
[testing] < #16 Helhum\Typo3Console\Core\Booting\Sequence->invoke()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Booting/RunLevel.php:99
[testing] < #17 Helhum\Typo3Console\Core\Booting\RunLevel->runSequenceForCommand()
[testing] <    vendor/helhum/typo3-console/Classes/Console/Core/Kernel.php:103
[testing] < #18 Helhum\Typo3Console\Core\Kernel->handle()
[testing] <    vendor/helhum/typo3-console/Scripts/typo3-console.php:31
[testing] < #19 {closure}()
[testing] <    vendor/helhum/typo3-console/Scripts/typo3-console.php:33
[testing] < #20 require()
[testing] <    vendor/helhum/typo3-console/typo3cms:3
[testing] < configuration:showactive [--json] [--] <path>
[testing] < In Typo3CmsDriver.php line 24:
[testing] <
[testing] <   [Deployer\Exception\ConfigurationException]
[testing] <   typo3cms configuration:showactive returned error code: "1"
[testing] <
[testing] < Exception trace:
[testing] <   at /html/typo3-test/releases/2/vendor/sourcebroker/deployer-extended-typo3/src/Drivers/Typo3CmsDriver.php:24
[testing] <  SourceBroker\DeployerExtendedTypo3\Drivers\Typo3CmsDriver->getDatabaseConfig() at /html/typo3-test/releases/2/vendor/sourcebroker/deployer-extended-typo3/deployer/11/set.php:66
[testing] <  SourceBroker\DeployerLoader\Utility\FileUtility->Deployer\{closure}() at n/a:n/a
[testing] <  call_user_func() at /html/typo3-test/releases/2/vendor/sourcebroker/deployer-extended-database/deployer/db/config/set.php:48
[testing] <  SourceBroker\DeployerLoader\Utility\FileUtility->Deployer\{closure}() at n/a:n/a
[testing] <  call_user_func() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/Configuration/Configuration.php:68
[testing] <  Deployer\Configuration\Configuration->get() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/functions.php:549
[testing] <  Deployer\get() at /html/typo3-test/releases/2/vendor/sourcebroker/deployer-extended-database/deployer/db/task/db_truncate.php:16
[testing] <  SourceBroker\DeployerLoader\Utility\FileUtility->Deployer\{closure}() at n/a:n/a
[testing] <  call_user_func() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/Task/Task.php:105
[testing] <  Deployer\Task\Task->run() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/Executor/SeriesExecutor.php:60
[testing] <  Deployer\Executor\SeriesExecutor->run() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/Console/TaskCommand.php:144
[testing] <  Deployer\Console\TaskCommand->execute() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/vendor/symfony/console/Command/Command.php:255
[testing] <  Symfony\Component\Console\Command\Command->run() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/vendor/symfony/console/Application.php:924
[testing] <  Symfony\Component\Console\Application->doRunCommand() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/Console/Application.php:133
[testing] <  Deployer\Console\Application->doRunCommand() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/vendor/symfony/console/Application.php:265
[testing] <  Symfony\Component\Console\Application->doRun() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/vendor/symfony/console/Application.php:141
[testing] <  Symfony\Component\Console\Application->run() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/src/Deployer.php:326
[testing] <  Deployer\Deployer::run() at phar:///html/typo3-test/releases/2/vendor/deployer/dist/dep/bin/dep:136
[testing] <  require() at /html/typo3-test/releases/2/vendor/deployer/dist/dep:4
[testing] < db:truncate [-p|--parallel] [-l|--limit LIMIT] [--no-hooks] [--log LOG] [--roles ROLES] [--hosts HOSTS] [-o|--option OPTION] [--] [<stage>]

[maikschneider]

Before doing any crazy workaround over here, I think I'm going to ask Helmut Hummel, why the configuration:showactive command isn't run with runLevel LEVEL_MINIMAL like configuration:showlocal or others. This would skip the authentication check that causes the error.

[kszymukowicz]

Good catch!

I suppose the reason is that local search only in LocalConfiguration.php file and active needs to run some part of TYPO3 to read from $GLOBALS['TYPO3_CONF_VARS'].

[maikschneider]

Okay, there is a good reason why showactive requires a working DB connection:

showlocal shows config (TYPO3_CONF_VARS) just from LocalConfiguration.php , showactive takes into account what extensions have set in ext_localconf.php , therefore it needs to read those files and by definition, reading those files establish a DB connection (the latter was deprecated only in 11)

[maikschneider]

I guess making db:truncate not crash with activated driver_typo3cms and empty database is not easy since it requires to do some checks before get('db_databases_merged'). This would be overkill for most of the use cases. I guess will implement a method for myself.

Maybe I will do a check for a missing {{deploy_path}}/.dep/releases file and invoke a database:updateschema before db:truncate.