MariaDB update to 10.11 or higher

[scifijunk]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

After a power outage linuxserver/mariadb had stopped running when my k3s cluster came back online.

Expected Behavior

For linuxserver/mariadb after the my k3s cluster came back online.

Steps To Reproduce

1. I had attempted to log in to my Friendica node in my k3s cluster and noted it gave me an error relating to database failure.
2. logged into Portainer to look at the logs for linuxserver/mariadb and noted error log number it displayed.
3. logged in to that specific node of my k3s cluster and looked at the error log.
4. Noted the log was repeating the same error relating to not having access to aria_log_control along with something about a crash.
5. Google searched for possible fixes but found questionable solutions.
6. Started to stair stepping the Tags in the image until I got the error again
7. Noted the errors happen on 10.11.3-r0-ls98 and not on 10.6.13-r0-ls97 or lower

Environment

- OS: Ubuntu 22.04.2 LTS (GNU/Linux 5.15.0-72-generic x86_64)
- How docker service was installed: Portainer

CPU architecture

x86-64

Docker creation

Portainer as follows:

Name: mariadb
Namespace: friendica-base
Application Type: Deployment
Creation: scifijunkie 2022-08-27 20:46:06 Deployed from application form

Configuration
Container: mariadb

Environment variable:

MYSQL_DATABASE: redacted
MYSQL_PASSWORD: redacted
MYSQL_ROOT_PASSWORD: redacted
MYSQL_USER: redacted
PGID: 1000
PUID: 1000
TZ: America/Chicago

Container logs

Error log as follows:

230528 22:12:22 mysqld_safe Starting mariadbd daemon with databases from /config/databases
2023-05-28 22:12:22 0 [Note] Starting MariaDB 10.11.3-MariaDB-log source revision 0bb31039f54bd6a0dc8f0fc7d40e6b58a51998b0 as process 394
2023-05-28 22:12:22 0 [Note] Using unique option prefix 'myisam_recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
2023-05-28 22:12:22 0 [ERROR] mariadbd: Can't lock aria control file '/config/databases/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds
2023-05-28 22:12:23 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2023-05-28 22:12:23 0 [Note] InnoDB: Number of transaction pools: 1
2023-05-28 22:12:23 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-05-28 22:12:23 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2023-05-28 22:12:24 0 [Note] InnoDB: Using Linux native AIO
2023-05-28 22:12:24 0 [Note] InnoDB: Initializing buffer pool, total size = 256.000MiB, chunk size = 4.000MiB
2023-05-28 22:12:24 0 [Note] InnoDB: Completed initialization of buffer pool
2023-05-28 22:12:24 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
2023-05-28 22:12:24 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.6.12. You must start up and shut down MariaDB 10.7 or earlier.
2023-05-28 22:12:24 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-05-28 22:12:24 0 [Note] InnoDB: Starting shutdown...
2023-05-28 22:12:24 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-05-28 22:12:24 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-05-28 22:12:24 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-05-28 22:12:24 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-05-28 22:12:24 0 [ERROR] Aborting

[github-actions]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[lennvilardi]

throw me same error, loading an old tag from 2 weeks works

[vivoras]

Same error. With the tag 10.6.13 it works fine

[nemchik]

We are aware of an issue when updating the container image where MariaDB version is updated from 10.6.13 to 10.11.3 and are planning to release an announcement on https://info.linuxserver.io/

Until the announcement is live, you can preview the announcement here https://github.com/linuxserver/cstate/blob/2505217ea6d06b00a1b413d44108c47f0a6cccee/content/issues/2023-05-28-mariadb.md

I will update this issue with the permanent link once the announcement is live.

[nemchik]

Live link is https://info.linuxserver.io/issues/2023-05-29-mariadb/

[el-fredo]

I get the same error and followed exactly the instructions in the announcement. But after I update the container back to the current tag (mariadb:latest), the error is back. The next command (mariadb-upgrade -u root -p) does not work anymore. I get the error message:
Reading datadir from the MariaDB server failed. Got the following error when executing the 'mysql' command line client
ERROR 2002 (HY000): Can't connect to local server through socket '/run/mysqld/mysqld.sock' (2)
FATAL ERROR: Upgrade failed

If I stay with the tag mariadb:10.6.13, then everything works fine.

Does anyone else have any further information for me on how to update MariaDB?

[j0nnymoe]

Its all explained in the @nemchik posted above your comment.

Though it's not unlikely some users might have edge cases and further steps might be required. Just not something that has come up in our testing.

[el-fredo]

As I said, I followed exactly the procedure listed in the link above. I think I should stay with the current working version for now and refrain from any updates.

[vivoras]

I have followed the instructions and it works perfectly. Thanks

[Tomte-Fan]

As I said, I followed exactly the procedure listed in the link above. I think I should stay with the current working version for now and refrain from any updates.

I upgraded 2 dockers and one went fine following the instructions, the other still gave me the errors even if I tried using the instructions several times. I had to manually stop mariadb in the docker (s6-svc -d /run/services/..) after running the mariadb-check commands and then stopping and upgrading the docker, then it started without errors and I could run the mariadb-upgrade.

[nemchik]

For anyone having issues even after following the instructions in the announcement, please post the following:

• your container logs (docker logs mariadb)
• the contents of the <random_name>.err log file listed in the container log
• the output of running the 4 repair commands from https://info.linuxserver.io/issues/2023-05-29-mariadb/ on 10.6.13
• the output of running the upgrade command from https://info.linuxserver.io/issues/2023-05-29-mariadb/ on 10.6.13

Note, the announcement does not explicitly state the need to run the upgrade command on 10.6.13, but the container logs may state that it should be run (and if so, you should!).

What we're looking for is any errors in any of the above that could help find a solution to the edge cases. We may not be able to provide a universal fix or set of instructions, but if we are able to help individual users get to a working state we'll do what we can.

[Sblop]

As I said, I followed exactly the procedure listed in the link above. I think I should stay with the current working version for now and refrain from any updates.

I upgraded 2 dockers and one went fine following the instructions, the other still gave me the errors even if I tried using the instructions several times. I had to manually stop mariadb in the docker (s6-svc -d /run/services/..) after running the mariadb-check commands and then stopping and upgrading the docker, then it started without errors and I could run the mariadb-upgrade.

I was getting crazy nothing worked. - However once I understood exactly what were written above this fixed the problem. Why I do not know but it works.

I ran the 4 commands on the previous image version, without any problems or errors.
If I use the new image it fails all the time.
Back to old version. Running this command stops mariadb inside the container:
s6-svc -d /run/service/svc-mariadb
note: no output.
Then I stopped the container, and replaced the image tag with latest and the container started. I had to manually run the update command:
docker exec -it mariadb mariadb-upgrade -u root -pPASSWORD
Now it works.

Thanks to @Tomte-Fan above.

[scifijunk]

Well, I ran all 4 commands and got no error only warnings about what is not supported by each command. The main warning I saw was about the storage and which command didn't support it. Anyway, I ran the s6-svc -d /run/service/svc-mariadb and while there is no real way to stop a container in k3s other than to delete and recreate it so I just redeploy the container with the latest tag.

It failed to start and it spit out an error log number. I switched back to 10.6.13-r0-ls97 and it started to run like it did before. The error log had close to the same repeating error as I stated originally. Now what I mean by that is it does not say anything like "mariadbd: Can't lock aria control file '/config/databases/aria_log_control' for exclusive use, error: 11. Will retry for 30 seconds". Now this is not a big deal to me since Friendica is more than happy using 10.6.13 and running the 4 MariaDB-check commands has, in fact, helped Friendica to run a little better.

I do thank all that gave ideas to fix this problem.

[el-fredo]

I ran the 4 commands on the previous image version, without any problems or errors. If I use the new image it fails all the time. Back to old version. Running this command stops mariadb inside the container: s6-svc -d /run/service/svc-mariadb note: no output. Then I stopped the container, and replaced the image tag with latest and the container started. I had to manually run the update command: docker exec -it mariadb mariadb-upgrade -u root -pPASSWORD Now it works.

Thanks to @Tomte-Fan above.

Thanks, this worked for me, too.