Bitnami lightsail mysql out of memory error

[TomasHog]

Platform

AWS

bndiagnostic ID know more about bndiagnostic ID

20230621-132520-12239

bndiagnostic output

No response

bndiagnostic was not useful. Could you please tell us why?

It gave no additional information that was useful

Describe your issue as much as you can

I have stopped:
sudo /opt/bitnami/ctlscript.sh status
php-fpm not running
apache not running
mysql not running

When i run:
sudo /opt/bitnami/ctlscript.sh start mysql &

I get the following in syslog:

Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822099] mysqld.bin invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822102] mysqld.bin cpuset=/ mems_allowed=0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822107] CPU: 0 PID: 11648 Comm: mysqld.bin Not tainted 4.4.0-1128-aws #142-Ubuntu
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822108] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822110]  0000000000000286 cf9aff91d0202246 ffff8800008c79e0 ffffffff81839e3a
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822112]  ffff8800008c7b98 ffff880003de8e00 ffff8800008c7a50 ffffffff81837b7e
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822114]  0000000000000000 ffff880016eadb40 ffff880007ce8e00 ffff8800008c7a38
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822116] Call Trace:
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822123]  [<ffffffff81839e3a>] dump_stack+0x6d/0x8b
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822127]  [<ffffffff81837b7e>] dump_header+0x5a/0x1c3
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822131]  [<ffffffff8139f491>] ? apparmor_capable+0x131/0x1b0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822134]  [<ffffffff8119941b>] oom_kill_process+0x20b/0x3d0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822136]  [<ffffffff81199828>] out_of_memory+0x1f8/0x470
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822139]  [<ffffffff8119fb53>] __alloc_pages_slowpath.constprop.89+0x943/0xaf0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822142]  [<ffffffff8119ff9f>] __alloc_pages_nodemask+0x29f/0x2b0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822150]  [<ffffffff811eb58c>] alloc_pages_current+0x8c/0x110
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822154]  [<ffffffff8119553b>] __page_cache_alloc+0xab/0xc0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822156]  [<ffffffff81197e80>] filemap_fault+0x160/0x440
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822160]  [<ffffffff812aea96>] ext4_filemap_fault+0x36/0x50
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822164]  [<ffffffff811c58b7>] __do_fault+0x77/0x110
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822166]  [<ffffffff811c9859>] handle_mm_fault+0x1259/0x1b80
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822099] mysqld.bin invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822102] mysqld.bin cpuset=/ mems_allowed=0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822107] CPU: 0 PID: 11648 Comm: mysqld.bin Not tainted 4.4.0-1128-aws #142-Ubuntu
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822108] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822110]  0000000000000286 cf9aff91d0202246 ffff8800008c79e0 ffffffff81839e3a
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822112]  ffff8800008c7b98 ffff880003de8e00 ffff8800008c7a50 ffffffff81837b7e
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822114]  0000000000000000 ffff880016eadb40 ffff880007ce8e00 ffff8800008c7a38
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822116] Call Trace:
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822123]  [<ffffffff81839e3a>] dump_stack+0x6d/0x8b
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822127]  [<ffffffff81837b7e>] dump_header+0x5a/0x1c3
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822131]  [<ffffffff8139f491>] ? apparmor_capable+0x131/0x1b0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822134]  [<ffffffff8119941b>] oom_kill_process+0x20b/0x3d0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822136]  [<ffffffff81199828>] out_of_memory+0x1f8/0x470
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822139]  [<ffffffff8119fb53>] __alloc_pages_slowpath.constprop.89+0x943/0xaf0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822142]  [<ffffffff8119ff9f>] __alloc_pages_nodemask+0x29f/0x2b0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822150]  [<ffffffff811eb58c>] alloc_pages_current+0x8c/0x110
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822154]  [<ffffffff8119553b>] __page_cache_alloc+0xab/0xc0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822156]  [<ffffffff81197e80>] filemap_fault+0x160/0x440
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822160]  [<ffffffff812aea96>] ext4_filemap_fault+0x36/0x50
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822164]  [<ffffffff811c58b7>] __do_fault+0x77/0x110
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822166]  [<ffffffff811c9859>] handle_mm_fault+0x1259/0x1b80
--
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822191] Mem-Info:
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822195] active_anon:101402 inactive_anon:1052 isolated_anon:0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822195]  active_file:53 inactive_file:57 isolated_file:0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822195]  unevictable:913 dirty:3 writeback:0 unstable:0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822195]  slab_reclaimable:2963 slab_unreclaimable:7531
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822195]  mapped:746 shmem:1310 pagetables:1752 bounce:0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822195]  free:1124 free_pcp:131 free_cma:0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822199] Node 0 DMA free:1904kB min:88kB low:108kB high:132kB active_anon:10612kB inactive_anon:264kB 
active_file:20kB inactive_file:24kB unevictable:108kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:108kB dirty:0kB 
writeback:0kB mapped:180kB shmem:296kB slab_reclaimable:448kB slab_unreclaimable:1232kB kernel_stack:144kB pagetables:220kB unstable:0kB bounce:0kB 
free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:572 all_unreclaimable? yes
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822204] lowmem_reserve[]: 0 455 455 455 455
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822207] Node 0 DMA32 free:2592kB min:2684kB low:3352kB high:4024kB active_anon:394996kB inactive_anon
:3944kB active_file:192kB inactive_file:204kB unevictable:3544kB isolated(anon):0kB isolated(file):0kB present:507904kB managed:482548kB mlocked:354
4kB dirty:12kB writeback:0kB mapped:2804kB shmem:4944kB slab_reclaimable:11404kB slab_unreclaimable:28892kB kernel_stack:3264kB pagetables:6788kB un
stable:0kB bounce:0kB free_pcp:524kB local_pcp:524kB free_cma:0kB writeback_tmp:0kB pages_scanned:20856 all_unreclaimable? yes
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822211] lowmem_reserve[]: 0 0 0 0 0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822214] Node 0 DMA: 2*4kB (U) 1*8kB (M) 2*16kB (M) 8*32kB (UM) 13*64kB (UM) 6*128kB (UM) 0*256kB 0*51
2kB 0*1024kB 0*2048kB 0*4096kB = 1904kB
Jun 21 13:18:41 ip-172-26-13-16 kernel:
un 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822230] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822231] 2031 total pagecache pages
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822235] 0 pages in swap cache
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822237] Swap cache stats: add 0, delete 0, find 0/0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822237] Free swap  = 0kB
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822238] Total swap = 0kB
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822239] 130973 pages RAM
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822240] 0 pages HighMem/MovableOnly
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822241] 6361 pages reserved
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822242] 0 pages cma reserved
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.822243] 0 pages hwpoisoned

Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.829983] Killed process 11645 (mysqld.bin) total-vm:757932kB, anon-rss:326984kB, file-rss:1036kB
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858873] in:imklog invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858875] in:imklog cpuset=/ mems_allowed=0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858879] CPU: 0 PID: 1242 Comm: in:imklog Not tainted 4.4.0-1128-aws #142-Ubuntu
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858880] Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858881]  0000000000000286 69dc245cee218dda ffff88001684b9e0 ffffffff81839e3a
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858883]  ffff88001684bb98 ffff88001b3c9c00 ffff88001684ba50 ffffffff81837b7e
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858885]  0000000000000000 ffff880016eadb40 ffff880007ce8e00 ffff88001684ba38
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858887] Call Trace:
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858891]  [<ffffffff81839e3a>] dump_stack+0x6d/0x8b
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858894]  [<ffffffff81837b7e>] dump_header+0x5a/0x1c3
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858897]  [<ffffffff8139f491>] ? apparmor_capable+0x131/0x1b0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858899]  [<ffffffff8119941b>] oom_kill_process+0x20b/0x3d0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858901]  [<ffffffff81198de2>] ? oom_unkillable_task+0xa2/0xe0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858903]  [<ffffffff81199828>] out_of_memory+0x1f8/0x470
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858906]  [<ffffffff8119fb53>] __alloc_pages_slowpath.constprop.89+0x943/0xaf0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858909]  [<ffffffff8119ff9f>] __alloc_pages_nodemask+0x29f/0x2b0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858912]  [<ffffffff811eb58c>] alloc_pages_current+0x8c/0x110
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858914]  [<ffffffff8119553b>] __page_cache_alloc+0xab/0xc0
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858916]  [<ffffffff81197e80>] filemap_fault+0x160/0x440
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858919]  [<ffffffff812aea96>] ext4_filemap_fault+0x36/0x50
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858922]  [<ffffffff811c58b7>] __do_fault+0x77/0x110
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858924]  [<ffffffff811c9859>] handle_mm_fault+0x1259/0x1b80
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858928]  [<ffffffff810c99f0>] ? wake_atomic_t_function+0x30/0x30
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.858932]  [<ffffffff81296084>] ? kmsg_read+0x44/0x60
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.859108] Out of memory: Kill process 1703 (ssm-agent-worke) score 16 or sacrifice child
Jun 21 13:18:41 ip-172-26-13-16 kernel: [ 3888.866788] Killed process 1703 (ssm-agent-worke) total-vm:1242264kB, anon-rss:8412kB, file-rss:0kB

[gongomgra]

Hi @TomasHog,

Thanks for using Bitnami. I haven't been able to download the bndiagnostic information, but according to the MySQL error you shared, your server is running out of free memory. Can you try using a bigger instance?

Apart from the above, can you run the bndiagnostic tool again and share the new code with us?

[TomasHog]

Yes. Where can i send it?
I havent done any changes. I dont understand why i need to upgrade. Looking in memory usage there is still a lot free.

[TomasHog]

bitnami-lampstack-info-20230621-132520-12239.zip

free -h
total used free shared buff/cache available
Mem: 486M 102M 5.9M 5.7M 378M 329M
Swap: 0B 0B 0B

vmstat -s
498448 K total memory
107056 K used memory
224856 K active memory
109052 K inactive memory
15644 K free memory
119332 K buffer memory
256416 K swap cache
0 K total swap
0 K used swap
0 K free swap
22785 non-nice user cpu ticks
181 nice user cpu ticks
9443 system cpu ticks
7727206 idle cpu ticks
9998 IO-wait cpu ticks
0 IRQ cpu ticks
72 softirq cpu ticks
1114 stolen cpu ticks
9048623 pages paged in
718068 pages paged out
0 pages swapped in
0 pages swapped out
1654357 interrupts
3574631 CPU context switches
1687349632 boot time
24927 forks

[gongomgra]

Hi @TomasHog,

According to the output you shared, your server only have 5.9 MB of free memory available, which is not enough to start the database process. Please review the current processes on your server (maybe there is another MySQL process still running?). If you can't find anything, we recommend you to increase the instance type.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.