Improve Sitemap generation on larger sites with small memory limit (128M)

[tobimori]

Currently, when rendering the sitemap on our Statamci site with a default memory limit of 128M and around 450 pages across all languages in two collections, all requests fail with a memory limit error.

2024/01/16 14:06:57 [error] 915416#915416: *709626 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /home/enfore-vnt2l/next.enfore.com/vendor/laravel/framework/src/Illuminate/Cache/FileStore.php on line 310; PHP message: PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /home/enfore-vnt2l/next.enfore.com/vendor/symfony/error-handler/Error/FatalError.php on line 1" while reading response header from upstream, client: 64.252.88.246, server: next.enfore.com, request: "GET /sitemap.xml HTTP/1.1", upstream: "fastcgi://unix:/run/php/php8.3-fpm-enfore-vnt2l.sock:", host: "www.enfore.com"

It'd be great if there was a way for sitemap generation to be "off loaded" in the background (e.g. in a queue worker on changes) instead of being on demand so we can avoid setting a high memory limit.

[aerni]

Thanks for bringing this to my attention. The sitemaps are quite intensive as they are querying and filtering a bunch of data. Offloading to a queue sounds like a feasible feature request. For now, I suggest increasing the memory limit.

[aerni]

Generating sitemaps in the background will be possible when PR #167 is merged.