Log rotation for all components

[morgan-patou]

Bug (improvement) description

As far as I could see, there doesn't seem to be log rotation for some of the components installed by the Ansible playbooks and the number of logs kept is pretty different for each as well.

Playbook version

Status as of 20-Feb-2024 - installing a Community 7.3.0

Details

Solr -- Appears to be "OK", it comes by default with 9 log files (of 4Mb each max)

# grep -i max /opt/alfresco/search-services-*/logs/log4j.properties
log4j.appender.file.MaxFileSize=4MB
log4j.appender.file.MaxBackupIndex=9
#

Share -- Does not appear to contain any rotation/max logs

# grep -i max /var/opt/alfresco/tomcat/webapps/share/WEB-INF/classes/log4j.properties
#

Alfresco -- Does not appear to contain any rotation/max logs

# grep -i max /var/opt/alfresco/tomcat/webapps/alfresco/WEB-INF/classes/log4j.properties
#

Nginx -- Does not appear to contain any rotation/max logs

# grep _log /etc/nginx/conf.d/*
/etc/nginx/conf.d/alfresco.conf:    access_log /var/log/alfresco/nginx.alfresco.access.log;
/etc/nginx/conf.d/alfresco.conf:    error_log  /var/log/alfresco/nginx.alfresco.error.log error;
/etc/nginx/conf.d/ssl.hostname.domain.com.conf:    access_log /var/log/alfresco/nginx.alfresco.access.log;
/etc/nginx/conf.d/ssl.hostname.domain.com.conf:    error_log  /var/log/alfresco/nginx.alfresco.error.log error;
#
# ## Nginx is configured to use /var/log/alfresco for logs
# ## But the Nginx logrotate only manages /var/log/nginx
# ## --> Nginx logs are never rotated/cleaned
#
# grep ^/ /etc/logrotate.d/nginx
/var/log/nginx/*.log {
# 

Tomcat -- Appears to be "OK", it comes by default with 90 log files

# grep -i max /etc/opt/alfresco/tomcat/conf/logging.properties
1catalina.org.apache.juli.AsyncFileHandler.maxDays = 90
2localhost.org.apache.juli.AsyncFileHandler.maxDays = 90
3manager.org.apache.juli.AsyncFileHandler.maxDays = 90
4host-manager.org.apache.juli.AsyncFileHandler.maxDays = 90
#

ActiveMQ -- Appears to be "OK", it comes by default with 5 log files of 1Mb each

# grep -i max /etc/opt/alfresco/activemq/conf/log4j.properties
log4j.appender.logfile.maxFileSize=1024KB
log4j.appender.logfile.maxBackupIndex=5
log4j.appender.audit.maxFileSize=1024KB
log4j.appender.audit.maxBackupIndex=5
#

Transform Service (AIO) -- Does not appear to contain any rotation/max logs BUT the start/stop script /opt/alfresco/ats-ate-aio.sh will overwrite the log file at each startup, so considering ATS isn't logging too much information, that might be "OK" to leave it like that?

I do not have access to enterprise components so I couldn't check the status there.

Proposal

I believe the configuration should be added to at least have log rotation for all components and optionally (if possible) align the number of logs to keep.

The biggest problem from my point of view being Nginx (+Alfresco/Share), since it is never rotated, the access log will grow very fast and fill the filesystem (even if the environment isn't used, because of Solr tracking).

1. Nginx
   Considering that Nginx installation depends on the OS, should the nginx log files be put back into the standard log folder (/var/log/nginx/), so that the logrotate configured by the OS package of Nginx is taken and applied OOTB (with its rotate 14 config)? This would need to be changed in HTTP + HTTPS templates (e.g. here and here)
   /!\ to be checked if SELinux is still OK on the default folder but normally the OS package takes care of that.

Alternatively, it can be added into a custom logrotate on nginx hosts, but in this case, it might be difficult to handle the owner/permissions and pre/post tasks as the OS package logrotate does. I believe using the OOTB folder would simplify the log rotation (but then it's outside of the volume /var/log/alfresco/ that is asked for all logs... so pros vs cons)

2. Alfresco + Share
   I assume this could be easily done by modifying the log4j.properties inside the WAR files (or v2 for higher versions of Alfresco) or adding a custom logrotate or a cronjob to delete log files older than XX or ...

What do you think on all that?

[gionn]

Hi Morgan, we know there is a lot of space for improvements and we have in roadmap to improve the overall logging approach of the playbook to make it more production-grade and ease the integration with 3rd party logging systems. (OPSEXP-2085)

I think that:

• moving the nginx logs path to the default folder
• configure system logrotate config for alfresco/share tomcat logs

are two good tasks which go in the right direction.

I am not sure if configuring log rotation at tomcat level via log4j has more advantages, but more in general I would prefer having a single system-wide approach via system logrotate.

A nice to have for nginx is to mv the eventual existing logs to the new folder and place symlinks in place of them, so if someone is used to it can still find them.

Thank you once more for taking the time to investigate and provide detailed suggestions for potential enhancements.

[morgan-patou]

I would also believe logrotate is easier to setup and manage for multiple logs. However, there is a point that might be annoying is that the Alfresco and Share logs are rotated by Tomcat with a suffix. E.g. from alfresco.log to alfresco.log.2024-04-04. So from a logrotate point of view, it might be difficult to have the rotation taking place properly, as all these files (with and without the suffix) would most probably be considered as different log files and not as a rotation of the normal log file. Not sure if that was clear? Basically, there are 2 rotations, one at Tomcat level and potentially the logrotate on top of it, so might be difficult to explain to logrotate what it needs to do.

Hence my point about modifying the war files log4j.poperties (or having a crontab that makes sure nothing older than 10/15/30 days is left in the folder, whatever its name might be).

[gionn]

Yes, log rotation should be managed in a single place only, so it needs to be disabled in the tomcat log4j configuration.

It should be possibile to have logrotate match all the eventually existing already-rotated tomcat logs, to compress and ultimately deleting them when they reach expiration.