[BUG] OpenSearch duplicate plugins and modules on upgrade

[Kampfmoehre]

Describe the bug
I set up OpenSearch 1.2.0 via this playbook some days ago and I got it to work. Now I wanted to update the OpenSearch version to fix the Log4J security vulnerability but I ran into some problems. Via journalctl I could see that duplicate libs, plugins and modules prevented OpenSearch from starting.

To Reproduce
Steps to reproduce the behavior:

1. Install OpenSearch via this Playbook
2. Update OpenSearch via this Playbook by setting OS_VERSION variable to something newer (1.2.1 in my case)
3. Run the playbook again
4. See error

Expected behavior
Update should work in normal environments.

Playbook Name
Opensearch

Host/Environment (please complete the following information):

• Ansible Version: 2.9.27
• Playbook Version: 1.1.0

Additional context
I tried resolving the error by manually deleting old plugin jar files but new errors keep popping up on another plugin. And when all duplicate plugins where removed the error would pop up again with different modules.

Example logs from journal:

uncaught exception in thread [main]
java.lang.IllegalStateException: jar hell!
class: org.opensearch.cli.Command
jar1: /usr/share/opensearch/lib/opensearch-cli-1.2.1.jar
jar2: /usr/share/opensearch/lib/opensearch-cli-1.2.0.jar

uncaught exception in thread [main]
java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.SocketAppender$Builder
jar1: /usr/share/opensearch/lib/log4j-core-2.11.1.jar
jar2: /usr/share/opensearch/lib/log4j-core-2.15.0.jar

java.lang.IllegalStateException: codebase property already set: codebase.protocol -> file:/usr/share/opensearch/plugins/opensearch-sql/protocol-1.2.1.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-sql/protocol-1.2.0.0.jar

java.lang.IllegalStateException: codebase property already set: codebase.reindex-client -> file:/usr/share/opensearch/plugins/opensearch-sql/reindex-client-1.2.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-sql/reindex-client-1.2.1.jar

java.lang.IllegalStateException: codebase property already set: codebase.repository-url -> file:/usr/share/opensearch/modules/repository-url/repository-url-1.2.1.jar, cannot set to file:/usr/share/opensearch/modules/repository-url/repository-url-1.2.0.jar

In the end the only way for me was to completely remove plugin and module directory (and create them again or the Playbook would fail). This should be handled by the playbook.

[stockholmux]

@peterzhuamazon could you look into this?

[peterzhuamazon]

@saravanan30erd any thoughts?

[saravanan30erd]

@Kampfmoehre Currently we don't have support for upgrade from this playbook because DEB/RPM packages are not available for opensearch installation/upgrade yet, and we are using binary files created from source for the installation. Once DEB/RPM packages are available then upgrade will be straightforward from the playbook.

@peterzhuamazon whats the status on creating RPM/DEB packages?
I can create cleanup tasks so users can use it to cleanup first and then install it from scratch again. But they have to do backup and restore the data. If we want to support the upgrade, then I think this is same duplication steps which should be available in RPM packages once its ready (yum install / upgrade).

[Kampfmoehre]

Thanks for the quick reply and the clarification. What would be the steps needed to perform a save upgrade, is it enough to clean plugin/modules dir or should I clean more or less? Until we have the packages it would be good to know a workaround, there might be others that want to upgrade OpenSearch to mitigate the Log4J security vulnerability.

[saravanan30erd]

@Kampfmoehre You can take the data backup first, then clean the complete opensearch directory, and install the new version from scratch, once the OS is up then restore the data. These are the steps more or less but I have to test it myself, and I will create a cleanup task in playbook after that.

[peterzhuamazon]

@Kampfmoehre Currently we don't have support for upgrade from this playbook because DEB/RPM packages are not available for opensearch installation/upgrade yet, and we are using binary files created from source for the installation. Once DEB/RPM packages are available then upgrade will be straightforward from the playbook.

@peterzhuamazon whats the status on creating RPM/DEB packages? I can create cleanup tasks so users can use it to cleanup first and then install it from scratch again. But they have to do backup and restore the data. If we want to support the upgrade, then I think this is same duplication steps which should be available in RPM packages once its ready (yum install / upgrade).

We are aiming to get rpm ready for 1.3 release at this point.

[remil1000]

Actually I had the very same issue updating to 1.2.1 and 1.2.2 for the log4j2 vulns

As the tar extract overwrite existing installation without removing files from previous releases I think it clutters classpath (or at least the JVM has issue finding the right .jar files to load)

I've been using symlink to address this problem:
https://github.com/remil1000/opensearch-ansible-playbook/blob/main/roles/opensearch/tasks/opensearch.yml#L29-L53

The other issue I see is the configuration is in the very same path as where the tar.gz is unpacked, so if the .tar.gz is not in /tmp/ anymore (cleanup or reboot) the whole configuration is overwritten on next run and cluster restarted

I'll try to backport the changes I've made on my fork and create a relevant PR

[peterzhuamazon]

Actually I had the very same issue updating to 1.2.1 and 1.2.2 for the log4j2 vulns

As the tar extract overwrite existing installation without removing files from previous releases I think it clutters classpath (or at least the JVM has issue finding the right .jar files to load)

I've been using symlink to address this problem: https://github.com/remil1000/opensearch-ansible-playbook/blob/main/roles/opensearch/tasks/opensearch.yml#L29-L53

The other issue I see is the configuration is in the very same path as where the tar.gz is unpacked, so if the .tar.gz is not in /tmp/ anymore (cleanup or reboot) the whole configuration is overwritten on next run and cluster restarted

I'll try to backport the changes I've made on my fork and create a relevant PR

Thanks @remil1000 we are still in the process to get the RPM ready and that should help a lot during upgrades.

[Kampfmoehre]

@peterzhuamazon could you also provide a deb package? That would cover most distributions out there and would really be helpful for a lot of people I guess.

[peterzhuamazon]

@peterzhuamazon could you also provide a deb package? That would cover most distributions out there and would really be helpful for a lot of people I guess.

In current road map we are going to work on deb once rpm released.

Thanks.