[Bug]: Configuration files overwritten when updating .rpm package

[smortex]

Describe the bug

OpenSearch 2.10.0 was released today. After updating from 2.9.0, our configuration files with customization where lost and replaced by the default configuration files.

According to our configuration management tools, changes to these files where unexpectedly lost:

• /etc/opensearch/opensearch-notifications/notifications.yml
• /etc/opensearch/opensearch-notifications-core/notifications-core.yml
• /etc/opensearch/opensearch-reports-scheduler/reports-scheduler.yml
• /etc/opensearch/opensearch-security/action_groups.yml
• /etc/opensearch/opensearch-security/allowlist.yml
• /etc/opensearch/opensearch-security/audit.yml
• /etc/opensearch/opensearch-security/config.yml
• /etc/opensearch/opensearch-security/internal_users.yml
• /etc/opensearch/opensearch-security/nodes_dn.yml
• /etc/opensearch/opensearch-security/roles_mapping.yml
• /etc/opensearch/opensearch-security/roles.yml
• /etc/opensearch/opensearch-security/tenants.yml
• /etc/opensearch/opensearch-security/whitelist.yml

To reproduce

1. Install OpenSearch 2.9.0 Debian package rpm package on CentOS
2. Customize the configuration files in /etc/opensearch/
3. Upgrade to OpenSearch 2.10.0

Expected behavior

Configuration files should be kept unchanged when they have local modifications.

Screenshots

No response

Host / Environment

Debian 11 CentOS

Additional context

No response

Relevant log output

No response

[smortex]

Weird, I updated another single-node cluster today and this time did not hit this issue. Both nodes are supposed to have been configured the same way with the same configuration management system, so this non-reproducible behavior is highly unexpected. I will investigate the issue and report my findings here.

[Divyaasm]

Hi @smortex, you can re-open the issue to post you findings. Thanks

[smortex]

We did a manual update from 2.9.0 to 2.10.0 and have not experienced this issue (we where prompted to decide what to do with file with local changes). The 2.11.0 release will allow us to try another update the usual way through configuration management and non-interactively. If we have some trouble, I will re-open to troubleshoot.

Thanks!

[smortex]

Re-opening as this issue do exist but I reported it against the wrong packages: upgrading .deb packages is fine, but updating the rpm packages overwrite the configuration files.

Updating from 2.10.0 to 2.11.0 on a CentOS node showed the issue again. I will edit the issue title and description accordingly. Edit: this has been done

[smortex]

@Divyaasm it looks like I cannot re-open the issue (I only see a comment button). Can you do this for me please? Or should I fill-in a new issue?

It will then be up-to-date in the #4087 meta-issue about packaging.

[smortex]

For what is worth, the issue was still present in the following updates:

• 2.10.0 -> 2.11.0
• 2.11.0 -> 2.12.0
• 2.12.0 -> 2.13.0 (latest version at the time of this message)

[peterzhuamazon]

Hi @smortex I will take a look sometime before 2.14.0 and hopefully can find the cause.

[peterzhuamazon]

Also, have you ever try to upgrade from 2.8.0 to 2.9.0?
I wonder if this issue has been there since the introduction of rpm or just started in 2.10.0.

Thanks.

[peterzhuamazon]

After making multiple attempts I finally able to observe a override on a changed config even when it should not.
Previous failed attempts was due to I am not properly cleanup the correct directories.

This is observed clearly on 2.10 to 2.11 upgrade. Will research more.

[peterzhuamazon]

Identified issues, will PR at some point for the fix.

Thanks.

[peterzhuamazon]

Seems working:

Total size: 1.0 G
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : opensearch-2.10.0-1.x86_64                                                                                                                                                                                                                                                                               1/2
warning: /etc/opensearch/jvm.options created as /etc/opensearch/jvm.options.rpmnew
warning: /etc/opensearch/opensearch-security/audit.yml saved as /etc/opensearch/opensearch-security/audit.yml.rpmsave
warning: /etc/opensearch/opensearch.yml created as /etc/opensearch/opensearch.yml.rpmnew

[peterzhuamazon]

The above is due to I use newly created 2.9.0 which include noreplace for all three, but released 2.10.0 which only has noreplace for jvm and opensearch yml, therefore according to the behavior the non-tracked one will use rpmsave instead of rpmnew.

This behavior will go back to normal after two releases with the same configs:
https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html

Thanks.

[peterzhuamazon]

Remove all manually defined config files in config_dir, and treat everything in config dir as config.

• Make sure opensearch rpm config folders configs preserve on updates #4665