[Upgrade] Keep the same Paths set for the old snap until unit is free to upgrade

[phvalguima]

At upgrade, two things should happen: (1) the refresh, which replaces the charm for the new charm version; and (2) the unit must wait for its turn to eventually be able to upgrade to the new version.

Therefore, for a certain time, we need to account the original Paths and following env. variables must be created with the old values. Besides, we also must account to the fact Paths will be upgraded within a hook, where it will:

1. Start with the old snap revision
2. Stop the service
3. Run the upgrade of the snap
4. Start the service with the new snap revision
5. Finish hook

During this time, we need to be able to update Paths and environment variables accordingly.

With current #242, this is the error that happens at this stage:

unit-opensearch-0: 20:13:16 ERROR unit.opensearch/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_plugin_manager.py", line 422, in _installed_plugins
    return self._opensearch.run_bin("opensearch-plugin", "list").split("\n")
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_distro.py", line 182, in run_bin
    return self._run_cmd(script_path, args, stdin=stdin)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/tenacity/__init__.py", line 325, in iter
    raise retry_exc.reraise()
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/tenacity/__init__.py", line 158, in reraise
    raise self.last_attempt.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_distro.py", line 338, in _run_cmd
    raise OpenSearchCmdError(output.stderr)
charms.opensearch.v0.opensearch_exceptions.OpenSearchCmdError: could not find java in OPENSEARCH_JAVA_HOME at /snap/opensearch/current/usr/lib/jvm/java-21-openjdk-amd64/bin/java


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_base_charm.py", line 615, in _on_config_changed
    if self.plugin_manager.run():
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_plugin_manager.py", line 158, in run
    logger.debug(f"Status: {self.status(plugin)}")
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_plugin_manager.py", line 328, in status
    if not self._is_installed(plugin):
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_plugin_manager.py", line 346, in _is_installed
    return plugin.name in self._installed_plugins()
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_plugin_manager.py", line 424, in _installed_plugins
    raise OpenSearchPluginError("Failed to list plugins: " + str(e))
charms.opensearch.v0.opensearch_plugins.OpenSearchPluginError: Failed to list plugins: could not find java in OPENSEARCH_JAVA_HOME at /snap/opensearch/current/usr/lib/jvm/java-21-openjdk-amd64/bin/java


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-opensearch-0/charm/./src/charm.py", line 264, in <module>
    main(OpenSearchOperatorCharm)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 544, in main
    manager.run()
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 520, in run
    self._emit()
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 509, in _emit
    _emit_charm_event(self.charm, self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/main.py", line 143, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 352, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 851, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/framework.py", line 941, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/opensearch_base_charm.py", line 627, in _on_config_changed
    self.status.set(BlockedStatus(PluginConfigChangeError), app=True)
  File "/var/lib/juju/agents/unit-opensearch-0/charm/lib/charms/opensearch/v0/helper_charm.py", line 84, in set
    context.status = upgrade_status
  File "/var/lib/juju/agents/unit-opensearch-0/charm/venv/ops/model.py", line 404, in status
    raise RuntimeError('cannot set application status as a non-leader unit')

[github-actions]

https://warthogs.atlassian.net/browse/DPE-4234

[phvalguima]

We should account here for some other scenarios, e.g. what happens if we stop the upgrade in the middle, then some units will live with this old snap revision for, potentially, quite some time. Another option, during a failed upgrade, users may decide to rollback or forward the snaps.