Secrets related error trace encountered while upgrading

[shayancanonical]

Steps to reproduce

1. juju add-model upgrades
2. juju deploy -n 3 mysql --channel=8.0/stable
3. wait for all units to become active
4. juju run mysql/leader pre-upgrade-check
5. juju refresh mysql --channel 8.0/edge

Expected behavior

No error traces in debug-log and no units should go into error state

Actual behavior

Units go into error state, then get refreshed and eventually into active after upgrading\

Versions

Operating system: Ubuntu 22.04.3 LTS

Juju CLI: 3.1.8-genericlinux-amd64

Juju agent: 3.1.8

Charm revision: 234

LXD: 5.21.1 LTS

Log output

Juju debug log:

unit-mysql-0: 19:31:03 ERROR unit.mysql/0.juju-log Uncaught exception while in charm code:                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 2955, in _run                                                                                                                             
    result = subprocess.run(args, **kwargs)  # type: ignore                                                                                                                                                        
  File "/usr/lib/python3.10/subprocess.py", line 526, in run                                                                                                                                                       
    raise CalledProcessError(retcode, process.args,                                                                                                                                                                
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-mysql-0/secret-get', 'secret://4cc512df-2e29-4fb5-88f2-9e293cc2bf71/cott3jj273aj9fhrnid0', '--label', 'database-peers.mysql.app', '--format=json
')' returned non-zero exit status 1.                                                                                                                                                                               
                                                                                                                                                                                                                   
The above exception was the direct cause of the following exception:                                                                                                                                               
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 818, in <module>                                                                                                                             
    main(MySQLOperatorCharm)                                                                                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 436, in main                                                                                                                               
    _emit_charm_event(charm, dispatcher.event_name)                                                                                                                                                                
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/main.py", line 144, in _emit_charm_event                                                                                                                  
    event_to_emit.emit(*args, **kwargs)                                                                                                                                                                            
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 351, in emit                                                                                                                          
    framework._emit(event)                                                                                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 853, in _emit                                                                                                                         
    self._reemit(event_path)                                                                                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/framework.py", line 942, in _reemit                                                                                                                       
    custom_handler(event)                                                                                                                                                                                          
  File "/var/lib/juju/agents/unit-mysql-0/charm/./src/charm.py", line 234, in _on_config_changed                                                                                                                   
    if not self._is_peer_data_set:                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 632, in _is_peer_data_set                                                                                                      
    and self.get_secret("app", ROOT_PASSWORD_KEY)                                                                                                                                                                  
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/mysql/v0/mysql.py", line 704, in get_secret                                                                                                             
    if not (value := self.peer_relation_data(scope).fetch_my_relation_field(peers.id, key)):                                                                                                                       
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1256, in fetch_my_relation_field                                                                        
    if relation_data := self.fetch_my_relation_data([relation_id], [field], relation_name):                                                                                                                        
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1245, in fetch_my_relation_data                                                                         
    data[relation.id] = self._fetch_my_specific_relation_data(relation, fields)                                                                                                                                    
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 534, in wrapper                                                                                         
    return f(self, *args, **kwargs)                                                                                                                                                                                
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2079, in _fetch_my_specific_relation_data                                                               
    self.component, self.secret_fields, relation, fields                                                                                                                                                           
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1813, in secret_fields                                                                                  
    self.static_secret_fields if self.static_secret_fields else self.current_secret_fields                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1832, in current_secret_fields                                                                          
    if content := self._get_group_secret_contents(relation, group):                                                                                                                                                
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2066, in _get_group_secret_contents                                                                     
    result = super()._get_group_secret_contents(relation, group, secret_fields)
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 996, in _get_group_secret_contents
    if (secret := self._get_relation_secret(relation.id, group)) and (
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 506, in wrapper
    return f(self, *args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2055, in _get_relation_secret
    return self.secrets.get(label, secret_uri, legacy_labels=self._previous_labels())
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 702, in get
    if secret.meta:
  File "/var/lib/juju/agents/unit-mysql-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 615, in meta
    self._secret_meta = self._model.get_secret(id=self._secret_uri, label=self.label)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 276, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 3311, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-mysql-0/charm/venv/ops/model.py", line 2957, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR only unit leaders can update an application owned secret label

unit-mysql-2: 19:31:03 ERROR unit.mysql/2.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/model.py", line 2955, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-mysql-2/secret-get', 'secret://4cc512df-2e29-4fb5-88f2-9e293cc2bf71/cott3jj273aj9fhrnid0', '--label', 'database-peers.mysql.app', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-2/charm/./src/charm.py", line 818, in <module>
    main(MySQLOperatorCharm)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/main.py", line 436, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/framework.py", line 853, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/framework.py", line 942, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-2/charm/./src/charm.py", line 234, in _on_config_changed
    if not self._is_peer_data_set:
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/mysql/v0/mysql.py", line 632, in _is_peer_data_set
    and self.get_secret("app", ROOT_PASSWORD_KEY)
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/mysql/v0/mysql.py", line 704, in get_secret
    if not (value := self.peer_relation_data(scope).fetch_my_relation_field(peers.id, key)):
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1256, in fetch_my_relation_field
    if relation_data := self.fetch_my_relation_data([relation_id], [field], relation_name):
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1245, in fetch_my_relation_data
    data[relation.id] = self._fetch_my_specific_relation_data(relation, fields)
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 534, in wrapper
    return f(self, *args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2079, in _fetch_my_specific_relation_data
    self.component, self.secret_fields, relation, fields
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1813, in secret_fields
    self.static_secret_fields if self.static_secret_fields else self.current_secret_fields
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1832, in current_secret_fields
    if content := self._get_group_secret_contents(relation, group):
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2066, in _get_group_secret_contents
    result = super()._get_group_secret_contents(relation, group, secret_fields)
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 996, in _get_group_secret_contents
    if (secret := self._get_relation_secret(relation.id, group)) and (
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 506, in wrapper
    return f(self, *args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 2055, in _get_relation_secret
    return self.secrets.get(label, secret_uri, legacy_labels=self._previous_labels())
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 702, in get
    if secret.meta:
  File "/var/lib/juju/agents/unit-mysql-2/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 615, in meta
    self._secret_meta = self._model.get_secret(id=self._secret_uri, label=self.label)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/model.py", line 276, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/model.py", line 3311, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-mysql-2/charm/venv/ops/model.py", line 2957, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR only unit leaders can update an application owned secret label

Additional context

[github-actions]

https://warthogs.atlassian.net/browse/DPE-4313