Enhanced Value Handling in Output Stage

[maximilianong]

Description

Objective:
Improve the handling of specific values in the output stage, ensuring data consistency and clarity for customers when certain values are not included in the response from the cleaning service.

Current Behavior (Version 24.03)
Customer Provided Values:

• Legal Name: If the address type is “LegalAndSiteMainAddress” or “LegalAddress”, take the value from the message. If the message is null, take it from the input stage.
• Short Name: If the message value is null, take the value from the input stage.
• Legal Form: If the message value is null, take the value from the input stage.
• isOwnCompanyData: Always take from the input stage.
• Name Parts: Always take from the input stage.

Customer Expectations:
Customers expect to receive these values in the response from the output stage.

Current System Behavior:
If a value is empty before being returned to the customer, it is copied from the output stage.

Updated Behavior (Version 24.05)
New System Behavior:
When the values are empty and not included in the response from the output stage, the system will behave as follows:

First-Time Data Set:

If the data set is newly shared and not all values were included by the cleaning service, the values will be set to ‘none’.

Existing Data Set in the Pool:

If the data set already exists in the pool and not all values were included by the cleaning service, or were shared differently by the sharing member, the values will be taken from the pool.

• Legal Name: Can't be empty. If it comes empty to the pool it will be thrown a task error. It will not be put into the pool or in the output stage.
• Short Name: If the value is detected null by the cleaning service, it should also be null in the pool and gate. Do not want to have different data in pool and gate. If it's null it should NOT be taken from the input stage.
• Legal Form: If the value is detected null by the cleaning service, it should also be null in the pool and gate. Do not want to have different data in pool and gate. If its null it should NOT be taken from the input stage.
• isOwnCompanyData: Taken from the input stage. The cleaning service should not be able to set this value or any of the attributes to null. It should be true or false. If someone claims to be the owner 'true' , cleaning service should leave it like this.
• Name Parts: Always take from the input stage.

Rationale:
The pool is considered the single source of truth.

If the cleaning service cannot find any values and the data set does not yet exist in the pool, using the information originally provided by the sharing member for the gate would result in inconsistent data between the pool and the gate for that data set.
If the cleaning service cannot find a legal form, and there is nothing in the pool either, the system should reflect the accurate state. If the sharing member wants to use the "old" legal form they originally shared, they must update their backend system.

By implementing these changes, we aim to ensure data consistency and accuracy, maintaining the integrity of the information shared between the pool and the gate, while providing clear insights to customers regarding the source and availability of each value.

Additional information

• I'm willing to contribute to this feature

Contribution by @kunyao-cofinity-x
Commiter to support this @nicoprow / @SujitMBRDI / @maximilianong

[maximilianong]

Will be taken to a deep dive session with this one:
#726

[Sebastian-Wurm]

Probably OK, but needs to be aligned with the CX-076. @zygokaktus: will you take this?

I am unsure, whether we also need to look at the other attributes as well. Also I am unsure if data records claimed as from my own company data must be regarded separately. Where is the source of truth then? If the GR process / cleaning service prefers to take over my data, the solution proposed here will probably also work.

[stephanbcbauer]

After a brief consultation with Sebastian, the feature is set to 'backlog' status. Associated committer is also fine with it.

[stephanbcbauer]

Put the label open decision, cause there is further discussion needed.