'externalSequenceTimestamp' attribute and function

[maximilianong]

Description

The gate needs a functionality that handles:

What is the timestamp of the data provided on customer side (system of the customer) - customer needs to provide this data.

Gate should only update the data when it has newer data according to the attribute (lastchangedoncustomerside) - job of the GR process.

Feature: Data Update with Timestamp Validation

Description:
The BPDM gate requires a functionality to handle data updates from the customer side based on timestamps. This feature ensures that the data is only updated if the new data from the customer has a more recent timestamp than the existing data.

Functional Requirements:

1. Timestamp Reception:

   • The system must be able to receive a timestamp (externalSequenceTimestamp) from the customer side along with the data payload.
   • This timestamp indicates when the data was last modified on the customer's system.

2. Timestamp Validation:

   • The gate should compare the received timestamp with the existing timestamp stored in the gate
   • If the received timestamp is more recent than the stored timestamp, the data should be updated.
   • If the received timestamp is not more recent, the data update should be rejected, and no changes should be made to the existing data.

Technical Implementation:

• Data Structure:
  • Each data entity should include a externalSequenceTimestamp attribute, which is a timestamp indicating the last modification time on the customer's system.

Example Workflow:

1. Customer sends data to the gate system with a timestamp attribute (externalSequenceTimestamp).
2. The gate system receives the data and invokes the GR process.
3. The GR process compares the received timestamp with the stored timestamp.
4. If the received timestamp is newer, the data is updated, and a success message is logged and sent to the customer.
5. If the received timestamp is not newer, the update is rejected, and an informative message is logged and sent to the customer.

By implementing this feature, the gate system ensures data integrity and consistency, only allowing updates when newer data is provided by the customer.

Additional information

• I'm willing to contribute to this feature
  Contribution will be done by @kunyao-cofinity-x
  Committers to support are @nicoprow / @maximilianong

[maximilianong]

Need a deep dive session.
Peer group: @Sebastian-Wurm / @zygokaktus / @HeyHardy

Standard adaption might need to be done:
0076 / 0074

[maximilianong]

This feature relates to: #643

[maximilianong]

Deepdive session to this feature will happen on the 22. of July at 11 AM CEST to 12 AM:

TEAMS MEETING LINK

[maximilianong]

Notes from the meeting (22.08.2024):

• maybe changing of the name of the attribute
• write down the business value of the attribute

Use case:

• If you send legal entity data and address data and you get the address back before the legal entity
• You send the legal entity first with timestamp X and address with timestamp Y afterwards
• To not get the address without a legal entity first you can use this attribute to identify

Info:

• This is an optional attribute

[Sebastian-Wurm]

What has already been implemented in BPDM #986 is, that the data record is rejected, if the existing timestamp (currentness) is after the incoming timestamp. This is checked for input AND output stage in the gate. Incoming timestamp == null will be handled as a more recent / current data record and thus overrules the comparison.

Legal entity and additional address are two different business partner data records. So, the described use case (see above) is not supported by the implementation of the pull request.

My assumption is, that the implementation is a solution for a sequence problem, which can occur, if messages are manually retriggered. Usually, such sequence problems are solved using a technical integer value, which is incremented and send with each message. The receiver then checks for each sender, if the received message arrived after a message with a higher integer value. Messages are usually batches of business partner data records. So, the integer value could be on the batch message level, rather than on the business partner data record. Having the same business partner data record (external ID!!!) multiple times in one batch is an error, which should be solved on the side of the sharing member and not by the BPDM implementation. Removing duplicates from a list is an easy thing, supported by most modern programming languages.

Mixing a business attribute (currentness) and the technical solution for the sequence problem / duplicate problem is a wrong approach. Also consider, that there might be multiple senders or source systems, which have unsychronized hardware clocks. In this case a timestamp comparison will lead to probably more recent data being overwritten.

Nonetheless, a currentness attribute in conjunction with the source system is from a business perspective a good thing to have, so that in a comparison UI it's more clear which of the compared business partner data records has seemingly the more recent information to it. So, my suggestion is to introduce the currentness attribute in the input and output stage of the gate if it's required for a comparison UI, but not use it for solving any sequence / duplicate problems. Input stage currentness can optionally be filled by the sharing members, output stage currentness must be filled by BPDM, coming from the pool and the data curation services.

[nicoprow]

What has already been implemented in BPDM #986 is, that the data record is rejected, if the existing timestamp (currentness) is after the incoming timestamp. This is checked for input AND output stage in the gate. Incoming timestamp == null will be handled as a more recent / current data record and thus overrules the comparison.

Legal entity and additional address are two different business partner data records. So, the described use case (see above) is not supported by the implementation of the pull request.

My assumption is, that the implementation is a solution for a sequence problem, which can occur, if messages are manually retriggered. Usually, such sequence problems are solved using a technical integer value, which is incremented and send with each message. The receiver then checks for each sender, if the received message arrived after a message with a higher integer value. Messages are usually batches of business partner data records. So, the integer value could be on the batch message level, rather than on the business partner data record. Having the same business partner data record (external ID!!!) multiple times in one batch is an error, which should be solved on the side of the sharing member and not by the BPDM implementation. Removing duplicates from a list is an easy thing, supported by most modern programming languages.

Mixing a business attribute (currentness) and the technical solution for the sequence problem / duplicate problem is a wrong approach. Also consider, that there might be multiple senders or source systems, which have unsychronized hardware clocks. In this case a timestamp comparison will lead to probably more recent data being overwritten.

Nonetheless, a currentness attribute in conjunction with the source system is from a business perspective a good thing to have, so that in a comparison UI it's more clear which of the compared business partner data records has seemingly the more recent information to it. So, my suggestion is to introduce the currentness attribute in the input and output stage of the gate if it's required for a comparison UI, but not use it for solving any sequence / duplicate problems. Input stage currentness can optionally be filled by the sharing members, output stage currentness must be filled by BPDM, coming from the pool and the data curation services.

@maximilianong @Sebastian-Wurm
I think it's a good idea to actually separate those two features. What is now called currentness we can specifically rename to make it clear as a technical attribute. In my opinion this issue then just needs to be adapted regarding the naming of the attribute to make it clearly distinct from a possible "semantic currentness" attribute in the future. Otherwise requirements and impact stay the same.

We already have a technical field managed by the sharing member named "externalId". Maybe we can use than naming convention for "external" for this attribute as well. Something like "externalRevision", "externalSequenceTimestamp" or similar.

[stephanbcbauer]

After a brief consultation with Sebastian, the feature is set to 'backlog' status. Associated committer is also fine with it.

[maximilianong]

After another refinement session we will change the attribute name to 'externalSequenceTimestamp'