Synchronise efforts and ideas on Validator and Datastore

[siemvaessen]

As both the IATI Validator and IATI Datastore have a dependency and have been built as separate services under different contracts by different suppliers we need to align efforts and ideas to ensure both services are in par with the objectives raised in the IATI context. One major enhancements come to mind that have require a similar approach and will benefit all end user.

Process data on activity level rather than dataset level

Both Datastore and Validator will be able to process data way faster as Datastore currently provides sha1 keys for change detection in datasets. The current process is based on reviewing a dataset and when datachange is detected, sha1 key is updated, validator picks up the new sha1 key, checks the dataset and the datastore will then start processing that dataset. In the new scenario Datastore keeps track of not just the sha1 keys for each dataset that it retrieves from the IATI (metadata) Registry but keeps track of sha1 keys for each and every activity. If this is in place Datastore will only process an IATI Activity when its sha1 keys changes and Validator will only pickup the changed IATI Activity with changes. This will increase data processing (in both services)

Pro's:

• data processing will dramatically increase

Con's:

• none?

[rolfkleef]

Guess what, I have plenty of thoughts on this ;-)

On validation itself:

• The current ruleset approach uses pattern matching, so feeding it an iati-activity rather than iati-activities as root element should be easy to do..
• Except: we currently take some information from iati-activities attributes (most notably: the declared IATI version) -- it should be reasonably straightforward to add these as parameters or by encapsulating each activity in an iati-activities element before applying the ruleset.
• Currently, XSD schema validation should be backward compatible (if it passes 2.02 schema validation, it should also pass 2.03 schema validation), but the IATI Rulesets are not backward compatible (a few rules are more restrictive in 2.03). So the IATI version is needed.
• Currently, there is duplication of components (file handling, schema validation, front-end). Once the Datastore can use only the ruleset component of the Validator, it should be a lot more streamlined already. If the ruleset validation results are stored in the Datastore too, a much more comprehensive front-end is possible.

On ingesting datasets:

• So far, I look at all files of a publisher together as an atomic update (dataset): "this set of activities represent the current state of what we do". That creates choices around files that cannot be retrieved and files with validation errors. If we also pick and choose from within files, that adds a few more scenarios when to consider an activity "deleted/no longer published".
• On the Dataworkbench version, we have added rules to check if an activity identifier is "known", defined as: it has either been published already, or it is in the current dataset. We also have a rule that warns when an activity identifier is declared twice in the same file (an issue we encountered several times some 5 years ago).
• It would be good to separate out "inter-activity business rules", and it would require an additional layer to validate those.
• Minor: I think using the sha1 of the XML serialisation is good enough to optimise processing, but a sha1 is prone to any change in the serialisation, even if it has no effect on the content (e.g. whitespace, ordering of ellements and attributes, reformatting). In the world of XML Signatures, there is a canonicalisation step to make a signature independent of such changes.

[siemvaessen]

Thanks @rolfkleef ~ we now have a branch that will check for data changes on activity level, while keeping the current dataset level change sha1 concept which ties into the Validator intact. Further down the line the Validator should follow this concept, so data is processed in an optimal way.