Machine testing of freshness (serviceLevels)

[john-bunge-ed]

Question:

Is the freshness parameter in the serviceLevels object supposed to be machine testable?

I applied test() to my datacontract object (see below), however no automated test results on freshness could be retrieved from the .checks or .logs elements.

datacontract_obj = DataContract(
    data_contract_file='datacontract.yaml',
    spark=spark)
datacontract_run = datacontract_obj.test()

datacontract_run.checks

datacontract_run.logs

freshness was specified in the datacontract.yaml as follows. Column timestamp_col is of timestamp format.

servicelevels:
  freshness:
    description: The maximum age of the youngest entry is 1 hour
    threshold: 1h
    timestampField: table_name.timestamp_col

[jochenchrist]

Hi @john-bunge-ed,
this is currently not yet implemented, but would totally make sense.
Current workaround would be to add the freshness-test as additional quality test.

[john-bunge-ed]

@jochenchrist
Where in the codebase would you suggest to add such an additional quality test?

[jochenchrist]

next to this:
https://github.com/datacontract/datacontract-cli/blob/main/datacontract/export/sodacl_converter.py#L15

[john-bunge-ed]

thanks @jochenchrist

[jochenchrist]

closed as duplicate to #407