Defining optional fields to resources or a way to ignore additional fields in data?

[pchtsp]

Hello again. I have a question regarding the checks that are done to the data in DataPackages.

Again, we're migrating from jsonschema (and /or marshmallow) and so concepts may not apply or not mean the same thing.

In order to guarantee backwards-compatibility for our schemas, we make use of non-mandatory fields in our data. For example, with the following frictionless schema:

{'name': 'dataschema',
 'resources': [
 // (other resources not shown)
               {'name': 'resources',
                'schema': {'fields': [{'constraints': {'required': True},
                                       'name': 'available',
                                       'type': 'number'},
                                      {'constraints': {'required': True},
                                       'name': 'id',
                                       'type': 'string'},
                                      {'name': 'type', 'type': 'string'}]}}]}

Note the "type" field does not have a "required" constraint. This (for us) implies that the json data can include or not the type attribute in each object of the "resources" table (i.e., an optional field). For example this data should comply:

{
// (more data not shown)
 'resources': [{'available': 21, 'id': 'R 1'},
               {'available': 24, 'id': 'R 2'},
               {'available': 99, 'id': 'N 1'},
               {'available': 106, 'id': 'N 2'}]
}

But we get from frictionless one "missing-label" error and four "extra-cell" errors:

{
// more info for missing-label
  'message': 'There is a missing label in the header\'s field "type" at '
             'position "3"',
  'description': 'Based on the schema there should be a label that is missing '
                 "in the data's header."},

{
// more info for extra-cell
  'message': 'Row at position "2" has a missing cell in field "type" at '
             'position "3"',
  'description': 'This row has less values compared to the header row (the '
                 'first row in the data source). A key concept is that all the '
                 'rows in tabular data must have the same number of columns.'},

Looking for a way to pass some scopes argument to the validate_package function I reached this page: https://framework.frictionlessdata.io/docs/guides/validation-checks that says:

The Baseline Check is always enabled. It makes various small checks that reveal a great deal of tabular errors. There is a report.tasks[].scope property to check which exact errors have been checked for.

And both previous errors are part of the "Baseline Check" so I guess I cannot deactivate them.

So I have three questions:

1. Is it possible to have optional fields for a resource that are accepted but not checked?
2. Is it possible for the validator to ignore additional fields that are not part of the schema (i.e., only check for missing data)? jsonschema does this by default, marshmallow can be configured to accept unknown fields.
3. Can we choose which Baseline Checks we want to enforce to our data?

I understand this are more "json" questions than "csv" ones. But the backwards-compatibility reasons should be relevant for all cases, I believe.

So now I'm wondering if it's even possible to have missing fields in json objects with respect to the schema. I understand that maybe in csv this is harder to see (because the structure is more rigid in terms of number of columns) but I thought that maybe in json data it would be

[lwinfree]

Hi @pchtsp! I'll answer the bits I can here:

You can skip checks (including baseline) like this:

report2 = validate("capital-invalid.csv", skip_errors=["duplicate-label"])

(https://framework.frictionlessdata.io/docs/guides/validation-guide#pickskip-errors) You should be able to find a list of possible errors to skip in the validate checks guide you link in your question.

Similarly, you can write custom checks: https://framework.frictionlessdata.io/docs/guides/validation-guide#custom-checks (ps - please let us know if these docs don't help so we can improve them :-) )

Frictionless specs definitely allow for optional (or not-required) fields. Do my above links help you with this question though? Please let us know if this doesn't help or if you have other questions!

Also @roll might be able to explain this in more detail.

[roll]

I think just ignoring errors will not fully help the case.

The problem here is that we need to provide detector.schema_sync as a part of the package's resource descriptor which is not supported by the Data Package/Resource Spec.

The Inquiry might be used - https://framework.frictionlessdata.io/docs/guides/framework/inquiry-guide - but it will not handle foreign keys

[pchtsp]

Thanks @lwinfree and @roll. I tried the skip_errors argument and I can confirm it correctly answers my third question. I'm not sure about the other two, though.

As @roll said, deactivating errors may be a little too radical since there are real errors that we may stop detecting. I'm not familiar with the Inquiry / Detector classes but... Would I be able to combine them to get a validation with: foreign keys, optional fields, ignoring non-schema fields? Even if it's in more than one step?

Thanks again to both.

[pchtsp]

I went and read https://framework.frictionlessdata.io/docs/guides/framework/detector-guide#schema-sync and tried something like the following:

def _validate_package(input, schema):
    package = Package(schema)
    for name, data in input.items():
        # Here we link out template's resource with actual data
        resource = package.get_resource(name)
        resource.data = data
        resource.detector = Detector(schema_sync=True)
    return validate_package(package)

validation = _validate_package(data, schema)

(I had to modify the frictionless/resource.py a little bit so it let me edit the detector property). But I still was not able to suppress the errors.

[roll]

@pchtsp
That's a great idea! I'll try to make it work

[roll]

@pchtsp
Your idea was very smart and only the bug was preventing it from working. With [email protected]:

from pprint import pprint
from frictionless import Package, Resource, Schema, Field, Detector, validate

package = Package(
    resources=[
        Resource(
            data=[["f1"], ["v1"], ["v2"], ["v3"]],
            schema=Schema(fields=[Field(name="f1"), Field(name="f2")]),
        ),
    ]
)

pprint(validate(package).valid)  # False


def validate_with_optinal_fields(package):
    for resource in package.resources:
        resource.detector = Detector(schema_sync=True)
    return validate(package)


pprint(validate_with_optinal_fields(package).valid)  # True

The function, of course, might be combined with #675