Pydantic schemas for validated and typed search results?

[andretheronsa]

When dealing with API I find Pydantic classes really useful. You can guarantee that the data you get back will be the type and values you expect, and if not you can simple raise predictable errors which makes it very clear what has changed in the API. Furthermore the type hints and class attributes make it really easy and clean to write code in a modern IDE.

Eodag works really well for my current use case, keeping track of Sentinel 1 acquisitions, since it abstracts the API away really efficiently, but this means that the data we get back can have very unexpected metadata issues. For example, last week something changed in Creodias, S1 dual pol values went from space seperated to ampersand seperated. There are also some products that don't have polarizationChannels keys at all. It is dissapointing that the DIASs don't have standard schemas but here we are...

With Pydantic and Sentry we could quickly identify and fix this issue. We dump the results of search all into product classes that define for example a Sentinel 1 product. This is actually 3 classes, matching and serialising to GeoJSON. So a base class that has a geo accessor. And a properties and geometry attributes. "Properties" is another class which represents the attributes and their values. Here we can add validation of what is expected and required and optional and their types. "Geometry" grabs the footprint and stores this as a shapely geometry. This gives us a way of operationalizing Eodag much more easily with more reliable code and better exceptions. After all, we don't just want a search result, we depend on the schemas.

Ideally, we need the same schema across DIASs per product. This is challenging as the different DIASs have decided to use different schemas for the same product for some reason.

I was wondering if there is scope to add and work on reliable Pydantic product definitions to Eodag? I could start this off with a PR for Sentinel-1. We could let searches return this instead of the current result, or improve the current result with this. Or as a plugin to convert the results into?

I assume most people are doing some kind of validation as the step after Eodag in any case.

Issues with this would be in deciding what makes sense to be required and optional fields for all use cases. For S1 for example we normally always need to know the polarisation. But this may not be true.

[sbrunato]

Hello @andretheronsa and thanks for the suggestion.

This could be implemented for EOProduct class that could inherit from Pydantic.BaseModel and then have a validator that checks some standard properties formatting.

I think this check should be optional, using a parameter like validate_metadata (default=False) in methods search, search_all, and EOProduct class init.

Would this match your needs ?

Also, a PR would be welcome for this feature 👍