feat(datasets): Replace geopandas.GeoJSONDataset with geopandas.GenericDataset

[harm-matthias-harms]

Description

Geopandas supports reading parquet and feather files since version 0.8.0. These offer performance improvements over classical file types, such as .geojson or .shp.zip (supported by geopandas.GeoJSONDataset). We have used a private implementation for some time, but want to contribute them to kedro-datasets. This PR closes #196.

Development notes

Extended the implementation of geopandas.GeoJSONDataset with an optional file_formatparameter which is needed to use custom read_* and to_* methods. Replaced the geopandas.GeoJSONDataset with geopandas.GenericDataset.

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes

[harm-matthias-harms]

If the implementation is okay, I can add the feather dataset in the same PR or open a new one.

[astrojuanlu]

Thanks for this contribution @harm-matthias-harms ! One question: do you think we could have some sort of geopandas.GenericDataset, like we do for pandas and Polars? (Just so that we avoid proliferation of datasets)

[harm-matthias-harms]

Thanks for this contribution @harm-matthias-harms ! One question: do you think we could have some sort of geopandas.GenericDataset, like we do for pandas and Polars? (Just so that we avoid proliferation of datasets)

To be honest, that would be my preferred solution as well. The current approach feels like a lot of duplicated boilerplate. I really like the Polars dataset and will try to modify the GeoJSON dataset tomorrow to make it work more generally.

[harm-matthias-harms]

I think it's fine to set a default value, especially because this ensures backward compatibility

[harm-matthias-harms]

We don't need this anymore because of backward compatibility, but it's more or less the generic method of geopandas, and it saves some boilerplate in most cases.

[astrojuanlu]

Thanks a lot @harm-matthias-harms! This looks like a step in the right direction.

My worry is that GeoJSONDataset is no longer GeoJSON-exclusive. So I'm wondering if we should more or less take the code you created here, but put it in a geopandas.GenericGeoDataset, and leave geopandas.GeoJSONDataset alone.

[harm-matthias-harms]

My worry is that GeoJSONDataset is no longer GeoJSON-exclusive. So I'm wondering if we should more or less take the code you created here, but put it in a geopandas.GenericGeoDataset, and leave geopandas.GeoJSONDataset alone.

@astrojuanlu I see two options:

1. GeoJSON is also not GeoJSON exclusive today because I can read many more formats than geojson with read_file.
2. We create a GenericDataset (this fixes the naming and offers the same functionality and more ) and deprecate the GeoJSONDataset to remove it in some future major versions. Then we don't have to maintain the same dataset twice in the future.

Please tell me how you would like to proceed here.

[astrojuanlu]

GeoJSON is also not GeoJSON exclusive today because I can read many more formats than geojson with read_file.

Hah, true... I see this wasn't discussed in kedro-org/kedro#190 originally.

I'm actually in favour of having a new dataset with a more generic name and deprecate the old one, but wondering if the churn is worth the advantages. Any thoughts @merelcht @noklam ?

[merelcht]

I'm actually in favour of having a new dataset with a more generic name and deprecate the old one, but wondering if the churn is worth the advantages. Any thoughts @merelcht @noklam ?

We can just remove the old one and add a new one with a generic name in a breaking release. I don't think it's worth the effort of doing a non-breaking release with a deprecation warning first and then remove the old one in the next breaking release just for this one dataset.

We could do a TSC vote if we feel the nature of the dataset changes too much.

[astrojuanlu]

Let's proceed that way then 👍🏼 @harm-matthias-harms are you willing to update the PR accordingly? Namely:

just remove the old one and add a new one with a generic name in a breaking release.

[harm-matthias-harms]

@astrojuanlu I updated the PR. Since I merged main the tests have been failing. It looks like a fsspec problem. I don't know much about ffspec, maybe you have a good idea of how to fix this.

Last successful run: https://github.com/kedro-org/kedro-plugins/actions/runs/10593320929/job/29354563096?pr=812
First run that failed: https://github.com/kedro-org/kedro-plugins/actions/runs/10719091250/job/29722424013?pr=812

[astrojuanlu]

I do not know.

Does geopandas handle remote filepaths? Like, if we do geopandas.read_{self._file_format}("s3://...") directly without

https://github.com/harm-matthias-harms/kedro-plugins/blob/9e88c5ee0284865b1256a5667bd6f7b691ddb3f7/kedro-datasets/kedro_datasets/geopandas/generic_dataset.py#L155-L156

will it work?

[harm-matthias-harms]

@astrojuanlu This was also a thought I had, but that currently makes more tests fail, and it may be better to have all datasets similar. I don't know why the CI fails.

pytest kedro_datasets/geopandas/generic_dataset.py --doctest-modules --doctest-continue-on-failure and pytest tests/geopandas/test_generic_dataset.py both pass locally with python 3.12. It looks like I have to investigate the CI a bit more. I hope I find some time at the end of the week for that.

[astrojuanlu]

Thanks for your patience! Let us know how it goes

[merelcht]

@astrojuanlu This was also a thought I had, but that currently makes more tests fail, and it may be better to have all datasets similar. I don't know why the CI fails.

pytest kedro_datasets/geopandas/generic_dataset.py --doctest-modules --doctest-continue-on-failure and pytest tests/geopandas/test_generic_dataset.py both pass locally with python 3.12. It looks like I have to investigate the CI a bit more. I hope I find some time at the end of the week for that.

It has to do with the docstring part and seems to affect python 3.10 specifically. I'm able to replicate the error locally, but haven't found what's wrong yet.

[harm-matthias-harms]

@astrojuanlu I figured out the problem... I also bumped geopandas to v1. They use pyogrio as the new default over fiona. pyogrio does not yet support an alternate file system like fsspec (geopandas/pyogrio#430). There are workarounds using buffers, but the simplest way would be to change the used engine to fiona until pyogrio supports fsspec.