Endpoint Structure: GeoJSON or JSON

[schnuerle]

Should endpoints be GeoJSON format with CDS details in the ‘properties’, or more freely structured JSON to align with MDS's design language with GeoJSON objects in them when needed?

Discussion from doc:

Michael Schnuerle

I wonder if GeoJSON is too limiting for what we want may out of an API? Object hierarchy, metadata, flexible structure. Is the only benefit that it's easily visualize using out of the box tools?

Marie Maxham

That's a pretty big benefit :)

Michael Schnuerle

Well another way it can be done is have the geoJSON be a value within the larger JSON structure. So you can still get to the valid GeoJSON easily, but it’s not the entire file structure.

Jacob Malleau

I agree with using JSON structure - get the flexibility but can still extract geoJSON.

Eric Mai

I think a strictly typed JSON object (with a GeoJSON property) would be easier and clearer to define than a special kind of GeoJSON object (because it would be GeoJSON but with specific expected metadata properties).

Jacob Baskin

GeoJSON is pretty permissive -- what we are doing below is defining all of the objects to be GeoJSON features, but then putting the "meat" of the object inside "properties" which is 100% permitted within the GeoJSON spec. I don't find this so constraining and doing it any other way means that you can't load the API responses into GeoJSON-supporting tools without some coding/ETL which seems pretty undesirable.

[jamesianbarr]

Agree with the folks suggesting using GeoJSON. Regarding the thread on separating regulation from location, could we consider using GeoJSON for spatial-related endpoints, and JSON for non-spatial endpoints?

[schnuerle]

Since the main endpoints of Area, Zone, and Space all have geospacial components, this would apply to most of it. I think we should pick one or the other. Even for Geographies defined in MDS, we still use a GeoJSON object as part of a JSON endpoint.

[drewda]

I'm with @interline-io where we're watching the development of CDS and considering aggregating public sources in the future. For our use-cases there is no material difference between JSON and GeoJSON. However, based on past experience with open transportation and geo tooling and datasets, I would recommend GeoJSON in this case. As others mentioned above, there is value in being able to point a novice to geojson.io or the equivalent to do basic debugging or editing of a data payload, with no transformation required.

It's true that feature properties can turn into dumping grounds in GeoJSON files, so if you all do go with GeoJSON, do consider also having a complete JSON Schema for validating and to serve as precise documentation for advanced producers/consumers.

[schnuerle]

Something to consider with a GeoJSON format approach, is that we may have some endpoints that have no geospatial components, but are referenced by ID from some that do (see this conversation). So not all the data will be in the properties area, and some key things may be external anyway.

How do the best visualization tools for GeoJSON handle complex arrays and objects stored in the properties? Do they spit all of it out in one big hierarchical blob, or is there some parsing/interactivity too?

To your other point, we will have a CDS schema to validate against eventually, just like we do with MDS.

[drewda]

Good question. I don't think most GeoJSON tools would work that well for drilling down into data associated across multiple endpoints/files, or for creating data that's linked across multiple endpoints/files. (That probably needs purpose-built UIs.) However, I think they would be sufficient for basic viewing, adding a new geometry when you know its associated ID, editing an existing geometry, or using it together with some other GIS datasets.

Here's how geojson.io handles arrays and objects in a contrived example. I'd say it's fine for seeing the contents of properties, perhaps adding or editing individual IDs. Not a great experience to edit in bulk. But at least it's not going to clobber/destroy nested contents of properties.

A similar experience in QGIS -- not great, but fine:

I don't have ArcGIS desktop or online at hand. They treat GeoJSON as a bit of a second-class data format, but I believe all the tooling is still able to handle basic JavaScript entities in properties coming in and out on GeoJSON.

[schnuerle]

I'm not sold on the benefits of GeoJSON for this. Here are some reasons.

Yes you can visualize the geography out of the box, and allow a tool like ArcGIS Online to serve them. But the geography is not the most interesting part of CDS Curb Areas/Zones/Spaces. And the geography likely won't change much. Plus without parsing the properties and linking to other external objects by ID, you really can't style or make sense of the what the geographies are telling you anyway.

It is another step, but having a GeoJSON object farther down the JSON hierarchy doesn't make it too hard to extract this or embed it into JSON (either via programming or by hand).

Serving a static Zone endpoint from ArcGIS is likely to be less common that serving a dynamic result to an endpoint consumer. So that lessens the need for the format to be GeoJSON for the host.

Lastly, there are still very good tools to parse and validate JSON that are just as useful for validating the GeoJSON (minus the geography part) and viewing its tree structure.

I don't see these endpoints as being geographic objects first and foremost. We may have other endpoints that have no geographic data (eg, /regulations). And the other CDS APIs (Events and Metrics) won't be geographic anyway so they will likely just be JSON, and MDS is all JSON with GeoJSON inside when needed.

[drewda]

Perhaps you or others involved in the project can speak to the type of user that is likely to provide (and to maintain) the geometries for the curb areas/zones/spaces?

If it's entry-level users using full-serve custom UIs or if it's advanced developers who are familiar with all these data formats, then using GeoJSON as the root document format for curb areas/zones/spaces is neither here nor there.

If it's a staffer in a GIS office or another type of analyst with intermediate-level technical skills, they will likely appreciate being able to view and edit those in GeoJSON. I would wager that if given the choice between easily seeing the geometries and being able to drill-down into data across associated IDs, they would pick the former.

Some GIS offices/depts are also used to insisting that "source of truth" for all geographic data across the wider org live within their systems. Using GeoJSON for those endpoints may provide some cities ways to make different departments feel as if they own different portions of the data (whether or not that data is literally sourced live from a GIS).

I'm extrapolating based on somewhat, but not exactly similar, projects with cities — probably better to hear from real, potential users.

[delkinsmpls]

From Mpls team: as it relates to GeoJSON vs JSON is that I think it would be good to try and be consistent with MDS standards where possible. I would imagine that many agencies or companies might be using both APIs, and from a development perspective it makes sense to not have discrepancies like that for pushing and pulling data. So if GeoJSON is the preferred approach, it might be good to migrate MDS to that as well eventually (or vice versa).

[schnuerle]

For what it's worth, I don't think MDS will ever be migrated to GeoJSON, given how a lot of it (especially served by agencies) doesn't contain geospatial data, and there isn't really a need to inherently visualize the MDS data geospatially given its nested complexity and modular interconnectedness.

[avatarneil]

A little late to the party (sorry!), but I think there are pros and cons with both. From a tooling perspective, I think that GeoJSON is the fairly obvious winner (note: all of the discussion above!).

That being said, I think that from a design perspective (and also implementation perspective), having GeoJSON be the only data format is a bit limiting. In my experience, in order to maintain high performance, most of those properties need to be pulled out at the persistence layer and added as additional columns; this means that in order to serve this data back to clients, an implementation will almost always have to be doing a healthy amount of transforms before returning data to the client. This is fine some of the time (when you have humans pulling data to put in a GIS tool), but what if you have machine consumers that are pulling from the API, and need to do the exact same thing when they persist data from CDS? I think that it adds a lot of overhead round-trip for almost any cases where you're not pulling CDS data in order to jam it into a GIS tool.

I almost want to propose that we have both: a not-embedded-in-GeoJSON version (the main source-of-truth in the spec), as well as the option to query for the GeoJSON-ified version (where we just jam everything into properties). Though there would be some added complexity, I think this would allow CDS to provide the best of both worlds; performance when serving machine-to-machine clients, as well as users fetching data to use in a GIS tool.

TLDR; instead of just having:

// where between <> is the expected properties
SomeCDSType = GeoJSON<{ propertyFoo, propertyBar }>

We could have:

SomeCDSType = { propertyFoo, propertyBar, geoJSON }

// only returned if querying for `asGeoJSON` or something along those lines
PureGeoJSONVersion = someCDSType.geoJSON<{ someCDSType.propertyFoo, someCDSType.propertyBar }>

Thinking even a bit further, I could totally imagine a library being maintained by the OMF purely to do these transforms to make adoption easier! Do a little survey to see what languages people are using, and then rally together some folks to make transformers for the top 3 :)

[schnuerle]

I'm not sure there is enough value to do both, especially for this beta v1.0 that needs some real world testing. We should pick one for now and run with it. If both were supported, it seems it would either be a burden on the city (eg, having to serve up both versions for every endpoint, and the consumer chooses) or a burden for the consumer (the city chooses which one to serve, but the consumer has to support both depending on the city).

Another thing to consider is that these can be a dynamic endpoints with parameters, so the scenario of a city extracting/serving a static endpoint from an ArcGIS system will likely be less common.

Also the other APIs, Events, and Metrics, will be JSON, which means this Curb (Regulations) part of CDS is the only one using GeoJSON.

The ability to extract GeoJSON from or insert GeoJSON into JSON it also not too much of a burden, and that's what we do in MDS.