Discuss how to handle Onroerend Erfgoed data sets accessed through json API

[maarten-vermeyen]

For the simple features datasets, we do very little processing on the end result on top of st_read(). When accessing datasets from non geographic authentic datasources this is more difficult. The API's often return complex json data structures. Within the context of roeference, we might want to simplify this datastructure as much as possible. For example:

df <- request('https://besluiten.onroerenderfgoed.be/besluiten') |>
        req_headers(Accept = 'application/json') |>
        req_url_query(per_pagina=50) |>
        req_throttle(30/60) |>
        req_perform_iterative(iterate_with_link_url(rel = "next"), max_reqs = 1000) |>
        resps_data(\(resp) resp_body_json(resp, simplifyDataFrame=TRUE)) 

Would return a simplified dataframe. Each besluit is a row. Some rows may contain vectors or (nested) lists for certain variables. While this might be ideal for some use cases, it offers little control to the end user. However, more complex use cases where the user might want to parse json themselves might be out of scope of roeference.

In addition, we need to determine sensible throttle and per_page values to avoid performance issues on the server end.