Trying to get WB WDI working

[nicolas-graves]

One more issue ;)

I'm also trying to use the WB WDI database, which should in principle have enough to be working, but is not. I'm currently working on the issue, this is to record my progress and get some feedback.

modified   sdmx/source/wb_wdi.py
@@ -11,8 +11,10 @@ class Source(BaseSource):
         if kwargs["resource_type"] == "categoryscheme":
             # Service does not respond to requests for "WB" category schemes
             kwargs["provider"] = "all"
-        elif kwargs["resource_type"] != "data":
+        elif kwargs["resource_type"] == "data":
             # Provider's own ID differs from its ID in this package
+            kwargs["provider"] = "WDI"
+        elif kwargs["resource_type"] != "dataflow":
             kwargs.setdefault("provider", "WB")

Now it's currently failing with

import sdmx
wdi = sdmx.Client("WB_WDI")
wdi.dataflow("A.SP_POP_TOTL.AFG", force=True)

Traceback (most recent call last):
  File "/home/graves/projects/src/sdmx/scratch.py", line 10, in <module>
    wdi.dataflow("A.SP_POP_TOTL.AFG", force=True)
  File "/home/graves/projects/src/sdmx/sdmx/client.py", line 465, in get
    response.raise_for_status()
  File "/home/graves/.guix-profile/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://dataapi.worldbank.org/v2/sdmx/rest/dataflow/WB_WDI/A.SP_POP_TOTL.AFG/latest/?references=all

Almost seems the right url, but I guess my current issue is that an involuntary / finds its way in the original url.

How I would have tackled that : define a url kwarg in modify_request_args that will override the one passed by default. Maybe there's also a way to avoid that additional / when the key kwarg is None or non-existent?

[khaeru]

One quick check: I see http://, but this has been updated on main; see commit 7dc758e

Does using this fix the error?

[nicolas-graves]

I'll check but I'm pretty sure that yes, it seems this is the redirect operated after the first request (dataapi prefix instead of api).

[nicolas-graves]

I can confirm it, my patch on top of current main HEAD. I can also confirm that the url.join at the end Client._request_from_args computes to https://api.worldbank.org/v2/sdmx/rest/dataflow/WB_WDI/A.SP_POP_TOTL.AFG/latest (without the patch: https://api.worldbank.org/v2/sdmx/rest/dataflow/WB/A.SP_POP_TOTL.AFG/latest).

Instead the example in https://datahelpdesk.worldbank.org/knowledgebase/articles/1886701-sdmx-api-queries is https://api.worldbank.org/v2/sdmx/rest/data/WDI/A.SP_POP_TOTL.AFG/?startperiod=2011&endPeriod=2011

[khaeru]

In these, A.SP_POP_TOTL.AFG looks an awful lot like a key (values for certain dimensions). Keys are valid parts of /data/… requests (to get a DataMessage, per the example you give from the WB documentation), but for a …/dataflow/ request (to get a StructureMessage with the DataflowDefinition) they are not relevant. One should use instead the ID of the given data flow.

[nicolas-graves]

Will work from now on in https://github.com/CIRED/sdmx/tree/wdi and submit a PR when things are working on my end.

[nicolas-graves]

#208