Skip to content

Commit

Permalink
📝 Switch from the requests lib to httpx
Browse files Browse the repository at this point in the history 
The requests lib feels dead. See also
https://blog.ian.stapletoncordas.co/2024/02/a-retrospective-on-requests
  • Loading branch information
@veit
veit committed Feb 27, 2024
1 parent 5e50ecb commit b99a8aa
Show file tree
Hide file tree
Showing 11 changed files with 119 additions and 187 deletions.
2 changes: 1 addition & 1 deletion docs/data-processing/apis/fastapi/install.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Pydantic can use the optional dependencies

Starlette can use the optional dependencies

:doc:`requests <../../requests/index>`
:doc:`httpx <../../httpx/index>`
if you want to use the ``TestClient``.
`aiofiles <https://github.com/Tinche/aiofiles>`_
if you want to use ``FileResponse`` or ``StaticFiles``.
Expand Down
0 ...-processing/requests/module.ipynb.license → ...data-processing/httpx/httpx.ipynb.license
View file
File renamed without changes.
17 changes: 17 additions & 0 deletions docs/data-processing/httpx/index.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
.. SPDX-FileCopyrightText: 2021 Veit Schiele
..
.. SPDX-License-Identifier: BSD-3-Clause
httpx
=====

`httpx <https://www.python-httpx.org>`_ is an http client with which requests
can be sent easily.

.. toctree::
:hidden:
:titlesonly:
:maxdepth: 0

install-example.ipynb
module.ipynb
205 changes: 85 additions & 120 deletions docs/data-processing/requests/requests.ipynb → ...ta-processing/httpx/install-example.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Requests installation and sample application"
"# httpx installation and sample application"
]
},
{
Expand All @@ -13,17 +13,17 @@
"source": [
"## Installation\n",
"\n",
"The requests library is useful for communicating with REST APIs. With [Spack](../../productive/envs/spack/index.rst) you can provide requests in your kernel:\n",
"The httpx library is useful for communicating with REST APIs. With [Spack](../../productive/envs/spack/index.rst) you can provide httpx in your kernel:\n",
"\n",
"``` bash\n",
"$ spack env activate python-311\n",
"$ spack install py-requests\n",
"$ spack install py-httpx\n",
"```\n",
"\n",
"Alternatively, you can install requests with other package managers, for example\n",
"Alternatively, you can install httpx with other package managers, for example\n",
"\n",
"``` bash\n",
"$ pipenv install requests\n",
"$ pipenv install httpx\n",
"```"
]
},
Expand All @@ -35,7 +35,7 @@
"\n",
"In this example we get our data from the [OpenStreetMap Nominatim API](https://nominatim.org/release-docs/develop/api/Overview/#nominatim-api). This can be reached via the URL `https://nominatim.openstreetmap.org/search?`. To e.g. receive information about the Berlin Congress Center in Berlin in JSON format, the URL `https://nominatim.openstreetmap.org/search.php?q=Alexanderplatz+Berlin&format=json` should be given, and if you want to display the corresponding map section you just have to leave out `&format=json`.\n",
"\n",
"Then we define the base URL and the parameters. Nominatim expects at least the following two parameters\n",
"Then we define the search URL and the parameters. Nominatim expects at least the following two parameters\n",
"\n",
"| Key | Value |\n",
"| --------- | ------------------------------------ |\n",
Expand All @@ -51,15 +51,15 @@
"metadata": {},
"outputs": [],
"source": [
"import requests\n",
"import httpx\n",
"\n",
"\n",
"base_url = \"https://nominatim.openstreetmap.org/search?\"\n",
"search_url = \"https://nominatim.openstreetmap.org/search?\"\n",
"params = {\n",
" \"q\": \"Alexanderplatz, Berlin\",\n",
" \"format\": \"json\",\n",
"}\n",
"r = requests.get(base_url, params=params)"
"r = httpx.get(search_url, params=params)"
]
},
{
Expand Down Expand Up @@ -92,75 +92,62 @@
{
"data": {
"text/plain": [
"[{'place_id': 261767431,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
"[{'place_id': 128497332,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 783052052,\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
" 'lat': '52.5219814',\n",
" 'lon': '13.413635717448294',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'place',\n",
" 'type': 'square',\n",
" 'importance': 0.6914982526373583},\n",
" {'place_id': 45802928,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
" 'place_rank': 25,\n",
" 'importance': 0.47149825263735834,\n",
" 'addresstype': 'square',\n",
" 'name': 'Alexanderplatz',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']},\n",
" {'place_id': 128243381,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'node',\n",
" 'osm_id': 3908141014,\n",
" 'boundingbox': ['52.5165661', '52.5265661', '13.4062804', '13.4162804'],\n",
" 'lat': '52.5215661',\n",
" 'lon': '13.4112804',\n",
" 'display_name': 'Alexanderplatz, Dircksenstraße, Mitte, Berlin, 10179, Deutschland',\n",
" 'class': 'railway',\n",
" 'type': 'station',\n",
" 'importance': 0.6560990777880803,\n",
" 'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_train_station2.p.20.png'},\n",
" {'place_id': 190976103,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
" 'place_rank': 30,\n",
" 'importance': 0.43609907778808027,\n",
" 'addresstype': 'railway',\n",
" 'name': 'Alexanderplatz',\n",
" 'display_name': 'Alexanderplatz, Dircksenstraße, Mitte, Berlin, 10179, Deutschland',\n",
" 'boundingbox': ['52.5165661', '52.5265661', '13.4062804', '13.4162804']},\n",
" {'place_id': 128416772,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 346206374,\n",
" 'boundingbox': ['52.5216214', '52.5216661', '13.4131913', '13.4131914'],\n",
" 'lat': '52.5216214',\n",
" 'lon': '13.4131913',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'highway',\n",
" 'type': 'pedestrian',\n",
" 'importance': 0.32000999999999996},\n",
" {'place_id': 51760167,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
" 'osm_type': 'node',\n",
" 'osm_id': 4598414751,\n",
" 'boundingbox': ['52.5165849', '52.5265849', '13.4089082', '13.4189082'],\n",
" 'lat': '52.5215849',\n",
" 'lon': '13.4139082',\n",
" 'place_rank': 26,\n",
" 'importance': 0.10000999999999993,\n",
" 'addresstype': 'road',\n",
" 'name': 'Alexanderplatz',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'railway',\n",
" 'type': 'station',\n",
" 'importance': 0.22000999999999998,\n",
" 'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_train_station2.p.20.png'},\n",
" {'place_id': 49544606,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
" 'osm_type': 'node',\n",
" 'osm_id': 4389211800,\n",
" 'boundingbox': ['52.5231653', '52.5232653', '13.414475', '13.414575'],\n",
" 'lat': '52.5232153',\n",
" 'lon': '13.414525',\n",
" 'display_name': 'Alexanderplatz, Alexanderstraße, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'highway',\n",
" 'type': 'bus_stop',\n",
" 'importance': 0.22000999999999998,\n",
" 'icon': 'https://nominatim.openstreetmap.org/ui/mapicons/transport_bus_stop2.p.20.png'},\n",
" {'place_id': 181190866,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
" 'boundingbox': ['52.5216214', '52.5216661', '13.4131913', '13.4131914']},\n",
" {'place_id': 127680907,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 301241483,\n",
" 'boundingbox': ['52.5226699', '52.5227698', '13.4152008', '13.4154146'],\n",
" 'lat': '52.5227318',\n",
" 'lon': '13.4152821',\n",
" 'display_name': 'Alexanderstraße, Mitte, Berlin, 10178, Deutschland',\n",
" 'osm_id': 301733776,\n",
" 'lat': '52.5222454',\n",
" 'lon': '13.4158136',\n",
" 'class': 'highway',\n",
" 'type': 'primary',\n",
" 'importance': 0.21000999999999995}]"
" 'place_rank': 26,\n",
" 'importance': 0.10000999999999993,\n",
" 'addresstype': 'road',\n",
" 'name': 'Alexanderstraße',\n",
" 'display_name': 'Alexanderstraße, Mitte, Berlin, 10178, Deutschland',\n",
" 'boundingbox': ['52.5222454', '52.5224356', '13.4153983', '13.4158136']}]"
]
},
"execution_count": 3,
Expand All @@ -187,17 +174,20 @@
{
"data": {
"text/plain": [
"[{'place_id': 261767431,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
"[{'place_id': 128497332,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 783052052,\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
" 'lat': '52.5219814',\n",
" 'lon': '13.413635717448294',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'place',\n",
" 'type': 'square',\n",
" 'importance': 0.6914982526373583}]"
" 'place_rank': 25,\n",
" 'importance': 0.47149825263735834,\n",
" 'addresstype': 'square',\n",
" 'name': 'Alexanderplatz',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']}]"
]
},
"execution_count": 4,
Expand All @@ -207,7 +197,7 @@
],
"source": [
"params = {\"q\": \"Alexanderplatz, Berlin\", \"format\": \"json\", \"limit\": \"1\"}\n",
"r = requests.get(base_url, params=params)\n",
"r = httpx.get(search_url, params=params)\n",
"r.json()"
]
},
Expand All @@ -229,14 +219,25 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"To ensure that the interaction was successful, we use the `raise_for_status` method of `requests`, which throws an exception if the HTTP status code isn’t `200 OK`:"
"To ensure that the interaction was successful, we use the `raise_for_status` method of `httpx`, which throws an exception if the HTTP status code isn’t `200 OK`:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"<Response [200 OK]>"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"r.raise_for_status()"
]
Expand All @@ -245,7 +246,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Since we don't want to exceed the load limits of the Nominatim API, we will delay our requests with the `time.sleep` function:"
"Since we don't want to exceed the load limits of the Nominatim API, we will delay our httpx with the `time.sleep` function:"
]
},
{
Expand All @@ -256,17 +257,20 @@
{
"data": {
"text/plain": [
"[{'place_id': 261767431,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
"[{'place_id': 128497332,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 783052052,\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
" 'lat': '52.5219814',\n",
" 'lon': '13.413635717448294',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'place',\n",
" 'type': 'square',\n",
" 'importance': 0.6914982526373583}]"
" 'place_rank': 25,\n",
" 'importance': 0.47149825263735834,\n",
" 'addresstype': 'square',\n",
" 'name': 'Alexanderplatz',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']}]"
]
},
"execution_count": 6,
Expand Down Expand Up @@ -302,7 +306,7 @@
" \"\"\"\n",
" search_url = \"https://nominatim.openstreetmap.org/search?\"\n",
" params = {\"q\": address, \"format\": format, \"limit\": limit, **kwargs}\n",
" r = requests.get(search_url, params=params)\n",
" r = httpx.get(search_url, params=params)\n",
" # Raise an exception if the status is unsuccessful\n",
" r.raise_for_status()\n",
"\n",
Expand All @@ -325,17 +329,20 @@
{
"data": {
"text/plain": [
"[{'place_id': 261767431,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
"[{'place_id': 128497332,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. http://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 783052052,\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801'],\n",
" 'lat': '52.5219814',\n",
" 'lon': '13.413635717448294',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'class': 'place',\n",
" 'type': 'square',\n",
" 'importance': 0.6914982526373583}]"
" 'place_rank': 25,\n",
" 'importance': 0.47149825263735834,\n",
" 'addresstype': 'square',\n",
" 'name': 'Alexanderplatz',\n",
" 'display_name': 'Alexanderplatz, Mitte, Berlin, 10178, Deutschland',\n",
" 'boundingbox': ['52.5201457', '52.5238113', '13.4103097', '13.4160801']}]"
]
},
"execution_count": 8,
Expand All @@ -347,48 +354,6 @@
"nominatim_search(\"Alexanderplatz, Berlin\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, you can use other parameters besides `address`. You can get an overview in the [Nominatim Docs](https://nominatim.org/release-docs/develop/api/Search/#parameters)."
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'place_id': 194632110,\n",
" 'licence': 'Data © OpenStreetMap contributors, ODbL 1.0. https://osm.org/copyright',\n",
" 'osm_type': 'way',\n",
" 'osm_id': 368946992,\n",
" 'boundingbox': ['52.5341829', '52.534317', '13.4237697', '13.4240606'],\n",
" 'lat': '52.53425085',\n",
" 'lon': '13.423921421750569',\n",
" 'display_name': '8, Marienburger Straße, Winsviertel, Prenzlauer Berg, Pankow, Berlin, 10405, Deutschland',\n",
" 'class': 'building',\n",
" 'type': 'apartments',\n",
" 'importance': 0.4200099999999999}]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"nominatim_search(\n",
" address=None,\n",
" street=\"8, Marienburger Straße\",\n",
" city=\"Berlin\",\n",
" country=\"Germany\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand All @@ -400,7 +365,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -436,7 +401,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
"version": "3.11.5"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
Expand Down
Loading

0 comments on commit b99a8aa

Please sign in to comment.