Skip to content

Commit

Permalink
added aiohttp_retry hint to README
Browse files Browse the repository at this point in the history
  • Loading branch information
d70-t committed Mar 28, 2022
1 parent 4f11240 commit b106523
Showing 1 changed file with 31 additions and 0 deletions.
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,37 @@ with fsspec.open("swift://server/account/container/object.txt", "r") as f:

`swiftspec` uses the environment variables `OS_STORAGE_URL` and `OS_AUTH_TOKEN` for authentication if available. To create these variables, you can use the `swift auth` command from the [python-swiftclient](https://docs.openstack.org/python-swiftclient/latest/cli/index.html).

## fault tolerance / automatic retry

Sometimes reading or writing from / to a swift storage might fail occasionally. If many objects are accessed, occasional failures can be extremely annoying and could be fixed relatively easily by retrying the request. Fortunately the `aiohttp_retry` package can help out in these situations. `aiohttp_retry` provides a wrapper around an `aiohttp` Client, which will automatically retry requests based on some user-provided rules. You can inject this client into the `swiftspec` filesystem using the `get_client` argument. First you'll have to define an async `get_client` function, which configures the `RetryClient` according to your preferences, e.g.:

```python
async def get_client(**kwargs):
import aiohttp
import aiohttp_retry
retry_options = aiohttp_retry.ExponentialRetry(
attempts=3,
exceptions={OSError, aiohttp.ServerDisconnectedError})
retry_client = aiohttp_retry.RetryClient(raise_for_status=False, retry_options=retry_options)
return retry_client
```

afterwards, you can use this function like:

```python
with fsspec.open("swift://server/account/container/object.txt", "r", get_client=get_client) as f:
print(f.read())
```

or:


```python
import xarray as xr
ds = xr.Dataset(...)
ds.to_zarr("swift://server/account/container/object.zarr", storage_options={"get_client": get_client})
```

## Develop

### Code Formatting
Expand Down

0 comments on commit b106523

Please sign in to comment.