Skip to content

Commit

Permalink
update docs
Browse files Browse the repository at this point in the history
  • Loading branch information
leo-schick committed Feb 22, 2023
1 parent ac2dea4 commit ec0466e
Show file tree
Hide file tree
Showing 4 changed files with 87 additions and 5 deletions.
3 changes: 2 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ This section focuses on the supported storages.
:maxdepth: 2

storages-overview
storages/local
storages/azure
storages/gcs
storages/local


API Reference
Expand Down
13 changes: 9 additions & 4 deletions docs/storages-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@ Overview

The following storages are supported

| Storage | Configuration class |
| ---------------------- | ------------------- |
| Local bash | LocalStorage |
| [Google Cloud Storage] | GoogleCloudStorage |
| Storage | Configuration class |
| ------------------------- | ------------------- |
| Local bash | LocalStorage |
| [Google Cloud Storage] | GoogleCloudStorage |
| [Azure Blob Storage] | AzureStorage |
| [Azure Data Lake Storage] | AzureStorage |

[Google Cloud Storage]: https://cloud.google.com/storage
[Azure Blob Storage]: https://azure.microsoft.com/en-us/products/storage/blobs
[Azure Data Lake Storage]: https://azure.microsoft.com/en-us/products/storage/data-lake-storage/


Function support matrix
Expand All @@ -18,6 +22,7 @@ Function support matrix
| --------------------- | ---- | ----- | ------ | ------ |
| LocalStorage | Yes | Yes | Yes | Yes
| GoogleCloudStorage | Yes | Yes | Yes | Yes
| AzureStorage | Yes | Yes | Yes | Yes

```{note}
A `Move` operation is not implemented by design. Most of the blob storages do not
Expand Down
75 changes: 75 additions & 0 deletions docs/storages/azure.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
Google Cloud Storage
====================

Accessing a Azure Blob Storage (GCS) with the shell tool `azcopy`.

Installation
------------

You need to install `azcopy`. Take a look at `Get started with Azcopy <https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10>`_.


Configuration examples
----------------------

```{note}
Currently some of the functions require a SAS token, and some of the functions
require a account key. It is recommended to provide both a SAS token
and a account key.
```

.. tabs::

.. group-tab:: SAS token

.. code-block:: python
import pathlib
import mara_storage.storages
mara_storage.config.storages = lambda: {
'data': mara_storage.storages.AzureStorage(
account_name='account-name',
container_name='container-name',
sas='sp=racwdlm&st=2022-05-11T10:04:05Z&se=2023-05-11T18:04:05Z&spr=https&sv=2020-08-04&sr=c&sig=u7tqxugyv5MbyrtFdEUp22tnou4wifBoUfIaLDazeRT%3D'),
# optional
storage_type = 'dfs' # use a dfs client instead of 'blob' (default value)
}
.. group-tab:: Account key

.. code-block:: python
import pathlib
import mara_storage.storages
mara_storage.config.storages = lambda: {
'data': mara_storage.storages.AzureStorage(
account_name='account-name',
container_name='container-name',
account_key='<key>',
# optional
storage_type = 'dfs' # use a dfs client instead of 'blob' (default value)
),
}
|
|
API reference
-------------

This section contains database specific API in the module.


Configuration
~~~~~~~~~~~~~

.. module:: mara_storage.storages
:noindex:

.. autoclass:: AzureStorage
:special-members: __init__
:inherited-members:
:members:
1 change: 1 addition & 0 deletions mara_storage/azure.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ def init_service_client(storage: storages.AzureStorage, path: str = None) -> Blo
client = BlobServiceClient.from_connection_string(storage.connection_string())
return client


class AzureStorageClient(StorageClient):
def __init__(self, storage: storages.AzureStorage):
super().__init__(storage)
Expand Down

0 comments on commit ec0466e

Please sign in to comment.