docs: usage

- overview structure and content
- usage structure and content
- added `BucketTag` in `__all__` of `models` so its documentation shows up
- closes inveniosoftware#198


Co-authored-by: Vlad Mihaescu <[email protected]>
Co-authored-by: Nicola Tarocco <[email protected]>

INSTALL.rst

@@ -7,3 +7,9 @@
 
 Installation
 ============
+
+Invenio-Files-REST is on PyPI so all you need is:
+
+.. code-block:: console
+
+   $ pip install invenio-files-rest

MANIFEST.in

@@ -6,17 +6,18 @@
 # Invenio is free software; you can redistribute it and/or modify it
 # under the terms of the MIT License; see LICENSE file for more details.
 
-include *.rst
-include *.sh
-include *.txt
 include .dockerignore
 include .editorconfig
 include .tx/config
-include LICENSE
+include *.rst
+include *.sh
+include *.txt
 include babel.ini
 include docs/requirements.txt
+include LICENSE
 include pytest.ini
 recursive-include docs *.bat
+recursive-include docs *.png
 recursive-include docs *.py
 recursive-include docs *.rst
 recursive-include docs Makefile

README.rst

@@ -23,9 +23,16 @@
         :target: https://github.com/inveniosoftware/invenio-files-rest/blob/master/LICENSE
 
 
-Files download/upload REST API similar to S3 for Invenio.
+Invenio-Files-REST is a files storage module. It allows you to store and
+retrieve files in a similar way to Amazon S3 APIs.
 
-*This is an experimental developer preview release.*
+Features:
 
-* Free software: MIT license
-* Documentation: https://invenio-files-rest.readthedocs.io/
+ * Files storage with configurable storage backends
+ * Secure REST APIs similar to Amazon S3
+ * Support for large file uploads and multipart upload.
+ * Customizable access control
+ * File integrity monitoring
+
+
+Further documentation is available on https://invenio-files-rest.readthedocs.io/.

docs/conf.py

@@ -30,6 +30,7 @@
     'sphinx.ext.intersphinx',
     'sphinx.ext.viewcode',
     'sphinxcontrib.httpdomain',
+    'celery.contrib.sphinx',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -157,7 +158,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-#html_static_path = ['_static']
+html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied
@@ -320,6 +321,7 @@
                          None),
     'invenio-rest': ('https://invenio-rest.readthedocs.io/en/latest/', None),
     'python': ('https://docs.python.org/3', None),
+    'sqlalchemy': ('https://docs.sqlalchemy.org', None),
 }
 
 # Autodoc configuraton.

docs/exampleapp.rst

@@ -6,10 +6,8 @@
     under the terms of the MIT License; see LICENSE file for more details.
 
 
-
-=====================
- Example application
-=====================
+Example application
+===================
 
 .. include:: ../examples/app.py
    :start-after: SPHINX-START

docs/index.rst

@@ -18,6 +18,7 @@ Invenio-Files-REST.
 .. toctree::
    :maxdepth: 2
 
+   overview
    installation
    configuration
    usage

docs/overview.rst

@@ -0,0 +1,136 @@
+..
+    This file is part of Invenio.
+    Copyright (C) 2015-2019 CERN.
+
+    Invenio is free software; you can redistribute it and/or modify it
+    under the terms of the MIT License; see LICENSE file for more details.
+
+
+Overview
+========
+Invenio-Files-REST is a files storage module. It allows you to store and
+retrieve files in a similar way to Amazon S3 APIs.
+
+Before getting started a brief overview will introduce you to the key concepts
+and terminology of the module.
+
+
+.. figure:: _static/InvenioFilesREST.png
+    :alt: Overview of InvenioFilesREST
+
+    Overview of Invenio-Files-REST
+
+
+Location
+--------
+The first concept to introduce is :code:`Location`. Locations are used to
+represent different storage systems. :code:`Location` has a :code:`name` and a
+:code:`URI` which could be a path in a local directory or a URI on a remote
+system. It is required to define at least one default location.
+
+See the API section of :py:class:`invenio_files_rest.models.Location` for more
+information.
+
+
+Storage
+-------
+A backend :code:`Storage` provides the interface to interact with a
+:code:`Location` and perform basic operations with files, such as retrieve,
+store or delete.
+
+By default, Invenio-Files-REST provides a simple local files storage
+:py:class:`invenio_files_rest.storage.PyFSFileStorage`. You can define
+your own storage with the configuration :py:data:`invenio_files_rest.config.FILES_REST_STORAGE_FACTORY`.
+
+An example of a remote storage system can be found at
+`Invenio-S3 <https://invenio-s3.readthedocs.io/>`_ which offers integration
+with any S3 REST API compatible object storage.
+
+See the API section of :py:class:`invenio_files_rest.storage` for more
+information.
+
+
+FileInstance
+------------
+Files on disk are represented with :code:`FileInstance`. A file instance
+records the path to the file, but also its `Storage`, size and checksum of the
+file on disk.
+
+See the API section of :py:class:`invenio_files_rest.models.FileInstance` for
+more information.
+
+
+Object
+------
+An :code:`Object` is an abstract representation of the file metadata: it
+doesn't come with its own data model but it is defined by
+:code:`ObjectVersion`.
+
+
+ObjectVersion
+-------------
+An :code:`ObjectVersion` represents a version of an :code:`Object` at a
+specific point in time. It contains the :code:`FileInstance` that describes and
+a set of metadata.
+
+When it has no :code:`FileInstance`, it marks a deletion of a file as a delete
+marker (or soft deletion).
+
+The latest version of the file is referred to as the :code:`HEAD`.
+
+Object version are very useful to perform operation on files metadata without
+accessing directly to the storage. For example, multiple :code:`ObjectVersion`
+can point to the same :code:`FileInstance`, allowing operations to be
+performed more efficiently, such as snapshots without duplicating files or
+migrating data.
+
+See the API section of :py:class:`invenio_files_rest.models.ObjectVersion` for
+more information.
+
+
+ObjectVersionTag
+----------------
+
+:code:`ObjectVersionTag` is useful to store extra information for an
+:code:`ObjectVersion`.
+
+A :code:`ObjectVersionTag` is in the form of :code:`key: value` pair and an
+:code:`ObjectVersion` can have multiple :code:`ObjectVersionTag`.
+
+See the API section of
+:py:class:`invenio_files_rest.models.ObjectVersionTag` for more information.
+
+
+Bucket
+------
+Consider the :code:`Bucket` as a container for :code:`ObjectVersion` objects.
+Just as in a computer, files are contained inside folders, each
+:code:`ObjectVersion` has to be contained in a :code:`Bucket`.
+
+The :code:`Bucket` is identified by an unique ID and is created in a
+given :code:`Location` with a given :code:`Storage`.
+
+:code:`ObjectVersion` are uniquely identified within a bucket by string keys.
+
+.. .note::
+
+    :code:`Objects` inside a :code:`Bucket` do not necessarily have the same
+    :code:`Location` or :code:`Storage` class as the :code:`Bucket`.
+
+A bucket can also be marked as deleted, in which case the contents become
+inaccessible. It can also be permanently removed: in this case, all contained :code:`ObjectVersions` will be also deleted.
+
+See the API section of :py:class:`invenio_files_rest.models.Bucket` for more
+information.
+
+
+BucketTag
+---------
+Similarly to :code:`ObjectVersionTag`, a :code:`BucketTag` is useful to store
+extra information for a :code:`Bucket`.
+
+A :code:`BucketTag` is in the form of :code:`key: value` pair and a
+:code:`Bucket` can have multiple :code:`BucketTag`.
+
+See the API section of :py:class:`invenio_files_rest.models.BucketTag` for
+more information.

docs/usage.rst

@@ -6,8 +6,7 @@
     under the terms of the MIT License; see LICENSE file for more details.
 
 
-
 Usage
-=======
+=====
 
 .. automodule:: invenio_files_rest

examples/app.py

@@ -6,7 +6,7 @@
 # Invenio is free software; you can redistribute it and/or modify it
 # under the terms of the MIT License; see LICENSE file for more details.
 
-"""Minimal Flask application example for development.
+r"""Minimal Flask application example for development.
 
 SPHINX-START
 
@@ -170,13 +170,13 @@
 
 import os
 import shutil
-from os import environ, makedirs
+from os import makedirs
 from os.path import dirname, exists, join
 
 from flask import Flask, current_app
 from flask_babelex import Babel
 from flask_menu import Menu
-from invenio_access import ActionSystemRoles, InvenioAccess, any_user
+from invenio_access import InvenioAccess
 from invenio_accounts import InvenioAccounts
 from invenio_accounts.views import blueprint as accounts_blueprint
 from invenio_admin import InvenioAdmin
@@ -192,6 +192,7 @@
 
 def allow_all(*args, **kwargs):
     """Return permission that always allow an access.
+
     :returns: A object instance with a ``can()`` method.
     """
     return type('Allow', (), {'can': lambda self: True})()