NEWT 2.0

NEWT - Nice and Easy Web Toolkit, a ReSTFul API for High Performance Computing.

#Modules The NEWT 2.0 is organized into 7 different, individually configurable modules. Each module is individually customizable by linking in a new adapter that conforms to the specification.

• Account: Gets information about users and groups on the system.
• Authentication: Deals with authentication of users.
• Command: Runs commands on the system.
• File: Create, remove, update, and delete objects in the file system.
• Job: Queues, monitors, and modifies jobs on the system.
• Status: Checks the status of each system.
• Stores: Provides a key-value storage system for applications using NEWT 2.0.

#Initial Setup NEWT 2.0 is shipped with a local implementation of each of its different modules so it will function out of the box

##Runtime Requirements

• python 2.7.* (https://www.python.org/downloads/)
• pip (http://pip.readthedocs.org/en/latest/installing.html#install-pip)
• libmagic
• sqlite (or other database for Django)

##Python Package Dependencies With pip: pip install -r requirements.txt. Without pip: install these packages manually:

###Required

• Django (https://docs.djangoproject.com/en/dev/topics/install/)
• filemagic (http://filemagic.readthedocs.org/en/latest/guide.html#installation)
• netaddr (https://pythonhosted.org/netaddr/installation.html)
• pyOpenSSL (https://github.com/pyca/pyopenssl)
• pytz (http://pytz.sourceforge.net/#installation)
• requests (http://docs.python-requests.org/en/latest/user/install/)
• wsgiref (https://pypi.python.org/pypi/wsgiref)

###Store Backend Packages

• pymongo (for MongoDB stores) (http://api.mongodb.org/python/current/installation.html)
• redis (for Redis stores) (https://kushal.fedorapeople.org/redis-py/html/user/install.html)

#Getting Started

##Understanding settings.py/local_settings.py The settings.py file (soon to be easier to configure) contains global variables used to edit the functionality of NEWT 2.0. settings.py can be edited, but it is recommended to create a local_settings.py to overwrite the global variables to preserve the default settings. (See newt/sample_local_settings.py)

##Running the server After changing any settings ./manage.py syncdb, then run ./manage runserver to start the server.

#Adapting NEWT Adapting any module of NEWT to your system is a relatively simple task given experience in Python and Django. In each module, there is an adapters folder where new adapters should be placed. The specification for each adapter is given in <adapter_name>/adapters/template_adapter.py.

##Abridged Example (django_adapter for Account) Say we want to create a new adapter for Account that uses the Django user model to obtain user information. We would first make a copy of template_adapter.py into django_adapter.py. To add functionality, we see that in the template_adapter.py of the Account module, there is a definition for get_user_info

def get_user_info(user_name=None, uid=None):
    """Returns information about the user

    Keyword arguments:
    user_name -- username
    uid -- user id
    """
    pass

We would edit the body of the function to make make calls to the Django user model to obtain the information we want:

def get_user_info(user_name, uid):
    try:
        if uid:
            user = User.objects.get(pk=uid)
        elif user_name:
            user = User.objects.get(username=user_name)
        else:
            raise Exception() # Invalid arguments were given
        user_dict = model_to_dict(user)
        del user_dict["password"]
        return user_dict
    except Exception:
        if user_name:
            return json_response(
                status="ERROR", 
                status_code=404, 
                error="User not found: %s" % user_name
            )
        else:
            return json_response(
                status="ERROR", 
                status_code=404, 
                error="User ID not found: %s" % uid
            )

Note: This is only one of the two required functions in the adapter; a full example can be seen in account/adapters/django_adapter.py.

A couple things to note:

• The function takes in arguments specified by the API url
• The function is allowed to return any Python object, not necessarily a Django response. NEWT automatically wraps the returned values in a Django response.
• To explicitly return a valid API response, use json_response(status="status", status_code=http_status_code, error="optional_error_message").

##Adapter Extras Adapters also support API calls that are not in the default API endpoints. By adding to patterns (near the end of the sample adapter), the extras_router (required in every adapter for now) will add them to NEWT.

patterns is a tuple list in the form of:

(
    (compiled_regex_exp, associated_function, request_required),
    ...
)

Note: The compiled_regex_exp must have named groups corresponding to the arguments of associated_function Note: request_required tells NEWT to pass the request object to your function. If request_required is True, the associated_function must have request as the first argument

Example:

patterns = (
    (re.compile(r'/usage/(?P<path>.+)$'), get_usage, False),
    (re.compile(r'/image/(?P<query>.+)$'), get_image, False),
    (re.compile(r'/(?P<path>.+)$'), get_resource, False),
)

##Models Some adapters may need to store application specific state in the Django database. In this case you can define a Django model (see https://docs.djangoproject.com/en/1.6/topics/db/models/). You should use place your models file in the adapters/ directory and set the NEWT_CONFIG['ADAPTERS'][application]['models'] to point to this in settings.py

For example this is how you would get Django to store a credential object in the authnz adapter. Note that you need to specify the app_label in the Meta section of your model to point to the parent application

authnz/adapters/my_models.py:

from django.db import models
from django.contrib.auth.models import User

class Cred(models.Model):
    class Meta:
        app_label = 'authnz'
    cred = models.TextField()
    user = models.ForeignKey(User)

settings.py:

NEWT_CONFIG = {
    
    'ADAPTERS': {

        'AUTH': {
            'adapter': 'authnz.adapters.my_adapter',
            'models': 'authnz.adapters.my_models',
        },

##Authenticated functions and adapters

You can enable authenticated access to the API at two levels:

1. Authenticate an entire view

In the views.py function for a given API resource, you can change the implementing view to inherit from AuthJSONRestView instead of JSONRestView.

eg. In the store adapter views.py file, you will need to change

from newt.views import JSONRestView

class StoreView(JSONRestView):
    def get(self, request, store_name):

to

from newt.views import AuthJSONRestView

class StoreView(AuthJSONRestView):
    def get(self, request, store_name):

2. Authenticate a specific adapter function

Inside your adapter you can use the login_required decorator to force authentication on a specific function eg. in file/adapters/globus_file_adapter.py

from common.decorators import login_required

# Add the login_required decorator to force authentication on get_machines()
@login_required
def get_machines(request):

#Contributors to NEWT 2.0

• Shreyas Cholia
• Terence Sun