API Gateway status

[coopernurse]

I see kappa/restapi.py, and the README in this sample discusses API Gateway:

https://github.com/garnaat/kappa/blob/develop/samples/python/README.md

but it doesn't look like this feature is ready.

Is anyone working on this? API Gateway has a lot of surface, so any explanation on the tentative direction would be interesting to hear.

[josegonzalez]

Doesn't look like it's feature ready. I'm not overly familiar with api gateway, so I'm open to any and all discussion on how to move that functionality forward :)

[coopernurse]

Heh, I'm not too familiar with it either, but the format Gordon uses looks pretty sensible:

https://github.com/jorgebastida/gordon/blob/master/examples/apigateway/settings.yml

API Gateway is pretty big, so we'd probably want to identify the subset to support initially. The most important features to me off the top of my head are:

• Ability to map HTTP paths and methods to the lambda
• Ability to map responses to HTTP status codes
• Ability to map response HTTP headers (e.g. Content-Type or Location)
• Probably some way of mapping kapp's 'environment' name to the API Gateway 'stage'
  • This part is a little fuzzy to me

Perhaps other folks with more API Gateway knowledge could chime in. I think it may be easier to start from scratch rather than try to finish the restapi.py work that's already in the tree, but I could be wrong.

[josegonzalez]

Are there other features that we should work on before api gateway? I'll try and do some research on what this actually is before I make any further comments :)

[coopernurse]

Re: feature priority: good question. My interest in this project is research into whether I could realistic migrate some existing production systems from traditional web service deployments to AWS Lambda. So my (selfish) immediate requirements are:

• Optionally use code from S3 rather than zip - Fixes #65 #68 - Java support (i.e. don't build the zip from _src)
• CloudWatch events - Fixes #52 #69 - Scheduled event support (I have daemon threads in some services, so I need some analog)
• Ability to wire lambdas to HTTP URL endpoints (API Gateway)
• Code promotion strategy. Kappa has this via environments, but I don't have my head fully wrapped around it, and I don't fully understand how it relates to the API Gateway notion of 'stages'

So for me, API Gateway is pretty important. It's also one area that's really difficult to configure by hand (multi-stage wizards, lots of things to screw up). One tricky part is that Lambda has no real notion of headers vs payload, but HTTP does, so API Gateway has this wacky rule/expression system where you can map headers into/out of the request/response bodies. That's important to configure if you, say, want to send a HTTP redirect, or return a binary asset from your Lambda.

The boto3 API Gateway docs are very complete, and I'm referencing the operations in their docs here: http://boto3.readthedocs.io/en/latest/reference/services/apigateway.html

The user could specify a request in YAML as something like:

# new top level block in 'kappa.yml'
apigateway:
    # 'my-api' is the 'name' of an API gateway "rest api", which contains 1..n resources
    my-api:
        # '/foo' is a resource - API gateway models resources as a tree with parentIds, but
        # that strikes me as an implementation detail that Kappa can/should hide
        /foo:
            # maybe allow a list here? if we do that we'll need to do some expansion 
            # as each http method maps to a separate 'put_method' call
            http_method: GET
            authorization_type: NONE       # optional, default = NONE
            authorizer_id: xyz             # optional, only relevant if authorization_type=CUSTOM
            api_key_required: false        # optional, default = false
            request_parameters: dict [string->bool]    # optional, default is to omit
            request_models: dict [string->string]      # optional, default is to omit
            # these params are used for the 'put_integration' call
            integration:
                request_parameters: dict [string->bool]    # optional, default is to omit
                request_templates:  dict [string->string]  # optional, default is to omit
                passthrough_behavior: string          # optional
                cache_namespace: string               # optional
                cache_key_parameters: list of strings # optional
            # this block is optional and would trigger a call to 'put_integration_response'
            integration_response:
                selection_pattern: string   # optional
                response_parameters: dict   # optional
                response_templates: dict    # optional
            method_response:
                status_code: string         # optional, default = 200
                response_parameters: dict [string->bool]  # optional
                response_models: dict [string->string]    # optional

The relevant API Gateway methods we'd be calling would be:

• get_rest_apis - to find api by name
• get_resources - to find resources by restApiId
• create_rest_api - if we don't find the api by name
• create_resource - if we don't find the path in the resource tree for this api
• put_method - called once per path and http method (e.g. GET /foo vs POST /foo)
• put_integration called once per path and http method - this registers the lambda with the method
• put_method_response - called optionally if method_response is present
  • Otherwise delete_method_response would be called
• put_integration_response - called optionally if integration_response is present
  • Otherwise delete_integration_response would be called
• delete_method - Called for each method returned by get_resources NOT present in the YAML file for this API whose integration points at this lambda function
• delete_resource - Nice to have - if all delete_method calls remove all methods for a resource, we could remove that resource entirely to avoid polluting the api with orphaned paths

Whew.. That's my rough take on it initially. The major omission I see is the API Gateway "stage" concept, but I'm willing to punt on that in the first version.

Most things in Kappa have a "convergence" model where (like CloudFormation) we try to make the AWS resources converge to reflect the YAML. This is no different, but the underlying AWS model is fairly complex. I would probably suggest we write this in a more functional style, as this is basically just a matter of loading the existing data, loading the YAML, and doing a pure diff that can return a data structure of the API calls needed to converge the two. The nice thing about boto3 is that you can represent all API calls as data, so we don't need to mock anything.

I realize this is a pretty long comment, but it helped me organize my thoughts on how this should look. Not sure how much time I'll have this week (gotta get back to day job) but maybe others can weigh in over the next few days.

[garnaat]

The restapi part is definitely not ready for prime time and probably isn't even directionally correct. I started playing around with it but since we didn't need it for the app we were building it just never got fleshed out.

My thinking is that we need two levels of support.

The first level would be the super-easy case where we bake in a lot of assumptions/conventions in and allow the user to get basic mapping of Lambda functions to REST endpoints in a really easy way and without knowing much about API Gateway.

The second would be close to what @coopernurse shows above. A full incarnation of the API Gateway features in our YAML.

Thoughts?

[mnp]

I would like to see working APIGateway example, even if all hardcoded (or at least fill-in-the-blank).

I've had zero trouble building Lambdas and testing them by hand, but AWS has made it much much harder to get that lambda working with API Gateway, roles, and logging, what with poor documentation and obscure errors that don't tell you what the problem is. Kappa could be a real help here.