Documentation updates

design/README.rst

@@ -7,7 +7,7 @@ is being planned.
 
 The primary aim of Proxy42 is to act as a HTTP proxy.
 
-Initial thoughts:
+Initial thoughts and design goals:
 
 - Pure erlang/otp application, that can be built into a Xen Unikernel via
   Erlang on Xen, and an executable via HiPE.
@@ -31,37 +31,19 @@ Initial thoughts:
 - Should support websocket proxying, with an option for intercepting messages
   within Proxy42
 
-Each incoming request is handled in a process of its own. This provides
-excellent isolation at little to no performance penalty thanks to Erlang VM.
 
-A brief outline of steps involved in proxying a request are as follows
+During the early stages of development, some of the goals evolved, and others had to be abandoned/postponed indefinitely.
 
-#. Acceptor pool accepts incoming requests, performs ssl termination if desired.
-#. Spawns a new process immediately and hands the socket over.
-#. The process is a gen fsm with the following phases. (gen_fsm vs gen_server?)
+HTTP parsing is nontrivial, even if it looks simple at first glance. It is
+already implemented in libraries like cowlib, redoing the work is pointless.
+We can build on top of existing erlang webservers. After a few exploratory
+investigations, we settled on vegur (backed by cowboyku - a fork of cowboy 0.9)
+to handle low level http interactions and to focus on developing routing,
+control and metrics.
 
-   - Parse headers and url
-   - Identify config: target upstream server, auth, rate limits
-   - Perform auth: Pluggable adapters? JWT based? Stored tokens?
-   - Check rate limits
-   - Forward to admin app or proxy app as appropriate.
-   - Initiate proxying, wait for response
-   - Add ``X-RateLimit-*`` and other headers, stream response
-   - Clean up and close the connection
-
-   Timestamps are recorded at each step in the process state, and sent to
-   analytics subsystem
-
-Analytics subsystem uses pluggable adapters. The adapters are expected to
-handle the following data:
-
-#. Counters for requests received, bytes transferred, etc
-#. Consolidated timing information.
-
-We will very likely have a default adapter backed by ets/mnesia with limited
-functionality, and a few adapters to stream the events or their aggregates to
-various tools over TCP/UDP.
-
-We need to store time taken per phase for each request to be able to chart the
-trends.  This data may be compacted into summaries (mean, median, stddev, etc)
-once in a while to save space. Details yet to be worked out.
+Current status
+==============
+We currently support only plain HTTP 1.1. Mnesia is used for internal storage.
+Clustering is not officially supported yet, though it is kept in mind when
+designing the components. A minimal Admin API exists, with plans of overhauling
+it entirely.

design/design.md → docs/design.md

@@ -1,3 +1,8 @@
+---
+id: design
+title: Design
+---
+
 - [API](#sec-1)
 - [API Owner](#sec-2)
 - [API Developer](#sec-3)

docs/internals/reading-the-code.md

@@ -0,0 +1,17 @@
+---
+id: reading-the-code
+title: Reading the code
+---
+
+## Who this document is for
+
+Developers intending to work on Proxy42 codebase, for developing new features or hunting bugs.
+
+## Prerequisites
+
+Knowledge of erlang and/or elixir is necessary to attempt debugging or
+feature development. If you intend to merely read the code, it may be possible
+to get by even if your're familiar with any programming language.
+
+## Overview
+

docs/internals/request-flow.md

@@ -0,0 +1,77 @@
+---
+id: request-flow
+title: Request Flow
+---
+
+When Proxy42 recieves a request, it passes through various "phases". It is
+helpful to understand these phases when developing plugins for Proxy42 or when
+working on the codebase.
+
+
+ 1. Incoming requests are accepted by a pool of ranch acceptors.
+ 1. Cowboyku takes control of the request, parses it, and transfers control to
+    vegur.
+ 1. Vegur runs `vegur_midjan_middleware` to process the request.
+    `vegur_request_log:{new/1,done/4}` are run at the beginning and end of
+    request respectively.
+ 1. Midjan is an ececutor. It can execute a chain of modules, going back and
+    forth if necessary. Midjan executes the following middleware chain, in the
+    listed order. Some of these middlewares expect a callback function on an
+    "Interface Module", which can maintain an internal "Handler State" to
+    configure their behaviour.
+    1. `vegur_validate_headers` - Performs basic header validation
+    1. `vegur_lookup_domain_middleware` - Looks up the API for request
+    1. `proxy42_authenticate_middleware` - Handles request authentication and
+       authorization. Ratelimits are also handled here for now.
+    1. `vegur_continue_middleware` - Intercepts and responds to 100 Continue
+    1. `vegur_upgrade_middleware` - Handles connection upgrades - ex: to
+       websockets
+    1. `vegur_lookup_service_middleware` - Picks a server to send the request
+       to. Load balancing/shaping is implemented by its callback.
+    1. `vegur_proxy42_middleware` - Establishes connection to API server and
+       proxies request and response.
+
+    Some of the names might be non intuitive since we adapted vegur's default
+    stack with minimal changes - Some modules will be renamed or rewritten in
+    near future. In particular, some existing logic in router will be moved
+    into middlewares, lookup_domain will be renamed to lookup_api, and applying
+    hierarchial settings will be the responsibility of middlewares rather than
+    router.
+
+The project has a single interface module, `proxy42_router`. This should be the
+entrypoint for contributors when getting started on the project.  The following
+functions/callbacks are exported by the interface module, and are listed in the
+order in which they will be executed.
+
+1. `init/2` - Called by `vegur_req_log:new/1` to initialize router state. This
+   is called Request Context (ReqCtx) throughout the router, and HandlerState
+   in the middleware.
+1. `lookup_domain_name/3` - Called by `vegur_lookup_domain_middleware`. Domain
+   level and API level settings are applied here.
+1. `auth/2` - Called by `proxy42_authenticate_middleware` to perform
+   authentication and authorization. The actual authentication is delegated to
+   authentication plugins. Developer and Application User level settings are
+   applied here.
+1. `rate_limit/3` - Called by `proxy42_authenticate_middleware` based on result
+   of `auth/2`. Ratelimit plugin in effect will be called to record the request
+   and check quota.
+1. `checkout_service/3` - Called by `vegur_lookup_service_middleware` to pick a
+   server from available backend API servers for requested API.
+1. `checkin_service/6` - Called by `vegur_lookup_service_middleware` just
+   before releasing the backend server into the pool - after request is
+   successfully completed or before requesting a new checkout if the chosen
+   server is unresponsive.
+1. `service_backend/3` - Called by `vegur_lookup_service_middleware` to resolve
+   the chosen server into an ip address and port.
+1. `backend_request_params/3` - Called by `vegur_proxy42_middleware` to
+   customize request parameters. Request path is rewritten if necessary.
+1. `transform_response_headers/2` - Called by `vegur_proxy42_middleware` to
+   customize response headers. Host header is rewritten in particular.
+1. `additional_headers/4` - Called by `vegur_proxy42_middleware` to customize
+   headers sent in either direction.
+1. `error_page/4` - Translates internal errors from middlewares to HTTP
+   statuses. Called by middlewares via `vegur_utils:handle_error/2`.
+1. `terminate/3` - Called at the end of request lifecycle via `vegur_request_log:done/4`.
+1. `feature/2` - The behaviour of `vegur_proxy42_middleware` and
+   `vegur_continue_middleware` can be customized by defining feature flags. We
+   do not use this currently.

docs/internals/writing-plugins.md

@@ -0,0 +1,6 @@
+---
+id: writing-plugins
+title: Writing plugins
+---
+
+

docs/plugins.md

@@ -0,0 +1,8 @@
+---
+id: plugins
+title: Plugins
+---
+
+This page lists the plugins available for Proxy42. To write your own plugins, please see Writing Plugins
+
+TODO: linkify