README

Lapaz is an asynchronous JRuby application framework using components supported by ZeroMQ, Mongrel 2
and MongoDB.

Lapaz 0.0.1 is not finished.  It's halfway through the middle.

Why?

Four unrelated projects happened.
1) ZeroMQ
2) MongoDB
3) Llama
4) Mongrel2

Lapaz is inspired by Llama but takes the concept in a different direction using ZMQ for inter-component
communication.  ZeroMQ (ZMQ) removed the broker intermediary and simplified inter-thread communication.
Mongrel2 went async using ZMQ to send requests to a handler and waiting for a reply.

What can I build with it?

Lapaz can build applications or parts of applications that occur asynchronously or are
functionally decomposable.

For example:
A price update receiver - price updates received from a message bus and are saved to a Mongo DB.
A notification server - receives a message and signals a third party via a webservice.

Mongrel2 adds HTTP accessibility to a Lapaz application primarily to serve client side javascript apps.

How does it work?

It is composed of components that communicate with each other. Components are assembled in a route,
actually more like a directional acyclic graph with messages flowing from the starting step through
the paths of the route.  A group of routes make up an application.

Each component has three pieces -
pull: subscribes to messages addressed to it,
process: process the pulled message,
push: publishes the processed message to the next component.

Specialised components redefine one or more of the above pieces.

A Producer will redefine the pull component and produce internal messages from an external source.
Producer examples are: a MongrelProducer that receives messages from a browser and is one half of the
Mongrel2 handler, XmppProducer, RabbitMQProducer. A Producer creates a new message with an iteration id.

A Processor will redefine the process piece - typically augmenting the pulled message in some way.
Processor exampes are: a TemplateRenderer that will use the message and a template to render HTML,
DbReader, DbWriter, YamlParser and Domain specific Components.

A Consumer will redefine the push piece and send a (transformed) message to an external target or
a named step in another route (perhaps running on a different machine).
Consumer examples are: a MongrelConsumer (the other half of the Mongrel2 handler) that sends a
response to Mongrel2, RabbitMQConsumer, SysLogConsumer, XmppConsumer, StdOut, EmailSender.

A Route is a sequence of components arranged in steps 0 to N. One component is at step 0, typically.
Multiple components can be in a step - they receive and process messages in parallel (multiplex).

Any component can demultiplex messages sent to it and will accumulate messages until the last
message is received. Accumulation is grouped by iteration id. When the last message is received they
are merged together and published to the next step.  Multiplexed messages are identified by a
mux id, e.g. 5.3 meaning - expect to accumulate five messages and this is the third (which may be
received before the second or first message).

Forwarding consumers can publish a message to a named step in another route.

This is a route from a test application. This route has no Producers at step 0 so by subscribing to
the topic "purchases/0" it waits for a message.  This route is engaged when another route sends it a
message.  It also shows the parallel enaction of Components in the subsequent step.

route(:route_name=>"purchases") do
  from Processor::Purchases, {:seq_id=>0,:name=>'start'}
  to Consumer::Forwarder,{:seq_id=>1,:forward_to=>'prices/start',:reply_to=>'purchases/render/3.3'}
  to Processor::Contacts,{:seq_id=>1,:mux_id=>'3.1'}
  to Processor::StockItems,{:seq_id=>1,:mux_id=>'3.2'}
  to Processor::TemplateRenderer,{:seq_id=>2,:name=>'render'}
  to Processor::LayoutRenderer,{:seq_id=>3}
  to Consumer::MongrelConsumer,{:seq_id=>4}
end

Multicast Publish and Subscribe for service discovery and inter-app messaging.

Using ZMQ epgm multicast capability there is support for inter-app service discovery
When two or more applications are run they will query all for an external services update.
All apps (except the requestor) will respond with a message containing their callable
external services (routes). All apps receive this message and update their internal caches.
When a route is designed that uses an external route it will be called and the reply will be
send to the designated route/step.  In the above example the forward_to =>'prices/start' might
be an external call and if so it will be invoked by looking it up in the external services cache.
This has been implemented with Lapaz component and routes and some small changes to the internals.

I have added a Lurker producer component that can subscribe to any topic.  This can be used for
logging or any other independant (of the design of a route) message consumption, e.g. send a XMPP
message when a shipment confirm message is seen.

TODO:
Add more components: RabbitMQ, SysloggerConsumer.
ZMQ request/reply entry point for synchronous peers (Rails, Sinatra) with a driver.
Sammy.js based example client app taking JSON from Lapaz.
Compensate, or how to reverse a previous steps state change when something goes wrong.
Cascade demultiplexing.
HTTP streaming when availble in Mongrel2.

BTW

time curl -i -H "Accept: application/json" http://localhost:6767/handlertest/purchases/1234-DSF.json
HTTP/1.1 200 OK
Content-type: application/json; charset=utf-8
Content-Length: 411

{"kind":"default","body":{"mongrel_req_body":"",
"purchases":[{"_id":{"$oid":"4ccf9fde652e5d81cd789c0a"},
"id":"1234-DSF","contacts":{"id":"886644","name":"Bob Smith"},
"items":[{"id":"4521","name":"Widget X","price":45.21,"ccy":"EUR"}],
"notes":"rest of purchase object here"}],
"prices":[{"_id":{"$oid":"4ccf9fde652e5d81cb789c0a"},"ccy_pair":"EURGBP","bid":"0.87330","offer":"0.87427"}]
},
"errors":[],
"warnings":[]}

real  0m0.048s
user  0m0.010s
sys   0m0.000s

More to follow...