Create cluster coordination wrappers

[dazza-codes]

Although risky, it might be possible to create an entire cluster coordination wrapper. Some ideas:

• cap {stage} cluster:up - do everything and do it idempotently (wow, really?)
• cap {stage} cluster:down - Are you sure? OK, tear it down!

Cluster settings:

• what services to stand up (and their dependencies)
• manage the order of spinning up infrastructure resources and installing services
  • e.g. ZooKeeper before Kafka, Mesos etc.
  • an example of dependency management is in the security group validation
  • try to avoid complex dependency graph management/maintenance
    • perfection is hard; is there an easy solution for this? (show Debian some love)
    • possibly relevant - https://github.com/Netflix/archaius/wiki/Overview
• ensure the service configuration dependencies are initialized and maintained

ZooKeeper is basically required for just about all of it, so how many nodes to install and whether to associate all dependent services with one ZK quorum or use multiple ZK quorums? Try to leave this detail to the settings, just add/manage attributes on services to connect them all up correctly.

What about connections to external services - some external dependencies might exist that do not require spinning up any resources, but just require configuration management for network security groups and connections. Again, an example of this kind of dependency management is in the security group validation for spinning up an EC2 instance.