Documentation of "AMI-friendly" approach

[beargiles]

I propose adding documentation of an "AMI-friendly" approach to writing suites of tests.

Background

The Cloudera CDH 6 software takes/took a looong time to install and configure. Even with a local cache we're talking about 20 minutes to simply download and unpack the software, then another 20 minutes for post-installation configuration. You still needed to do a little more work to actually put them into a cluster but even the free version of the "Cloudera Manager" made that pretty painfree once you had done some prep work.

Cloudera was aware of the impact the time it to spin up new instances and recommended an approach where the customer:

Preparation

• installed and partially configured the software
• backed out a few files, e.g., one contained a system UUID
• create an AMI image

Use

• create a new instance using the previously prepared AMI image (takes a few minutes vs. 20)
• performs the post-installation configuration
• performs the additional steps required to add to a cluster.

I've used this approach with great success.

Note: this approach isn't limited to EC2. It can be used with any platform that supports 'freezing' the current state of the system. E.g., docker images, k8s images, etc.

Molecule tests + AMIs

This approach works well with molecule. It's definitely beneficial with long-running launches, but can also be used with lighter tasks if you prefer to use pre-canned instances.

The general idea is:

Creation

• Create a molecule test that starts with a standard AMI (or prebuilt custom AMI) and installs all required software and performs a partial configuration. E.g., you might configure mail forwarding and monitoring but not a host digital certificate since you don't know the final hostname or IP address. This test should verify that the software was installed, any unique information has been deleted, etc.
• Modify 'destroy' so it creates an AMI image after a successful test.
• Record the new AMI's image_id somewhere useful. E.g., use 'lineinfile' to modify '.env.yml'
• Repeat this for all supported combinations of operating systems, applications (e.g., different LDAP servers), and versions.

Use

• Alter the other molecule tests so they launch the new AMI.
• Perform any required configuration
• Run the tests

This can be recursive. E.g., I use this to create a 'baseline' image for multiple OSes, then use that image to create pre-installed images for different services.