Cloudformation Dependency

[pilgrim2go]

Hi,

I'm using Aloisius 0.4.
I have following Cloudformation template ( Using troposphere)

base.py

iam.py

user.py

That is IAM template is depending on Base template
and User template depends on IAM template.

With new 'waiters' feature, can Aloisius implement such dependency chaining?

I'm digging the code but haven't figured it out yet.

Thanks again for your share.

[adonig]

Hi @pilgrim2go,

please have a look at the example code in the README.md file. You should be able to import your troposphere templates and then setup a hierarchy by declaratively adding your stacks to the file and chaining the stacks using the outputs. There is a dependency between two stacks, if one stack uses an output of another stack as input parameter. If there is a dependency aloisius executes the operations on those stacks in a sequential manner. If there is no dependency between two stacks, the parallelisation should happen automatically :-)

Best
Andreas

[pilgrim2go]

@adonig:
As far as I know, if you have 5 Stack and adjust the following line

in stack.py

    _executor = ThreadPoolExecutor(max_workers=3)

You'll see unexpected result ( eg some threads lost).

Due to myy lack of concurrency experience, I can't show the reason why yet. But each ThreadPool per Stack looks weird to me at first.

Besides, I can't find any line of code can specify Stack dependency. Such feature is really needed.

[adonig]

Oh, no no no. It is automatically set to the number of CPUs your computer has:

_executor = ThreadPoolExecutor(max_workers=multiprocessing.cpu_count())

That makes sense because your computer can run that many tasks in parallel.

Stack dependencies are currently specified implicitly over stack outputs and stack input parameters. Here's an example:

stack1 = Stack(...)
stack2 = Stack(..., Parameters={'input1': stack1.outputs['foo']})
stack3 = Stack(...)

In that scenario, the operations on stack1 and stack3 are executed in parallel, while the operation on stack2 has to wait until the operation on stack1 is finished (because otherwise the outputs of stack1 would not exist).

[diasjorge]

If you don't use the outputs you could force blocking using aloisius.stacks.wait() before executing the next one

[adonig]

If there is use for it, we could think about adding a constructor parameter DependsOn so it would be possible to write something like:

stack1 = Stack(...)
stack2 = Stack(..., DependsOn=[stack1])

[pilgrim2go]

@adonig : Yes, I prefer DependsOn solution. That makes Stack definition more clearer and we can build complex dependency graph.
Also I 1thought multiprocessing.cpu_count() is for process ( vs thread). Since my cpu_count is 4 that why explicitly set number=4 is the same.

Btw, I ended up using your solution to do something simple ( my case).

sc = StackCollection()
s1 = Stack('vpc')
s2 = Stack('cluster', depends=s1).

sc.add(s1)
sc.add(s2)

and then I can do something like
sc.runall()
sc.delete() 
or
sc.runall(queryset=QuerySet(__name__'vpc')

Code is copied from Alosius ( worked for me but not a generic solution)