SPIKE: Find a tool for running scenarios

[ThetaSinner]

The intended deployment pattern for Wind Tunnel was to colocate a binary for Holochain and the scenario on each test node and start many scenario+Holochain pairs across a network. This has the advantage of scaling better than the TryCP method we've landed on for now. It also removes the extra network traffic that we currently require to coordinate scenarios.

If we don't solve this problem, then we need to rewrite the remainder of the original wind tunnel scenarios that were meant to be run this way. They would have to be re-written to use TryCP instead.

AC:

• Any tool we pick must be simple to deploy, operate and maintain.
• The tool should be simple enough to learn that we can have multiple people learn to use it.
• We don't generally require synchronization or communication between the scenario instances but we should be able to start them all at a similar time. I.e. It wouldn't be acceptable to start them in series if we have to wait for Holochain to start up with each one.
• It must be possible, with documentation, to run the tool locally so that we can iterate on the setup.
• Should support running on our supported platforms. That would be Mac, Linux, Windows, Android (see tx5 CI) and iOS. It doesn't have to work up front, we just want to be able to get there.
  • This is relevant to packaging. If we're using Nix to make the scenario binaries then we'd need cross compilation to support Windows. Maybe this is a good reason to consider using Rust for building scenarios?

[ThetaSinner]

Have previously looked at Nomad. Would we still consider using that? Or are there better options.

Might we get more value from a custom tool than the work we'd have to put into an existing tool that doesn't do quite what we need.