Assets (Collectors, Testlets, etc) Re-think

[Mierdin]

Several comments, such as https://github.com/Mierdin/todd/pull/22#discussion_r62946419 have been brought up regarding assets like testlets and collectors in ToDD.

The concept of embedding these inside the todd-server binary solved an early problem but is turning out to be more trouble than it's worth. In short, we need to come up with a solution that doesn't require the use of go-bindata.

Here are some requirements for this re-think:

• Need to be able to take existing testlets or collectors written in bash and simply copy them into the operational directory at install time. Instead of embedding them with go-bindata, these can simply be copied from the repo into the appropriate location
• Testlets or collectors should also be able to be written in Go, and perhaps the "native" ToDD testlets can be re-written in Go, provided they follow the testlet input and output specifications (meaning they cannot be tightly integrated with ToDD. They have to be totally self-sufficient). When ToDD is being installed or compiled, these should be compiled individually, and the resulting binaries should be placed in the appropriate directory on the system.

EDIT: Would also like to be able to add new testlets easily. Since open sourcing ToDD I've envisioned something similar to what ansible-galaxy does for Ansible roles. Preferably this can/should be done at any time, not requiring todd-server restart, or anything like that

[bigmstone]

As with every decision there are trade-offs. The choice here is cross-platform/portability vs lower barrier of entry to writing testlets. In my view Todd should take the approach of lowering the barrier of entry to writing testlets because Todd's value is intrinsically linked to its ability to run these test.

This choice isn't free though. So with that in mind I have a few thoughts.

Todd should be packaged with a set of testlets that are Go-Native and cross platform. These testlets should be developed and maintained in their own repository outside of Todd proper and only brought into Todd when it's packaged for distribution (Binary tarball, deb, rpm, etc).

Todd should also be enhanced to include some additional metadata about testlets that allow you to have several versions of the same testlets for multiple platforms. For instance it's totally valid that someone managing Linux machines and Windows machines might want to write a shell script for Linux and a Batch script for Windows that have the same functionality. The metadata would allow Todd to decide which testllet to distribute under certain circumstances.

[mrlesmithjr]

I have to say I really agree with @bigmstone on developing the testlets in their own repository. As he mentioned keeping ToDD proper clean would be ideal.

[vcabbage]

I agree with much of what's been said here.

One point I do question is the idea of writing the native testlets in Go, then compiling them individually. The problem I see is that, theoretically, ToDD can support any platform/architecture that Go supports. If the model of todd-server distributing testlets to agents is retained for native testlets, todd-server would need to bundle a binary for every supported platform.

This architecture would be reasonable if we were talking about an interpreted language like Python, where the source is what is executed, but I'm not sure it's the best approach for a compiled language.

My suggestion would be to include native testlets in todd-agent. They could still be developed in a seperate repository, if that's desired.

I don't want to get too far into the weeds with implementation details, but I think a rough description would be helpful. A testlet interface could be created, then any type that implemented the interface could be imported and registered into the agent. (Or the imported testlet could register itself, ala drivers for database/sql.)

Example interface:

type Testlet interface {
    Execute(options map[string]interface{}) (results map[string]interface{}, error)
}

Example native testlet:

package ping

type Ping struct {}

func (p *Ping) Execute(o map[string]interface{}) (map[string]interface{}, error) {
    host, ok := o["host"].(string)
    if !ok {
        return nil, errors.New("Hostname invalid or not provided")
    }

    // Ping logic

    results := map[string]interface{}{
        "packet_loss_percentage": packetLoss,
        "avg_latency_ms": avgLatency,
    }

    return results, nil
}

---

I had a slightly different take on one of @bigmstone's points:

In my view Todd should take the approach of lowering the barrier of entry to writing testlets because Todd's value is intrinsically linked to its ability to run these test.

I fully agree with making it easy for a user to define their own testlets. Where I see things a bit differently is that the goal, in my opinion, should be that the builtin testlets are flexible enough to do what the vast majority of users might need, but user defined testlets are available when necessary.

Ideally, if someone writes their own testlet, they would also open an issue with the logic of their testlet. If deemed useful to the community, it would eventually be implemented as a native testlet.

[jedelman8]

Think the original plan was to write testlets in any language assuming they are executable and return JSON. I'm still in favor of that just because I'd like to write them in Python.

[bigmstone]

Think the original plan was to write testlets in any language assuming they are executable and return JSON. I'm still in favor of that just because I'd like to write them in Python.

@jedelman8 I don't think @vcabbage is advocating for removing that functionality, but rather moving the most-used or most-useful Todd tests to native in-client code that executes out of memory and doesn't need to be distributed and compiled for specific platforms.

That said I still lean the other direction from this stance for a couple of reasons.

1. The two most useful tests Todd performs right now are connectivity (ping) and throughput (iperf). We stand to gain little from reinventing those wheels since we can leverage the existing binaries to run those tests. Todd's usefulness is not that it's a great performance tester but that it can distribute those tests and collect the results to/from a wide audience. Stay laser focused on that goal. Don't try to do something better than projects that have been around for years to solve those problems. If you come back with the argument that we can distribute the right binary with the native code to control it then that's a circular argument because of the original stance on compile languages. (Disregarding the fact that non-compiled languages are absolutely hell to distribute)
2. Keep Todd pure. Todd tries to solve a large enough problem on its own. Distributing and collecting results. That problem deserves all the time/effort you can give it and you only give up a small amount of file system space on the server to store multiple versions of a binary. This isn't complex to manage. It just takes some metadata around the testlet to describe its target. This is true of interperated languages too. We've all bee in a situation when running OS level functions in a language like python where the walls break down because of implementation details. Requiring multiple versions of a single application to support linux/windows/etc. So this functionality is desirable regardless of the type of testlet being distributed. If you have to build that scaffolding regardless then you might as well not pollute the code base with tests and keep them where they belong - in another repository.

[vcabbage]

@jedelman8 @bigmstone Correct, I'm not advocating for removing the ability to use any arbitrary executable. I do wonder how Python would work with the current model. It's all good if you only use standard modules, but once you want to pull in anything 3rd party it becomes problematic. Certainly not an insurmountable problem, in fact, the way stackstorm handles packs might be a a good place to get inspiration from.

---

@bigmstone's points are quite valid, we just have differing view points.

I suppose the best way to summarize my view is to quote a Go proverb, "a little copying is better than a little dependency." I always want software to be as easy to use and stable as possible. Every external dependency (server, library, interpreter, filesystem location, etc) is another friction point. I don't know if there are testing frameworks for bash scripts, but it doesn't sound fun.

Just about every system I've worked on has ping, but they don't have the same ping. So you can have seperate scripts for systems that diverge. Which you then need to maintain. Some implementations of ping have more features than others. Do all the scripts have to be written for the lowest common denominator or do you they differ in options? Can you still run a ping test with Windows and Linux systems in the same group?

I assume iperf is the same on all systems. At least when they're running the same version. But few systems I've worked with have iperf installed by default. That certainly can be left up to the user.

As @bigmstone said, every decision comes with trade-offs.

---

Ultimately, I think the best course of action is to look at how you want the user to interact with ToDD. Ignore the "how" and just design what it should look like to the user. Once this is done you can work backward to implement the functionality. Typical top down design process.

[bigmstone]

I can agree with everything you say there, and you're right - we're both standing on the same mountain but one of us is looking left and the other is looking right. There's no wrong decision here - just trade offs.

I'm of the opinion that Todd's role is to solve the distribution and collection problem of running existing or custom made tests. Not the actual testing itself. If that's the case then it's left to the user to figure out how they want to handle divergence in ping implementation. To quote the Unix Philosophy "do one thing and do it well." I see the addition of testing inside of Todd a departure from that philosophy and trying to do things outside of Todd's original scope. Once you start eating the world where do you stop?

[bigmstone]

Also I wish someone would apply the "once you stop eating the world where do you stop" philosophy to SystemD.

[jedelman8]

Right. One of Matt's bullets stated writing testlets in go. Was not @vcabbage.

On May 12, 2016, at 10:35 PM, Kale Blankenship [email protected] wrote:

@jedelman8 @bigmstone Correct, I'm not advocating for removing the ability to use any arbitrary executable. I do wonder how Python would work with the current model. It's all good if you only use standard modules, but once you want to pull in anything 3rd party it becomes problematic. Certainly not an insurmountable problem, in fact, the way stackstorm handles packs might be a a good place to get inspiration from.

@bigmstone's points are quite valid, we just have differing view points.

I suppose the best way to summarize my view is to quote a Go proverb, "a little copying is better than a little dependency." I always want software to be as easy to use and stable as possible. Every external dependency (server, library, interpreter, filesystem location, etc) is another friction point. I don't know if there are testing frameworks for bash scripts, but it doesn't sound fun.

Just about every system I've worked on has ping, but they don't have the same ping. So you can have seperate scripts for systems that diverge. Which you then need to maintain. Some implementations of ping have more features than others. Do all the scripts have to be written for the lowest common denominator or do you they differ in options? Can you still run a ping test with Windows and Linux systems in the same group?

I assume iperf is the same on all systems. At least when they're running the same version. But few systems I've worked with have iperf installed by default. That certainly can be left up to the user.

As @bigmstone said, every decision comes with trade-offs.

Ultimately, I think the best course of action is to look at how you want the user to interact with ToDD. Ignore the "how" and just design what it should look like to the user. Once this is done you can work backward to implement the functionality. Typical top down design process.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[bigmstone]

Ah yeah. This was for native testlets that get "packaged" with Todd. Not a design decision that gets forced on everyone else.

[jedelman8]

Oops. My bad.

On May 13, 2016, at 7:21 AM, Matthew Stone [email protected] wrote:

Ah yeah. This was for native testlets that get "packaged" with Todd. Not a design decision that gets forced on everyone else.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

[Mierdin]

I am still pondering this thread, but just wanted to thank you all for the awesome discussion. Lots of really cool ideas and points.

[Mierdin]

Just realized I didn't directly respond to @jedelman8 - the explanations provided thus far have been correct. I don't want to compromise on the loosely coupled interface provided by testlets, and any move to writing "default" testlets in Go should be done with care to preserve this. The interface example provided by @vcabbage should help with this - that was more or less a Golang version of the testlet input/output "standard".

[Mierdin]

An update on this discussion - I've decided to integrate multiple parts of this discussion into a single solution.

Note that this only applies to testlets - fact collection will be addressed elsewhere, and frankly will likely just be built directly into the agent (since I don't think fact collection has nearly the extensibility requirement that testlets do)

@vcabbage echo'd the Go proverb "a little copying is better than a little dependency". I took this to heart when looking at the existing bash testlets. Especially for simple things like "ping", I have started implementing testlets in Go. https://github.com/Mierdin/todd/pull/80 actually removes the "ping" testlet from the repository, as that's been implemented here instead.

My first approach was going to integrate this testing into the agent itself. I since decided this wasn't the best approach, for a few reasons:

• Though it's possible to enforce the "testlet standard" through Go interfaces and the like (Thanks @vcabbage for the recommendations there) there is no better reference example than to build these as separate binaries. Anyone that wants to build their own can plainly see that we're taking our own medicine. I felt that implementing testlets in the same way we're telling users to will provide a more seamless experience
• @bigmstone was right in saying that the primary goal of ToDD is the distribution and orchestration of tests and the collection of data - not the testing itself. In order to maintain this focus, as well as avoid creating a large failure/problem domain, the testlets are implemented in their own repositories, and stand alone - again, as reference implementations of the testlet standard.

BTW, the old code that contains all kinds of infrastructure for in-agent testing still exists in the toddproject/archive repo if we decide to go in that direction in the future - nothing is set in stone.

Hope that makes things clear. I'll keep this thread open for a little longer, because the fact collector issue is still technically unresolved.

[Mierdin]

Closing this issue, as I believe the discussion is finished. Any remaining tasks will be tracked in #112