Do we really need all the avocado-runner-* sub-commands?

[beraldoleal]

To be honest, I'm opening this discussion, because I'm trying to find an good excuse to have all the sub-commands that we have, and probably I'm missing something.

Today we have a few commands starting with avocado-runner-*:

avocado-runner                       avocado-runner-golang                avocado-runner-tap
avocado-runner-avocado-instrumented  avocado-runner-noop                  avocado-software-manager
avocado-runner-exec                  avocado-runner-python-unittest       
avocado-runner-exec-test             avocado-runner-robot   

I undestand that we are trying to make the runners standalone applications and that is fine for now. But I would like to know if it is just me that find this following command redundant and confusing:

$ avocado-runner-exec-test runnable-run -k 'exec-test' -u /tmp/test.sh

IMO, this is a little confusing and increases the learning curve for our users. I would prefer to hide the internal details from them. I'm not sure if users should know what is a 'runnable'.

I also understand that we need this as a way to serialize and deserialie the runner options when running in a different machine, and this is made by the command line. But if this is the only case, then, I would like know if would be possible to have only the "recipe" file and a simple command to run tasks from this recipe, leting the command-line clear.

But, like I said, probably I'm missing something. What it is?

[clebergnu]

One of the reasons is that the Avocado job can find suitable runners based on the executable names. Think about runners written in languages other than Python that will not register entry points, nor have Python modules matching a certain name.

But, I've meant to simplify that use case, and remove that duplication for a long time. This issue was the missing bit, so thanks! See: #4292

[beraldoleal]

Thanks, @clebergnu , but I'm still not sure if adding a default 'kind' here will make this less confusing.

We will still have a lot of sub-commands like "avocado-runner-exec-test runnable-run". I'm still not sure if we need to expose this as a command line for our users.

I'm still questioning why we don't remove the commands lines and keep only a simpler/clear way: "from recipe".

avocado-runner -r <recipe_file.json>

and this will call the runner API to run the proper test.

[willianrampazzo]

I agree it may be confusing to have lots of avocado-runner-* commands. My personal preference would be one runner manager entry point dispatching the correct runner kind. After @clebergnu comment, I could think of one way of doing that: having the runners located in one single directory and the runner manager where the runners are located now. PR #4292 would still be valid, but we would not need all those runner entry points except for the manager.

[clebergnu]

Thanks, @clebergnu , but I'm still not sure if adding a default 'kind' here will make this less confusing.

We will still have a lot of sub-commands like "avocado-runner-exec-test runnable-run". I'm still not sure if we need to expose this as a command line for our users.

I don't think 0.01% of users will interact with those runners, do you? On the other hand, comparing to the previous runner, it's almost impossible to reproduce what the Avocado runner will effectively run, and the reason for having those interfaces is to be able to replicate that exactly. This makes debugging much simpler.

I'm still questioning why we don't remove the commands lines and keep only a simpler/clear way: "from recipe".

avocado-runner -r <recipe_file.json>

and this will call the runner API to run the proper test.

OK, I think I get your point. I like the apparent simplification, but, this proposal would:

1. remove the need to parse a lot of the command line (yay!), BUT:
2. move the "sub command" selection to inside the recipe file

WRT 2, I see a few drawbacks:

a) one will always need to have one additional file being sent to the execution environment
b) avocado-runnner will now have to read and parse the recipe file, and then it may bail out because it can not execute sub command "foo" -- this is currently handled by the command line transparently.
c) not all sub-commands can be eliminated. For instance "capabilities" is very useful to the upper layers

WRT "a", maybe we can make avocado-runner read from a file descriptor (like STDIN), but it sounds a bit more fragile than the command line.

Anyway, this is worth discussing, but I feel like we'd just be moving the complexity to a different place.

[clebergnu]

I agree it may be confusing to have lots of avocado-runner-* commands. My personal preference would be one runner manager entry point dispatching the correct runner kind. After @clebergnu comment, I could think of one way of doing that: having the runners located in one single directory and the runner manager where the runners are located now. PR #4292 would still be valid, but we would not need all those runner entry points except for the manager.

So what you're proposing is similar to /usr/bin/git and /usr/libexec/git-core/*. It's kind of the reversed solution we currently have here. The upside of our current approach is that a avocado-runner-qemu-iotest runner could live in the project's own source/build tree and "just work".

[beraldoleal]

Thanks, @clebergnu , but I'm still not sure if adding a default 'kind' here will make this less confusing.
We will still have a lot of sub-commands like "avocado-runner-exec-test runnable-run". I'm still not sure if we need to expose this as a command line for our users.

I don't think 0.01% of users will interact with those runners, do you? On the other hand, comparing to the previous runner, it's almost impossible to reproduce what the Avocado runner will effectively run, and the reason for having those interfaces is to be able to replicate that exactly. This makes debugging much simpler.

That is my point. If they will not interact, why are we exposing this? I'm thinking as an end-user, and this is not good, this could be an entry barrier. If users will not need it, but when typing avocado<TAB><TAB> will make them curious about what this is. Getting the help for any of those commands will make not a good first impression, in my option.

We could try to separete better what is needed by Avocado internal and what is needed by our users, so we can hide the internal details for them and try to make their life's easier. Still without loose any debbuging information.

I'm still questioning why we don't remove the commands lines and keep only a simpler/clear way: "from recipe".
avocado-runner -r <recipe_file.json>
and this will call the runner API to run the proper test.

OK, I think I get your point. I like the apparent simplification, but, this proposal would:

1. remove the need to parse a lot of the command line (yay!), BUT:

2. move the "sub command" selection to inside the recipe file

Yay! too, right? Because this is the whole point, hide internal stuff from the users.

WRT 2, I see a few drawbacks:

a) one will always need to have one additional file being sent to the execution environment

I can't see yet this as problem.

b) avocado-runnner will now have to read and parse the recipe file, and then it may bail out because it can not execute sub command "foo" -- this is currently handled by the command line transparently.

I lost you here. Not sure if I understood.

c) not all sub-commands can be eliminated. For instance "capabilities" is very useful to the upper layers

Maybe the problem is that we are trying to describe something using command lines outputs and arguments. Maybe a runner could just provide a "json" metadata file, describing the capabilities and all stuff needed by upper layers. Again, just to hide from the users.

WRT "a", maybe we can make avocado-runner read from a file descriptor (like STDIN), but it sounds a bit more fragile than the command line.

Yes, agreed, but like I said, I dont see the problem of sending the file, yet.

Anyway, this is worth discussing, but I feel like we'd just be moving the complexity to a different place.

This is the point: by moving the complexity away from the users we could have simple interfaces and command lines for them and make our lifes as developers easier because when extending runners/runnables we don't need to think on the command-line, only on the 'recipe' part.

[clebergnu]

Thanks, @clebergnu , but I'm still not sure if adding a default 'kind' here will make this less confusing.
We will still have a lot of sub-commands like "avocado-runner-exec-test runnable-run". I'm still not sure if we need to expose this as a command line for our users.

I don't think 0.01% of users will interact with those runners, do you? On the other hand, comparing to the previous runner, it's almost impossible to reproduce what the Avocado runner will effectively run, and the reason for having those interfaces is to be able to replicate that exactly. This makes debugging much simpler.

That is my point. If they will not interact, why are we exposing this? I'm thinking as an end-user, and this is not good, this could be an entry barrier. If users will not need it, but when typing avocado<TAB><TAB> will make them curious about what this is. Getting the help for any of those commands will make not a good first impression, in my option.

The overall discussion is usability, but there are many different smaller topics. The most important ones I can think of:

1. Pros/cons of exposing various "avocado-runner-" executables
2. Pros/cons of having a command line parser for avocado-runners

Those 0.01% of "Avocado users" interacting directly with "avocado-runner-*" will be those that are interested in integrating Avocado to their projects, either by understanding how a runner will perform with a given input or will write their own (either reusing existing runners or from scratch). This has to do with both points 1 and 2 above. This is what I can say about that:

The pros of 1:

• No dependency on specific locations to find runners
• No middle-man
• No additional point of failure
• Trivial to have new runners working

The cons of 1:

• May confuse users that that do "avocado-"

We could try to separete better what is needed by Avocado internal and what is needed by our users, so we can hide the internal details for them and try to make their life's easier. Still without loose any debbuging information.

And what's your proposal here?

BTW, I think Avocado walks a very thin line because some of our users and use cases, will require such internals to be exposed. See the drama on avocado.core.Job, or even here, where an Avocado job (via status server/repo) will expect some data in a given format to come from runners. Is this "data in given format" an internal aspect? Or does it become an interface the moment out "user" is integrating Avocado into his own project? It's the porcelain and plumbing separation that is a bit less obvious when users are developers.

I'm still questioning why we don't remove the commands lines and keep only a simpler/clear way: "from recipe".
avocado-runner -r <recipe_file.json>
and this will call the runner API to run the proper test.

OK, I think I get your point. I like the apparent simplification, but, this proposal would:

1. remove the need to parse a lot of the command line (yay!), BUT:

2. move the "sub command" selection to inside the recipe file

Yay! too, right? Because this is the whole point, hide internal stuff from the users.

WRT 2, I see a few drawbacks:
a) one will always need to have one additional file being sent to the execution environment

I can't see yet this as problem.

One example: the podman runner, we should be able to prepare a container image (with all requirements, runner included) and with that, have a single operation for running a task. The requirement of a file as input for a runner doubles that, and will now require two "podman " executions.

b) avocado-runnner will now have to read and parse the recipe file, and then it may bail out because it can not execute sub command "foo" -- this is currently handled by the command line transparently.

I lost you here. Not sure if I understood.

Right now avocado-runner foo will rely on command line argument handling. It returns very early and it's very little of our own code.

With this change, it would go past the command line handling (which would probably be successful because of the -r recipe) and then our custom code would look at the command in the recipe, find "foo" and say that it's not a command we can handle. It's just moved further into the process and probably requires a bit more logic.

c) not all sub-commands can be eliminated. For instance "capabilities" is very useful to the upper layers

Maybe the problem is that we are trying to describe something using command lines outputs and arguments. Maybe a runner could just provide a "json" metadata file, describing the capabilities and all stuff needed by upper layers. Again, just to hide from the users.

I'd strongly advise against that... because:

• it does not account for dynamic aspects
• it brings the same problems of depending on specific locations for finding the runners themselves. It's like you have qemu in your build tree, and you want to know what accelerators it supports... where do I find the metadata file? Now, what if qemu is installed on /usr/bin, will I drop a metadata file there?

virsh capabilities and even qemu-system-x86_64 -accel ? are examples of why it makes sense to have those information built into the apps themselves.

WRT "a", maybe we can make avocado-runner read from a file descriptor (like STDIN), but it sounds a bit more fragile than the command line.

Yes, agreed, but like I said, I dont see the problem of sending the file, yet.

Anyway, this is worth discussing, but I feel like we'd just be moving the complexity to a different place.

This is the point: by moving the complexity away from the users we could have simple interfaces and command lines for them and make our lifes as developers easier because when extending runners/runnables we don't need to think on the command-line, only on the 'recipe' part.

So this last paragraph is about point 2, and I see the pros:

• requiring less code to be written (only one type of serialization/deserialization)

And the cons:

• will still require some sub-commands (I don't see how to say, drop capabilities without drawbacks)
• will require an extra step test/try out the runners (write recipe file, run the tool)
• will require an extra copy step of the recipe file on some execution environments

Let me know if this analysis make sense to you.

[willianrampazzo]

I agree it may be confusing to have lots of avocado-runner-* commands. My personal preference would be one runner manager entry point dispatching the correct runner kind. After @clebergnu comment, I could think of one way of doing that: having the runners located in one single directory and the runner manager where the runners are located now. PR #4292 would still be valid, but we would not need all those runner entry points except for the manager.

So what you're proposing is similar to /usr/bin/git and /usr/libexec/git-core/*. It's kind of the reversed solution we currently have here. The upside of our current approach is that a avocado-runner-qemu-iotest runner could live in the project's own source/build tree and "just work".

The idea I was proposing does not change much of what we have now. It would remove the entry points and let the runner manager find it. We can always set the runner path with a parameter. This would be a mid-term between what we have now and what @beraldoleal is suggesting as I see one of his concerns is having lots of commands listed when typing avocado-<TAB><TAB>.

Moving the runners to a specific directory is just a matter of organization.

Anyway, I saw you listed No dependency on specific locations to find runners like a pro, so my approach would be a con in your list.

[beraldoleal]

Thanks, @clebergnu , but I'm still not sure if adding a default 'kind' here will make this less confusing.
We will still have a lot of sub-commands like "avocado-runner-exec-test runnable-run". I'm still not sure if we need to expose this as a command line for our users.

I don't think 0.01% of users will interact with those runners, do you? On the other hand, comparing to the previous runner, it's almost impossible to reproduce what the Avocado runner will effectively run, and the reason for having those interfaces is to be able to replicate that exactly. This makes debugging much simpler.

That is my point. If they will not interact, why are we exposing this? I'm thinking as an end-user, and this is not good, this could be an entry barrier. If users will not need it, but when typing avocado<TAB><TAB> will make them curious about what this is. Getting the help for any of those commands will make not a good first impression, in my option.

The overall discussion is usability, but there are many different smaller topics. The most important ones I can think of:

1. Pros/cons of exposing various "avocado-runner-" executables

2. Pros/cons of having a command line parser for avocado-runners

Those 0.01% of "Avocado users" interacting directly with "avocado-runner-*" will be those that are interested in integrating Avocado to their projects, either by understanding how a runner will perform with a given input or will write their own (either reusing existing runners or from scratch). This has to do with both points 1 and 2 above. This is what I can say about that:

As an end-users I would like to have simple logic: "run this command/test", I don't need to understand how or if is a runnable or a task, users should don't care. Those are the users that I think we are loosing because of the complexity. We are forcing them to understand internal concepts just to have they tests running.

As a avocado contributor (or advanced user), if I would like to write my own runner (0.01%), IMO, the proper place to read this would be "Avocado Contributor Guide".

The pros of 1:

* No dependency on specific locations to find runners

* No middle-man

* No additional point of failure

* Trivial to have new runners working

The cons of 1:

* May confuse users that that do "avocado-"

To have a better overview here, we should better describe each of those topics. I also think that we have more than one cons, like I said before.

But I'm sorry if in the middle of the discussion I gave the impression that I was talking about the commands itself. I'm more concerned right know about the subcommands. And to avoid having a too long and broader discussion I would like to focus on the sub-commands.

We could try to separete better what is needed by Avocado internal and what is needed by our users, so we can hide the internal details for them and try to make their life's easier. Still without loose any debbuging information.

And what's your proposal here?

I'm trying to understand the the big picture first, and understand the requirements. I'm just pointing that from the user perspective this is not good. What I could tell from this thread, is that I'm not alone here. And we are all avocado developers.

BTW, I think Avocado walks a very thin line because some of our users and use cases, will require such internals to be exposed. See the drama on avocado.core.Job, or even here, where an Avocado job (via status server/repo) will expect some data in a given format to come from runners. Is this "data in given format" an internal aspect? Or does it become an interface the moment out "user" is integrating Avocado into his own project? It's the porcelain and plumbing separation that is a bit less obvious when users are developers.

Yes, we have advanced users and very good developers using avocado. But sometimes we forgot that those users don't need a complex framework to run the tests. We need to better document our interfaces and what we expect from "contributors", but letting our users (even the advanced ones) away from the complexity.

I'm still questioning why we don't remove the commands lines and keep only a simpler/clear way: "from recipe".
avocado-runner -r <recipe_file.json>
and this will call the runner API to run the proper test.

OK, I think I get your point. I like the apparent simplification, but, this proposal would:

1. remove the need to parse a lot of the command line (yay!), BUT:

2. move the "sub command" selection to inside the recipe file

Yay! too, right? Because this is the whole point, hide internal stuff from the users.

WRT 2, I see a few drawbacks:
a) one will always need to have one additional file being sent to the execution environment

I can't see yet this as problem.

One example: the podman runner, we should be able to prepare a container image (with all requirements, runner included) and with that, have a single operation for running a task. The requirement of a file as input for a runner doubles that, and will now require two "podman " executions.

Why we couldn't add this recipie as a requirement on the container image? Why do we need two "podman" executions? Or why we couldn't have mount volumes as we discussed before? Or in a more futuristic/async scenario: why we dont bring the container up and send the json recipie via socket/event/pub&sub?

b) avocado-runnner will now have to read and parse the recipe file, and then it may bail out because it can not execute sub command "foo" -- this is currently handled by the command line transparently.

I lost you here. Not sure if I understood.

Right now avocado-runner foo will rely on command line argument handling. It returns very early and it's very little of our own code.

With this change, it would go past the command line handling (which would probably be successful because of the -r recipe) and then our custom code would look at the command in the recipe, find "foo" and say that it's not a command we can handle. It's just moved further into the process and probably requires a bit more logic.

I'm not talking about removing the avocado-runner-foo, just to have less subcommands.

c) not all sub-commands can be eliminated. For instance "capabilities" is very useful to the upper layers

Maybe the problem is that we are trying to describe something using command lines outputs and arguments. Maybe a runner could just provide a "json" metadata file, describing the capabilities and all stuff needed by upper layers. Again, just to hide from the users.

I'd strongly advise against that... because:

* it does not account for _dynamic_ aspects

* it brings the same problems of depending on specific locations for finding the runners themselves. It's like you have `qemu` in your build tree, and you want to know what accelerators it supports... where do I find the metadata file?  Now, what if `qemu` is installed on `/usr/bin`, will I drop a metadata file there?

I used the metadata file as example, but could be something registered by events and sent via sockets.

virsh capabilities and even qemu-system-x86_64 -accel ? are examples of why it makes sense to have those information built into the apps themselves.

I don't think those are good examples for nice/simple end-users interfaces. ;) Besides that, virsh/qemu users needs those informations, they are familiar with that. Avocado users should not be aware of "runnable"/"tasks"/"kind" to run a test/command.

WRT "a", maybe we can make avocado-runner read from a file descriptor (like STDIN), but it sounds a bit more fragile than the command line.

Yes, agreed, but like I said, I dont see the problem of sending the file, yet.

Anyway, this is worth discussing, but I feel like we'd just be moving the complexity to a different place.

This is the point: by moving the complexity away from the users we could have simple interfaces and command lines for them and make our lifes as developers easier because when extending runners/runnables we don't need to think on the command-line, only on the 'recipe' part.

So this last paragraph is about point 2, and I see the pros:

* requiring less code to be written (only one type of serialization/deserialization)

And the cons:

* will still require some sub-commands (I don't see how to say, drop `capabilities` without drawbacks)

* will require an extra step test/try out the runners (write recipe file, run the tool)

* will require an extra copy step of the recipe file on some execution environments

Let me know if this analysis make sense to you.

I think that is a nice discussion and I'm learning a lot, first of all, thanks for you patience. But at same time I feel that I will not be able to draw a proper solution without the big picture. You have bias already because you are familiar with the problem and with the current code. And I’m here playing the devil’s advocate. ;)

IMO, if we really need to improve here (and I strongly think that we should not only for our users but for the code maintanability), we should make a step back and describe the problem with the requirements, so we could try to find a common ground.

We could try to discuss this in meetings if is taking so many replies.

[pevogam]

Just a quick chime in here on my "vanilla" side. As a not so advanced user that doesn't know all the blows and whistles of the Avocado command line, I can say I don't understand most of options listed by @beraldoleal when opening this issue 😅 Not sure if this is useful to anyone at all but just adding it here in case it is a super small form of feedback.