Enhanced Client Milestone

[rstyd]

We need to work out a plan for the enhanced client.

Currently we have two options:

1. Repurpose an existing workflow submission application
2. Hack together our own

For the exiting workflow submission tool side, Tim has mentioned we could work with SAW. Tim said they've been looking to add CWL support to their tool. This has a bundled editor which is nice. The one issue with this approach is it will require bolting on the capabilities to submit and manage BEE jobs. We could accomplish this with an Eclipse plugin, but the interface would probably be pretty clunky just from my experience with Eclipse.

There's also Rabix composer. It's currently no longer supported, but with an Apache 2.0 license, I believe we could make a fork and enhance it with the features we'd need. This could be a very powerful solution since it includes a built in editor and facilities for visualizing workflows. We could reuse the workflow visualization to add visualizing the current state of a workflow running on a system. The big downside to this approach is we'd be inheriting a legacy code base that we'd responsible for which is not great.

Finally, we could consider putting together our own client. From my research so far, the best solution appears to be an electron app. Rabix composer is also based on electron. Electron is a javascript based API for developing native desktop apps. This would be the most flexible solution, but it could require a lot of work. It still may be less work than Rabix would be. The other issue which we'll have to look at with rabix composer as well is that electron apps are known for using a ton of ram. We'll have to see if this is an issue. There may be ways to mitigate it.

Let me know if you have any other ideas.

[jtronge]

One other possibility, given that we already include Flask as a dependency, is to set up some sort of web interface, perhaps as a third component. I believe that Flask includes Jinja as a template engine which makes developing web front ends super easy (I think Jinja might already be one of our dependencies). We might not even need to create another Flask daemon, we could just integrate the client API into the WFM interface. There are also some nice abstractions that Flask provides that could be useful if we wanted to do something like that (Blueprints). As for visualizing the DAG, I'm not sure but we might be able to include some sort of JavaScript library that could be run in the browser (no need for node.js).

That's just an idea though, I don't know if a web interface is exactly what we need here.

[rstyd]

I was actually thinking about that as well, and we should definitely consider it. We already have flask as a dependency so just use that. The feature set is very strong. My thought process for an electron app is that we could release just the executable without requiring any dependencies. For a flask app to run on the users desktop, it requires a bit of setup and then also there needs to be the daemon running as well. My hope is we can come up with a plug and play solution so the users just download an executable and can start running. There might be a way to do that with just flask.

[jtronge]

I get what you're saying. Installing an electron app is definitely a lot easier than installing BEE with poetry and a bunch of dependencies. Wouldn't we want to already set up a connection to whatever system they are planning launch jobs on though? That way the WFM would already be running in that case. Or is this just more of a static visualization client?

[rstyd]

That's a great question, and I think we'll need to discuss it more. I think we'd want it to be static.

[rstyd]

I’ve compiled a list of some questions for the CLI backend.

What server is the workflow manager running on? Is it a LANL HPC cluster, Darwin, summit, cloud resources, or some other resource?

This will need to be saved in the configuration options for the client. Our approach for connecting to the machine will vary depending on the resource needed.

How will we send commands to the workflow manager running on a server from the client?

For internal LANL usage at least, we’ll need users to have kerberos authentication enabled on their machines. I think the easiest way to handle this is to leverage ssh. If a user is logged in via kerberos, LANL machines are already configured to send the kerberos ticket with each ssh command. So, we can send commands using ssh user@machine command. We could install a client routing application in the users .local/bin that would forward commands from ssh to the workflow manager. So, the client would send something like ssh user@machine bee_forward submit and that would get packaged into an HTTP request to the WFM. Commands that require files will either need to use redirection to send the file over stdin like ssh user@machine bee_forward < /path/to/file, or we can just scp the file over. This might be a terrible approach. I'd really appreciate some other ideas.

How do we get data from the workflow manager to the CLI?

The aforementioned approach has a pretty big limitation which is that we can send commands to the workflow manager, but we can’t easily get data back. We could use scp for this purpose, but scping repeatedly seems like a bad idea. We'll have tons of small files that will have to go somewhere and write a bunch of stuff to a network file system which is asking for trouble. ssh's command functionality returns the stdout and stderr of the command. We could use this by having the forwarding application print a jsonpickle object to stdout upon completion. Then, the CLI will take the json and re-encode it. We’ll need the object definition on both sides, but it’s very doable.

What do we do about latency?
There's going to be a decent amount of latency to deal with. We're going to need the CLI to run requests using a multiprocessing approach so the client doesn't just hang while requests are sent out. It'd probably be good enough to have some sort of loading bar while a request goes through. This should be supported by whatever framework we end up using.

[jtronge]

SSH seems like a good way to go for this. For the cloud launching code, I use SSH to set up a tunnel between the machine the WFM is running on and the remote instance that the TM is running on. This lets us use the same REST API, with data going in both directions. I wonder if we might be able to do something similar here, letting the new client connect to the WFM through its REST API.

[rstyd]

Here are some questions about the client frontend.

What will it look like?
We'll need to design the GUI. A wireframing tool will be really helpful for this. I've been looking at miro.

What framework will it use?
@jtronge is looking at electron. We'll see what he thinks about it and go from there. There are several caveats about electron (tends to have larger executables and can be ram inefficient, but it seems to currently be the best way to write a cross-platform GUI application with minimal effort. Most other frameworks like react native, tauri, and proton native either don't support platforms we care about (Mac, Linux) or are too early in development.

What will the functionality be like?
I've started a google doc where we can flesh this out. There is no content yet.
https://docs.google.com/document/d/1Cp5pjOqEygwWyTtJ5eJLAFG1Klh-6cS4hvcc3pVegS0/edit?usp=sharing

[pagrubel]

The minimum to meet the milestone is to have a gui that does what the present client does and to allow the user to visualize the current state of their workflow as a DAG.

However, do we want to add the capability to build a cwl? Or at least plan for it?

[pagrubel]

I know this should most likely be in a later timeframe, but wanted to mention it while thinking of it. In the design document @rstyd mentions sending workflows to any location. Should we also have some means of setting up those locations from the client, such as start the tm, wfm and scheduler, or any other modules needed. Or at least verify they have been started.