Allow setting user conf path through environment variables

[ScottWales]

Teach the uiserver about two environment variables:

$CYLC_CONF_PATH (default ~/.cylc) sets a path to read the uiserver configuration from, and also allows USER_CONF_ROOT in the server configuration to be updated by users. This variable matches the behaviour of cylc-flow.

$CYLC_UISERVER_LOG_PATH (default ~/.cylc/uiserver) sets the output path for Cylc uiserver logs. These logs are created before the jupyter configuration is loaded so are unable to be configured there.

Closes #489

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

There are a couple of differences between this PR and the cylc-flow use of CYLC_CONF_PATH:

1. In cylc-flow, CYLC_CONF_PATH overrides both the user and site configuration paths, not just the user one.
2. In cylc-flow, CYLC_CONF_PATH is flat, i.e. it's $CYLC_CONF_PATH/global.cylc not $CYLC_CONF_PATH/flow/global.cylc.

If the aim is to configure Cylc UI Server defaults for users, then you'll want to use the site-configuration to do this, (e.g. c.CylcUIServer.logging_config or something similar) CYLC_CONF_PATH won't help if it turns off the site-config (note the site-config path is already configurable via CYLC_SITE_CONF_PATH).

[ScottWales]

Thanks @oliver-sanders, I didn't notice CYLC_CONF_PATH did a full override.

Rather than user configuration I'm wanting to relocate all the other files that the uiserver creates in the home directory under USER_CONF_ROOT, as that's going to be read-only in our production ui server. Some paths can be modified by the configuration but others cannot, for instance the log file path and INFO_FILES_DIR.

What do you think of the same functionality under a different environment variable name, say CYLC_UISERVER_RUNTIME_DIR?

[oliver-sanders]

Some paths can be modified by the configuration but others cannot, for instance the log file path and INFO_FILES_DIR.

Note, the "log file path" location CAN be changed, just not with the rotating log file handler you linked to.

Would be open to configuring that path via the logging_config (where it's already provided) though.

I'm wanting to relocate all the other files that the uiserver creates in the home directory under USER_CONF_ROOT, as that's going to be read-only in our production ui server

I'm not really sure what doing that would achieve? I'm guessing that the intention here is to freeze the user's configuration, preventing them from changing anything, which doesn't really make sense for a single-user application which runs entirely in userspace:

• If it's to prevent accidental edits to the UIS configuration, that would make sense, although there are easier ways to achieve this (e.g. a site-config).
• If it's about security, then I don't think this will help, the UIS runs in userspace which the user has control over.

[ScottWales]

It's an imposed security constraint from our systems people, the web server VM running Cylc hub has /home mounted read-only to limit the impact of a compromised web server. Interactions with the Cylc server VM have to happen over the api.

[hjoliver]

, the web server VM running Cylc hub has /home mounted read-only

@ScottWales you mention the hub here, but from your description above, the reason for this PR is to relocate the Cylc UI Server logs? The UI Server does not have to (in fact, is not expected to) run where the (privileged) hub runs.

The UIS needs to run (as the user) where it has access to user-owned scheduler DBs and log files, so that it can serve critical offline data.

Note also the UI Server log is not where workflow interventions are logged with user attribution (if other authenticated users have been authorized by the owner to do that) - that goes to the scheduler logs. The UI Server log is more of a technical base-level Jupyter Server thing, which underlies the Cylc functionality.

Does this change anything, or have I misunderstood your intent here?

[ScottWales]

Ok, that's interesting - we have the uiserver being spawned on the same host as hub using sudospawner-singleuser. How is your configuration spawning uiserver?

[hjoliver]

Ok, that's interesting - we have the uiserver being spawned on the same host as hub using sudospawner-singleuser. How is your configuration spawning uiserver?

We have thus far not deployed Cylc 8 with Jupyterhub. Same at MO I believe. For the moment we have told users to run cylc gui at the terminal, which starts a UI Server and prints a token-protected URL for the user. "Single user mode" is good for everything except cross-user visibility and the authorization layer.

Hub deployment is coming soon but the team has still had higher-priority issues to knock off first. It's not a concern, Jupyterhub is proven tech. It is used to spawn notebook servers on other hosts all the time. There's are a bunch of hub spawners available, to support different ways of starting servers on other hosts. On the team, we have tested a few of these in the past. All that needs to be done, I think, is to see what spawners are available, choose one that is appropriate for your environment, and test it out. Worst case scenario we might need to modify an existing spawner or perhaps even make a custom one, but that should not be a big job.

[oliver-sanders]

It's an imposed security constraint from our systems people, the web server VM running Cylc hub has /home mounted read-only to limit the impact of a compromised web server. Interactions with the Cylc server VM have to happen over the api.

This makes sense for the Hub (i.e. jupyter hub) which should run under a special user account, but this does not make sense for the UI Server which runs as the user.

• The Hub
  • A Jupyter Hub instance.
  • Spawns servers on behalf of users.
  • Runs a reverse-proxy allowing users to connect to these workflows using a friendly URL.
  • Does NOT manage workflows, meddle with user accounts, etc, it just spawns servers.
  • Recommend running this under a special user account, and spawning user's servers using sudo (guide). Otherwise can be run under root.
• The UI Server
  • A Jupyter Server instance with the Cylc extension loaded.
  • (Note Jupyter Lab is also a Jupyter Server extension like the Cylc UI Server so both can be run on the same server if desired).
  • Runs as the user.
  • May monitor/control/run that user's workflows.

The cylc hub command is a wrapper for the jupyterhub command, the wrapper loads this configuration file which helps with Cylc integration. You don't need to use the cylc hub command, you could use jupyterhub and configure it yourself, the wrapper is only there for convenience.

Jupyter Hub is not a Cylc product, so I'm not in a good place to advise with this, they do have a very active forum where you could ask about this sort of read-only configuration. Like most systems, Jupyter Hub does need to be able to create files (database and possibly log files, etc) so you won't be able to go completely read-only.

We have thus far not deployed Cylc 8 with Jupyterhub. Same at MO I believe

We haven't deployed Jupyter Hub ourselves yet, but don't let that stop you, it's ready to go, we just haven't had the time yet. Note deploying Jupyter Hub + Cylc UI Server is no different to deploying Jupyter Hub + Jupyter Lab.

I would strongly recommend waiting for cylc-uiserver 1.4.0 which we will release later today with a spot of luck. This upgrades the Cylc UI Server to work with the latest versions of Jupyter components. Watch out for an announcement here: https://cylc.discourse.group/t/cylc-8-2-0-released/702/3

[oliver-sanders]

Ok, that's interesting - we have the uiserver being spawned on the same host as hub using sudospawner-singleuser

Servers can be spawned in a variety of ways, it is possible to spawn them onto other hosts. E.G. see the "SSH Spawner".

https://jupyterhub.readthedocs.io/en/stable/reference/api/spawner.html

We would like to spawn our servers onto a pool of VMs using Cylc's load balancing (and possibly auto-migration) functionalities to make managing larger deployments easier.

The idea is to write our own spawner which uses Cylc's load balancing, then spawns the server over SSH. Work was started in this direction, however, it has fallen down our priority ladder at the moment. If this is of more imminent interest to you, we would be happy to share our experience and collaborate on it. See #213