The role of the UI server in production from a user (and permissions) point of view (cylc8)

Cylc Cylc Development
21

No problem, thanks for posting your findings, that’s good to know.

22

Oliver,
Here is another question we have regarding user access to other users workflows. Involves using a sudo user.

  • Shared application user – we run our flows as a nologin type application user (APPUSER1) and we want that user to grant permissions to users that are in the associated group (APPGROUP1) so that they can control the flows for appuser1 through the uiserver. Appuser1 does not have a login so we cannot start a cylc hubapp process as appuser1 by logging into the UI. The individual users can sudo to run commands as appuser1, so it is possible to start it up with something like “sudo appuser1 cylc hubapp”?

Trying this I get:

# su APPUSER1 -c “module load cylc-test;cylc hubapp”

[I 2023-08-08 18:37:13.017 CylcHubApp mixins:547] Starting jupyterhub single-user server version 4.0.1

[I 2023-08-08 18:37:13.017 CylcHubApp mixins:561] Extending cylc.uiserver.hubapp.CylcHubApp from cylc

[I 2023-08-08 18:37:13.017 CylcHubApp mixins:561] Extending jupyter_server.serverapp.ServerApp from jupyter_server 1.24.0

[I 2023-08-08 18:37:13.042 CylcHubApp manager:344] cylc.uiserver | extension was successfully linked.

[D 2023-08-08 18:37:13.046 CylcHubApp application:1028] Exiting application: jupyter-server

JUPYTERHUB_API_TOKEN env is required to run jupyterhub-singleuser. Did you launch it manually?

Can you point us at some way to make this work?

23

Spawning servers for nologin accounts is a problem which we will also face.

Jupyter Hub has its own authorisation layer. One of the things you can do with this is to delegate users the permission to start each others servers. This avoids the requirement for a user to have already started their server before another user can access it. This should resolve the issue for nologin accounts.

Unfortunately, there are two hitches:

  1. This requires a newer version of certain Jupyter components than the Cylc UI Server currently supports. I’m actively working on this one, we should have a new release out in the next few weeks with some revised documentation for configuring multi-user access under the new authorisation regime.
  2. Jupyter Hub appears to require a user to have logged in before it will spawn a server for them, even if it was another user who requested the server to be spawned. This will require investigation, it’s possible that all we need to do is insert a placeholder row for the user in the database. Otherwise we may need to propose some changes to Jupyter Hub.

Alternatively, there may be other authenticators you could use to side-step the issue.

24

Oliver,
Got the response via Mr. Hillier to our “official” request to the MetOffice for assistance with configuration of Cylc 8. I can refine our questions a bit from the Reponses to that.

*> The Cylc GUI can be deployed in two ways: *
> 1. As a standalone single-user application like the Cylc 7 GUI.
> 2. As a multi-user Jupyter Hub application (new in Cylc 8).
*> *
> You’re quite a bit ahead of the Met Office with “cylc hub”, JupyterHub is a neat way of deploying the GUI, but is not strictly necessary. At the Met Office, we have yet to deploy the multi-user setup, our users are all running their own single-user GUI applications. This is the simplest way to get up and running with the new GUI.
*> *
> If your users are able to run the “cylc gui” command as the “auser” using sudo (or alternative), then this approach should work for you too.
> • The user starts the GUI server as “auser” via sudo (or alternative).
> • By default, the GUI will open in a web browser running under the “auser” account.
> • Alternatively, the user can use the “–no-browser” option, and open the URL (printed to the terminal) as themselves.
> It is also worth being aware of “cylc tui” which is a command-line utility for monitoring and controlling workflows which works out of the box with no configuration needed.

We can not use simple X11 forwarding, which you suggested via “cylc gui”. We are each behind a firewall on either side of the tunnel this runs across. HTTPS is the only approved protocol we can use to traverse this other than ssh. Also, performance has been very poor for X11 forwarding via firefox over ssh when we have attempted it. We have tried the TUI and it works with very simple workflows but fails to operate with more complex workflows like we use in actual production environments.
Uploading: image.png…

A recent development in Jupyter Hub allows you to delegate users the permission to start other users servers. This is extremely helpful for working with those “auser” accounts as it removes the requirement to authenticate with Jupyter Hub as the auser account. We are currently working to upgrade the Cylc UI Server to work with the latest versions of various Jupyter components to pick up this new functionality. This work is currently under review, we expect to release Cylc UI Server version 1.4.0 with these changes over the next few days if all goes well.

If pushing ahead with a Jupyter Hub setup I would recommend waiting for this release (we’ll announce it with upgrade instructions here Cylc 8.2.0 Released - #2 by wxtim - Cylc Announce - Cylc Workflow Engine ).

Our biggest issue is being able to spawn a UI Server as the auser (sudo) and have it be multi-user capable where by a user with the correct permissions can play/stop/pause a workflow on that UI Server. It sounds like you are stating that we may be able to make this work with some tweaks to our JupyterHub settings via this upgrade to cylc1.4 UI Server? Can you give us a firmer date where by this new update will be released? We would be interested in field testing it for you.

The other issues with the load balanced spawner and such can wait until after we solve the multi-user UI Server that runs as the no-login sudo user. That is keeping us from being able to transition to Cylc 8 and is the biggest issue at hand.

25

There’s no need to run a browser over X11 forwarding, we definitely would not recommend that.

The cylc gui command starts your UI Server and prints a secure URL (protected with a one-off token) to the terminal. You can connect to that with your local browser, given the same network routing needed for doing it via hub.

Yes, TUI is still considered to be “experimental” (hence the prominent warning at the top of it). The problem is we haven’t yet hooked it up to the much more efficient incremental data subscription that feeds the web UI. When that’s done we believe TUI will be able to handle very big workflows too.

26

Yes you should be able to achieve this. We can get the uiserver 1.4 release out next week probably. The main pull request that is needed is setup: migrate to jupyter server 2 by oliver-sanders · Pull Request #450 · cylc/cylc-uiserver · GitHub; if you are able to do a pip install from that branch you could test it out sooner. The relevant docs will also be updated after that but you can preview them here: https://github.com/cylc/cylc-doc/blob/ecdb5489990d39ee5d7a801a9768b68333b93797/src/user-guide/sharing-access-to-workflows.rst. Thank you!

27

OK, let me confer with the group and see what we can make of these updates.

28

Let me have the guys take a look at that and I’ll provide feedback.

29

Thanks for the preview and insight into what needs to be done to make this work. Would it be possible to have either you (Ronnie) or Oliver join a Teams call Tuesday (1830 your time, 1230 our time ) so we could discuss some of this? If so, please respond with an email address you are comfortable me sending it to.

30

Hello,

Happy to join you in a call, however, I think it might be a bit premature until we’ve got the new release out which will include documentation covering the basics of Jupyter configuration.

On the Jupyter Hub side, you will need to wait for the new release, we should be able to get that out later this week. Beyond that, there shouldn’t be any need for any Cylc development to assist your deployment, it’s all plugins and configuration (note the load balanced spawner is not a solution to your auser accounts). Once the new release is out we’re happy to provide what assistance we’re able to in order to help get you get Jupyter Hub set up.

However, Jupyter Hub is an optional extra which adds a lot of additional complexity. You would be much better off getting started without Jupyter Hub by running in single-user mode (as we do), then upgrading to a Jupyter Hub deployment in due course as time allows (which we haven’t yet found time for).

The cylc gui command works out of the box, it deploys exactly the same web app as the Jupyter Hub setup and uses the same HTTP(S) communications, no X11 involved as it’s a web app not a native GUI (apologies if the name cylc gui is confusing). It shouldn’t have any issues with your auser accounts and does not involve spawning servers via root or sudo so is not affected by the user group issues you have encountered when using Jupyter Hub.

Your next step would be to try and run the GUI in single-user mode on the remote system, and connect to it via your web browser locally. When you start the GUI it prints a URL including a security token to your terminal e.g:

$ cylc gui
    Starting Cylc UI Server
    ...
    Or copy and paste one of these URLs:
        http://localhost:8888/cylc?token=12345asdfg
     or http://127.0.0.1:8888/cylc?token=12345asdfg

To access the GUI from your local system, change the localhost part to the domain of the remote system:

http://<address-of-the-remote-system>:8888/cylc?token=12345asdfg

Then open that URL in your web browser. The token is used to set a secure cookie, the bearer of the token has full control over the server, password authentication is also available.

Notes:

  1. Your remote system will probably have a whitelisted range of open HTTP(S) ports, you can use the --port argument to start the GUI on an open port or use c.ServerApp.port to configure it, you’ve likely already configured this for Jupyter Hub. Sadly Jupyter Server don’t seem to support port ranges so you may need to assign each user their own port e.g. by configuring it in ~/.cylc/uiserver/jupyter_config.py.
  2. HTTP is the default for Jupyter Server, configure c.ServerApp.keyfile and ServerApp.certfile to point at your certificate to upgrade to HTTPS communications. I’m guessing you’ve already got you certificates sorted out for your Jupyter Hub deployment, you can use the same certificate for Jupyter Server as Jupyter Hub.
  3. For security, it’s best to carry this connection over a VPN or other encrypted channel, although this is likely already mandated by the remote platform. Same goes for a Jupyter Hub deployment.

If you’d still like to call let me know, 18:30 is out of office hours, but I may be free for about 30 minutes until 19:00 tonight (note I would be joining from an insecure location).

Cheers,
Oliver

31

Agree on waiting until the team can digest the changes in the new release. Appreciate the insights and example. Once it gets released we will take a crack at configuring it but will probably more than likely drop a question or two your way. As always, thanks for the help.

32

Any progress with single-user mode?

33

We are able to spawn a UI server as the auser (sudo no login) using the token URL and access it from a browser on the ORNL side of our VPN tunnels. It is a bit of another thing to try it from the AF side using ports other than 443 since AF only allows that port to traverse for HTTPS. With it being a range that just amplifies the problem. Firewalls exist between the two sides that we don’t control. In lieu of that, we are looking at ways to proxy the traffic on a range of ports to 443. We do this with some of the other traffic so we should be able to get this to work.

With that in mind. What is the progress on UI server v1.4 release? In thinking a bit about this. If we can spawn a UI server in single user mode like you’ve stated before as one of our auser’s, is it only the auser that can connect to Cylc-flows started under that UI server? If that’s the case, that doesn’t help us. With the current Hub setup we authenticate regular users to be able to login with their System credentials. If we configured the permissions correctly with v1.4 release, would those users be able to manipulate Cylc-flows started under the single user mode auser spawned UI server? From what the guys are telling me that is what they need in order to make this work for us and it isn’t clear to us that is how it works.

In short, this is what we need.

1.) Be able to spawn a UI server as the auser (This one works!)
2.) As a properly authenticated user with permissions logged into the Hub, be able to manipulate the Cylc-flows under that auser spawned UI Server.

34

With it being a range that just amplifies the problem

You cannot configure a port range for Jupyter Server?

In lieu of that, we are looking at ways to proxy the traffic on a range of ports to 443

Jupyter Hub uses configurable-http-proxy but can also use traefik, these might be an easy start point.

If we can spawn a UI server in single user mode like you’ve stated before as one of our auser’s, is it only the auser that can connect to Cylc-flows started under that UI server?

If we configured the permissions correctly with v1.4 release, would those users be able to manipulate Cylc-flows started under the single user mode auser spawned UI server?

Multi-user mode enables Cylc’s authorisation layer. With this you will be able to authorise regular users to access and run whitelisted commands on the auser account server.

In single-user mode token (or password) authentication is used, the bearer of the token has full control over the server. If you provide the relevant users access to the token they will be able to access and run commands on that server. This is equivalent to sharing the workflow’s passphrase in Cylc 7 which I’m guessing was how you are currently working with these auser accounts?

As a properly authenticated user with permissions logged into the Hub, be able to manipulate the Cylc-flows under that auser spawned UI Server.

The authorisation layer is brand new functionality in Cylc 8 so you should be able to migrate your existing working setup/practices away from Cylc 7 without adopting this new feature, unless you’re looking to change account setup at the same time as transitioning to Cylc 8?

With that in mind. What is the progress on UI server v1.4 release?

We’ve encountered a nasty GUI bug which we are trying to resolve before release, we will try and push in a couple of extra changes to enhance multi-user mode if possible including the ability to switch server via the web app (without having to edit the URL).

35

Thanks for the quick replies Oliver. Appreciate it.
We are fine with creating port ranges on the Jupyter server. What I was alluding to there is we intend to have multiple auser spawned UI Servers, one for each project, that we have production workflows for. So there will be many. We will have to proxy the range of ports those run on back to the AF side of the tunnel. It’s work we have to do in order to link the 2 sides, really nothing with Cylc. Like I said, we do this already with some of our other traffic so we should be able to get that working. It looks like as long as we have the magic token , it doesn’t matter what user we are and we will be able to manipulate the workflows. Looks like what we need is going to be there Oliver. I am very appreciative of your help in making myself and my collogues understand how this works. Best of luck with getting your bug figured out. It’s usually something simple and it can drive you mad trying to find it.

Cheers!

36

Oliver, Hadn’t checked back in a week or so. Have you been able to clear the bug with the v1.4 server so you can do the release?

37

cylc-uiserver 1.4 will be released in the next day or two

38

We have a release!

This upgrades cylc-uiserver to work with Jupyter Hub v4.0+. The documentation has been updated to show how to set up multi-user deployments with Cylc authorisation.

https://cylc.github.io/cylc-doc/stable/html/user-guide/sharing-access-to-workflows.html

Quick overview:

  • Cylc UI Server now works with the latest Jupyter components.
  • Jupyter Hub can now be configured to allow users to spawn each others servers.
  • This means you can authorise a user to spawn and access a server for an “auser” account which they are not able to log into themselves.

There is one barrier you’ll hit with those “auser” accounts which I mentioned somewhere further up this thread. At present Jupyter Hub requires a users to be registered before it will spawn UI Servers for them. Users are automatically registered when they first authenticate with Jupyter Hub. Alternatively, users can be registered via the Jupyter Hub API which is handy for “auser” acounts:

2 Likes
39

Oliver thanks for all of your help regarding this. We have successfully configured our UI Server to allow the auser’s to share their workflows.

Question regarding a bug that was submitted by one of our folks (Monthly task in an otherwise hourly workflow has unexpected scheduling runahead limit behavior · Issue #5705 · cylc/cylc-flow · GitHub) and when the release 8.2.2 is expected to be released. Does it have a projected date?

40

Great news!

There’s no planned release date as such, we were hoping to get a couple of GUI bugfixs in for 8.2.2, but they’re taking a little while to iron out. We can look at accelerating this release.