Hi, looking at Global Configuration — Cylc 8.4.0 documentation - I was wondering if there was any way to extend the ranking algorthim to be a bit smarter. From how I read the documentation, the load limiting is based on current memory, load etc. But, if you launch a workflow at a temporary quiet period on that VM, the results could be a bit misleading.
I wondered if there was some way to give weights to workflows for example, or other things outside of the psutil attributes. For example,
VM1 = [a, b,c] # all large workflows
VM2 = [z, y, x] # all small workflows
Run cylc vip, and its quiet, so it launches on VM1, but, then, the workflows there start running and the VM is flooded, causing slowness due to resource contention. But, if there was a way to say a and x has a weight of 9, b of 6, c of 8, whereas z,y are weight 1 and 3. So, the new workflow should ideally end up on VM2 as it will on average have lower load.
More smarts would be expected/planned resource load profiles and Cylc could compare against expected peak loads and launch workflows in such a way to smooth out expected/planned load.
We can use any of the functionalities of the psutil module, do basic maths with any numbers returned and hardcode thresholds, etc. The expressions you provide are parsed by Python in a restricted environment which only allows certain operations (e.g. you cannot import modules). This isn’t presently pluggable.
This ranks hosts by available memory, but excludes hosts with high server load. We find ranking by available memory works pretty well, our servers are about as evenly loaded as could be hoped for. Here’s a screenshot of today’s memory loading (0-100%), the servers are all within 5% of each other: