A lot of different conversations going on here!
@hilary.j.oliver
For consistency we should probably add
--yesto the unadornedcylc reinstallcommand too.
Yes, there is already an open PR, see this post further up the chat: Cylc reinstall auto yes? - #5 by oliver.sanders
multiple cycles of 7000 tasks per cycle that all want to run at once is definitely pushing the boat out.
It’s pushing the boat out for sure, but we are pushing scaling harder than that with some workflows. I recently encountered a Cylc 7 workflow with over 100’000 tasks! So 7’000 should be well within the capabilities of Cylc.
@srennie
I prefer to do “reinstall, validate run dir, reload” so that I don’t need to think about validating with the correct opt-conf-keys. I have a script for this, though a compound command would be nice. Or is there a reason not to do it this way?
The cylc vr command does this for you, it was added in Cylc 8.1.0.
https://cylc.github.io/cylc-doc/stable/html/reference/changes.html#combined-commands
A colleague is running ~2000 tasks in 3 active cycles and does see some issues with the load.
We regularly run Cylc 7 workflows with tens of thousands of tasks without hitting scaling limits of Cylc. I would expect ~2000 task cycles to run just fine.
If you’re hitting scaling limits with, make sure you’re not doing any many-many triggers (which are a known bottleneck because the number of dependencies increases to the square of the number of tasks).
@joshuatorrance
Thanks for the info, please report these problems so they can be investigated. This is not the expected behaviour of Cylc!
large workflows stopping (or perhaps crashing?) where their smaller (or idle) cousins continue on
Could you try running this large workflow in debug mode and report any traceback or symptoms:
$ cylc vip --debug
WebUI only displaying a fraction of the current tasks
This issue was fixed in a recent release of Cylc. Please ensure both the workflow and UI are running with the latest versions. If the issue can be repeated with the latest versions please let us know.
workflows can be very slow to start up, extreme examples can take an hour or two from running
cylc playto noticing that the WebUI showing it as running
The WebUI scans for workflows at a defined interval, make sure this hasn’t been configured at a couple of hours! If Cylc is genuinely taking over an hour to start a workflow…
Firstly, please ensure you are using the latest version of Cylc 8 as there have been efficiency improvements in recent releases e.g: Cylc 8.1.x Release Announcements - #3 by oliver.sanders
If the issue can be repeated with the latest release of Cylc, then this is a likely symptom of many-to-many task triggers which have N^2 scaling to the number of tasks so will slow things up a bit.
If that’s not the case, could you either provide us with the graph of this workflow for investigation OR run this command:
$ cylc validate <workflow> --profile
And send us the profile.prof file which is generated. This file contains the timings of each function and will help us to identify the issue. We should be able to speed things up a bit!