Hi,
We’re using cylc to manage the transfer of data from our old archive to JASMIN. Each cycle is responsible for transferring a chunk of data and running several other tasks to verify the data transfer. The amount of data each cycle is dealing with varies so later cycles may run quicker that previous cycles. Tasks have no dependency on their previous task instance, however they are only spawned in later cycles once the task instance from the previous cycle has been submitted. This is causing a bottleneck. I understand there is a way to force cylc to spawn all tasks for a cycle upon cycle initialisation, but this is not documented. Please could you send me instructions on how to do this?
Thanks.
Regards,
Ros.
Hi Ros,
I understand there is a way to force cylc to spawn all tasks for a cycle upon cycle initialisation
You can do this with the settings ‘max active cycle points’ & ‘spawn to max active cycle points’. Set the former to a higher than default value, as appropriate, & set the latter to True.
Take care, though, because this could (depending on the context) mean that a very large number of tasks will run simultaneously. If you want to manage that further, you can use queues to limit the number active at once.
but this is not documented
These are in fact documented, as linked above, but they are hidden away in the ‘suite.rc’ configuration reference. Sorry if they couldn’t be found readily. We’ll think about how we can make these aspects more reachable & prominent.
Hope that helps!
Hi Sadie,
Thank you - that’s perfect.
Cheers,
Ros.