Hey guys,
I’m quite new on the use of cylc, i need your light. I run my tasks twice a day, for each of my runs and tasks there is log files that are associated. For instance my suite executes 150 tasks for one run which means 300 log files per day. I don’t want to have so many files, naturally i have created a script that erased the log files of the pervious run.
I want to know if there is an option or command line with cylc that avoids to write the log files, except if there is an error with one of the taks.
Thank you for reading me and i hope that you will help me to fullfil my curiosity
Kindly regards
Hi @ClementL,
Sorry, there’s no way to turn off suite and job logging in Cylc. Would you believe, no one has ever asked us this before!
It’s true that a large and/or ongoing cycling workflow can generate a lot of log files, but that’s generally regarded as essential information - if anything goes wrong you need to be able to diagnose the problem by looking at the logs.
To deal with the proliferation of job logs we designed the log directory hierarchy to make housekeeping easy, e.g. cycle point is at the top (for each suite) so that you can delete the output from old cycle points once they’re no longer needed.
Unfortunately Cylc does not have built-in job-log housekeeping functionality (it is planned) but it is not too difficult to do yourself. We typically put special housekeeping tasks in the workflow to delete old job logs at the cycle point level.
Note also - in case your problem is a restrictive home directory, you can configure Cylc to automatically symlink the standard run directory path to another (e.g. large data) location.
Regards,
Hilary
2 Likes
Correction, that’s coming natively to Cylc soon, but for the moment it’s done by configuring rose suite-run, for those of us who use Rose with Cylc. (I’m getting ahead of myself!) Credit @dpmatthews for noticing my mistake 
2 Likes
The common approach to this is to clean up the log files from previous cycles.
Rose provides an app called “rose_prune” which performs cycle-aware housekeeping, so for example you could say cleanup all job logs from cycles more than P1D (one day) behind the oldest cycle currently active.
1 Like