Hi. I understand that CYLC_SUITE_DEF_PATH from Cylc 7 is obsolete in Cylc 8. However, I don’t see any kind of analog in Cylc 8 that points to the location of the installed workflow (which under normal conditions might be something like ${CYLC_WORKFLOW_RUN_DIR}/…/_cylc-install). Is there such a variable and I’m just missing it in the docs?
Thanks.
That variable existed because Cylc 7 (and earlier) did not not natively support installation of workflow source files into the run directory.
Cylc 8 supports (and requires) workflow installation, so the “live” workflow config is the installed copy, which leaves you free to work on the source without affecting installed copies. There should not be any reason to access the original workflow source location at runtime, and in fact doing so is dangerous (do you want running workflows to be affected by source changes, or vice versa?).
So the equivalent variable in Cylc 8 is $CYLC_WORKFLOW_RUN_DIR, which is the location of the installed workflow source files (a.k.a. $CYLC_RUN_DIR/$CYLC_WORKFLOW_ID).
Actually $CYLC_SUITE_DEF_PATH is technically just deprecated - you can still use it, but now it points to the installed workflow (i.e., the run directory).
OK, my issue here really appears to be that I wasn’t far enough along in the migration effort to realize that the entire source directory gets copied into the workflow run directory. Without that, I needed access to the workflow source because binaries I use require some static data files that, in my old Cylc 7 suite, were copied into the run directory during initialization so that the run directory was self-contained. I wasn’t aware that that copy wouldn’t be needed under Cylc 8, because the entire source directory gets copied in.
Thanks!
I’m finding that for suites that do this (rely on seeded data under cylc-run/cylc-work/share/etc) that are transitioning from Cylc 7 is if you frequently have to restart your models (i.e., in an alpha/beta environment) you will have to recopy all those static files and binaries every time your perform a “cylc vip” or otherwise install the suite.
I believe using concurrent flows is the correct solution for this if you can catch the failure quickly, but more often than not we find that it’s faster to restart the run rather than walk it up which will mean repopulating the run# directory. Especially if relevant data was already scrubbed, like if the failure happens on a Friday night and now you’re 3 days behind by Monday.
Is the solution to define an R1 point with a task that seeds these needed files if you’re doing a full restart? Or am I over-complicating this?
Hi @russbnavy,
Are you saying you manually copy files (from some external location, not the source directory) to the newly created (i.e., installed) run directory before start-up?
If so, yes, you should put an initial task in the workflow (R1) to do that automatically every time you start a new instance of the workflow from scratch.
Starting over in an existing run directory is unsafe outside of dev and testing, but you’re the boss and if you want to do it you just need to remove the .service directory left by the previous run:
$ cylc clean <workflow-id> --rm=".service"
(I thought we planned to provide a cylc play option to force a re-do in an existing run directory, but I’m not sure … it might be controversial).
Yes, exactly.
Ok great, I just wanted to make sure I’m on the right path.
I think the R1 requirement answered this concern, I agree I’d rather not muck with Cylc’s expected internal state of runs. Alternatively I suppose folks could also just use their own environment variables/working space and aren’t forced to use the ones predefined by Cylc 8.
I will say I initially didn’t like the seperate run directories but now I quite like them and they’re very easy to clean up/maintain/see what’s taking up so much space during runs and for operations, roll back to a previous run if a new install immediately breaks down.