Hi Jin,
I tried to use Docker and Singularity for running a complete workflow within a container (e.g. GitHub - kinow/cylc-singularity: Singularity containers for Cylc). I havenāt executed that on our HPC yet, but it should work (may require some experimentation and some changes).
And I think I talked with Hilary some weeks ago about how other workflow engines support running parts of a workflow (or a sub-workflow, a cwl workflow, etc) within a container. This is much harder for us right now as it would require changes in how Cylc executes commands.
But an alternative would be to just write shell commands in script
, pre-script
, that run the singularity container.
Within the container the task job script executes the commands under pre-script, script, etc.
Right now I think you would have to take control of the contents of script
, pre-script
, and use a container to run your workflow steps.
Kinda like this I think?
# ~/cylc-run/singularity1/suite.rc
[scheduling]
cycling mode = integer
initial cycle point = 1
[[dependencies]]
graph = "foo"
[runtime]
[[foo]]
pre-script = "singularity exec --no-home --no-privs docker://python:2.7.18-stretch python --version"
script = "singularity exec --no-home --no-privs docker://python:2.7.18-stretch python -c 'for i in range(10):print i'"
And running it
$ cylc run singularity1
...
$ cylc cat-log singularity1 foo.1
Suite : singularity1
Task Job : 1/foo/01 (try 1)
User@Host: kinow@ranma
2020-09-28T10:59:07+13:00 INFO - started
0
1
2
3
4
5
6
7
8
9
2020-09-28T10:59:13+13:00 INFO - succeeded
I used my development version of Cylc 8 to run these examples, but it should work with Cylc 7. This workflow was executed with Python 3.8, but the pre-script
and script
were executed with Singularity+Python 2.7.
So in theory you could use it to run any container image with docker/singularity/etc, just taking care to use any options your environment require (e.g. singularity exec --no-home --nonet --workdir /u01/scratch/some-user ...
), bind paths, use variables like $CYLC_SUITE_WORK_DIR
, $CYLC_TASK_WORK_DIR
, etc.
Hope this helps.
Bruno
p.s: my pre-script
ended up in my job.err script, which was a surprise to me, so Iām debugging it, but shouldnāt be a problem here as Iām using Cylc 8