I tried to use Docker and Singularity for running a complete workflow within a container (e.g. https://github.com/kinow/cylc-singularity). I haven’t executed that on our HPC yet, but it should work (may require some experimentation and some changes).
And I think I talked with Hilary some weeks ago about how other workflow engines support running parts of a workflow (or a sub-workflow, a cwl workflow, etc) within a container. This is much harder for us right now as it would require changes in how Cylc executes commands.
But an alternative would be to just write shell commands in
pre-script, that run the singularity container.
Within the container the task job script executes the commands under pre-script, script, etc.
Right now I think you would have to take control of the contents of
pre-script, and use a container to run your workflow steps.
Kinda like this I think?
cycling mode = integer
initial cycle point = 1
graph = "foo"
pre-script = "singularity exec --no-home --no-privs docker://python:2.7.18-stretch python --version"
script = "singularity exec --no-home --no-privs docker://python:2.7.18-stretch python -c 'for i in range(10):print i'"
And running it
$ cylc run singularity1
$ cylc cat-log singularity1 foo.1
Suite : singularity1
Task Job : 1/foo/01 (try 1)
2020-09-28T10:59:07+13:00 INFO - started
2020-09-28T10:59:13+13:00 INFO - succeeded
I used my development version of Cylc 8 to run these examples, but it should work with Cylc 7. This workflow was executed with Python 3.8, but the
script were executed with Singularity+Python 2.7.
So in theory you could use it to run any container image with docker/singularity/etc, just taking care to use any options your environment require (e.g.
singularity exec --no-home --nonet --workdir /u01/scratch/some-user ...), bind paths, use variables like
Hope this helps.
pre-script ended up in my job.err script, which was a surprise to me, so I’m debugging it, but shouldn’t be a problem here as I’m using Cylc 8