Cylc message for message triggers - how to output from python?

Cylc Cylc Support
1

Hi, what is the correct way to output a cylc message from a python script? (Sorry for the super simple question, I swear I googled it first!)

It works fine for me from a bash script (the messages that are output trigger all the right tasks), but when I put something like:

yesstr="cylc message -- '${CYLC_SUITE_NAME}' '${CYLC_TASK_JOB}' 'File found.'"
os.system(yesstr)

in a python script - the message appears in job.out exactly the same as with the bash script - but it’s not in that task’s “prereqs and outputs” - then the task it’s meant to trigger does not have its prerequisite satisfied, and it never starts, the cylc graph just stops there.

Now, I’m keen to use python rather than bash if possible, because I also need to read in a .json file as part of the task - which I’m aware is possible in bash, but I thought I’d just ask here about cylc messages first before looking into that.

Thanks in advance for your help,
Susan

(For those who can access it, the suite I’m developing is here: http://fcm1/projects/roses_mi/browser/b/b/4/1/9/trunk )

2

Hello,

Cylc7 doesn’t really expose a Python API (also you may want your Python script to run in a different interpreter to Cylc) so you will probably want to call the cylc message command from within your Python script.

The standard way to do this is via the Python subprocess module. Here’s a simple subprocess example showing how to run the bash command echo 'hello world':

from subprocess import Popen

Popen(['echo', 'hello world'])

And here’s some boilerplate to show how you could use subprocess to call cylc message (untested):

from subprocess import Popen, PIPE, DEVNULL
from shlex import quote

proc = Popen(
    ['cylc', 'message', '--', quote(suite_name),  quote(task_name), quote(message)],
    stdout=PIPE, stderr=PIPE, stdin=DEVNULL
)
if proc.wait:  # wait for the process to completed
    # the cylc message command failed, handle this as appropriate
    pass
else:
    # is passed
    pass
1 Like
3

Hi @SusanSun,

You should probably take @oliver.sanders advice and use subprocess.Popen instead of the old os.system - see Python docs: https://docs.python.org/2/library/os.html#os.system

However for simple use cases like this, os.system() is OK. Your example only fails because your single quotes protect the CYLC variables from being interpreted by the shell. If you run the suite with --debug you will see this in the job.err log:

ClientInfoError: Contact info not found for suite "${CYLC_SUITE_NAME}", suite not running?

i.e. the job is trying to connect to a suite called, literally, ${CYLC_SUITE_NAME}.

Remove the single quotes from the environment variables and it will work:

yesstr="cylc message -- ${CYLC_SUITE_NAME} ${CYLC_TASK_JOB} 'File found.'"
os.system(yesstr)

Hilary

1 Like
4

@SusanSun,

p.s. regarding use of the internal Cylc 7 Python API, (instead of calling the cylc message CLI in a subshell) I would recommend you follow @oliver.sanders advice here too:

We are planning to provide a stable Python API to Cylc internals at Cylc 9. However just for the record, if you have a Python task script there is nothing to stop you from calling directly into the Cylc 7 Python library right now. To see how to do that, look inside the cylc message script to see that it calls record_messages from cylc.task_message. To see how to use that function, with the Cylc library in your PYTHONPATH do:

from cylc.task_message import record_messages
help(record_messages)

Then, in your Python task script:

# My Cylc 7 task script (which sees the task job environment).
import os
from cylc.task_message import record_messages
# ...
suite = os.environ['CYLC_SUITE_NAME']
task_job = os.environ['CYLC_TASK_JOB']
record_messages(suite, task_job, [['INFO', 'File found.']])
# ...

This is obviously cleaner than calling the cylc message CLI in a subshell, but because we don’t officially provide a stable Python API yet the downsides are:

  • we don’t document how to do it
  • the API is subject to change without notice
  • (you also need to run your script in the same Python as the task job, with access to the same Cylc library. E.g. for Cylc 7, you can’t use Python 3)

Hilary

1 Like
5

Oops, how silly of me! Thank you for pointing out those single quotes @hilary.j.oliver , and for your advice on using the Python subprocess module @oliver.sanders ! And thank you for all the background info too, this is really helpful.

1 Like
6

Just a final note on use of the Cylc library in Python scripts (in task jobs or otherwise) - @oliver.sanders quite rightly berated me for advertising how to do that :grimacing: It is indeed possible, but if you do it before we officially support a stable API, your task jobs might just mysteriously break on moving to a new Cylc version, and that might be difficult to debug, so best not to do it … for now!

1 Like