# Running on remote

**URL:** <https://cylc.discourse.group/t/running-on-remote/150>\
**Category:** Cylc Support\
**Created:** [September 26, 2019, 11:26pm UTC](https://cylc.discourse.group/t/running-on-remote/150 "2019-09-26T23:26:11Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\
**Post date:** [September 26, 2019, 11:26pm UTC](https://cylc.discourse.group/t/running-on-remote/150/1 "2019-09-26T23:26:11Z")

</div>

I have a system which primarily uses pbs, but has a subset of nodes that use slurm. These nodes are not binary compatible with the cylc host nodes. The job submits and runs correctly but cylc tries to run on the compute node and crashes. Is there a way to set up polling for just that one job? Using [[hosts]] in the global config file doesn’t seem to do what I want since that seems to want to ssh to the remote to submit the job.

```auto
[meta]
    title = "The cylc Hello World! suite"
[scheduling]
    [[dependencies]]
        graph = "hello => hello_casper"

[runtime]
    [[hello]]
        script = "sleep 10; echo Hello World!"
    [[[job]]]
      batch system = pbs
      batch submit command template = qsub -q regular -l walltime=01:00:00 -A NCGD0042 '%(job)s'
    [[[directives]]]
       -r = n
       -j = oe
       -V =
       -S = /bin/bash
       -l = select=1:ncpus=36:ompthreads=36

    [[hello_casper]]
        script = "sleep 10; echo Hello World from casper!"
    [[[job]]]
      batch system = slurm
    [[[directives]]]
       --ntasks=1
       --cpus-per-task=8
       --patition=dav

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [September 27, 2019, 1:33am UTC](https://cylc.discourse.group/t/running-on-remote/150/2 "2019-09-27T01:33:52Z")

</div>

Hi Jim,

> The job submits and runs correctly but cylc tries to run on the compute node and crashes.

I presume you mean the `cylc message` command crashes on the job host? (when it tries to send back job status messages).

Cylc is written in Python so it should not matter that your Slurm host is not binary compatible with the Cylc host.

So: is `cylc message` just failing because on the job host because it can’t send messages back (in which case, check network configuration), OR is `cylc` generally non-functional there because some Python library is missing, or something like that?. Your `job.err` file should reveal which.

> Using [[hosts]] in the global config file doesn’t seem to do what I want that seems to want to ssh to the remote to submit the job.

What are you setting under the `[[hosts]]` heading exactly? If you configure a task with a remote host and batch system Slurm, Cylc will ssh to the host and submit the job to Slurm there, but that is via the task (or family) `remote` section, not global config `hosts` section.

If you have a local Slurm client (i.e. on the Cylc host) then don’t specify a remote in the suite configuration - then it is a local job as far as Cylc is concerned, regardless of where Slurm executes the job. ( **however** note in this case you must have a shared filesystem between the local and job hosts).

> Is there a way to set up polling for just that one job?

If you can’t get network comms back from a job host you can configure job polling (per host) as a “task communication method” - see [https://cylc.github.io/doc/built-sphinx-single/index.html#job-polling](https://cylc.github.io/doc/built-sphinx-single/index.html#job-polling)

Hilary

---

<div class="post-metadata">

**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\
**Post date:** [September 27, 2019, 1:47am UTC](https://cylc.discourse.group/t/running-on-remote/150/3 "2019-09-27T01:47:45Z")

</div>

> [@hilary.j.oliver](#):
>
> So: is `cylc message` just failing because on the job host because it can’t send messages back (in which case, check network configuration), OR is `cylc` generally non-functional there because some Python library is missing, or something like that?

```auto
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/glade/u/apps/ch/opt/python/2.7.16/gnu/8.3.0/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] =__get_hash(__func_name)
  File "/glade/u/apps/ch/opt/python/2.7.16/gnu/8.3.0/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5

```

also this

```auto
Traceback (most recent call last):
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/bin/cylc-message", line 140, in <module>
    main()
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/bin/cylc-message", line 136, in main
    return record_messages(suite, task_job, messages)
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/lib/cylc/task_message.py", line 82, in record_messages
    'messages': messages})
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/lib/cylc/network/httpclient.py", line 275, in put_messages
    results = self._call_server(func_name, payload=payload)
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/lib/cylc/network/httpclient.py", line 338, in _call_server
    return self.call_server_impl(url, method, payload)
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/lib/cylc/network/httpclient.py", line 369, in call_server_impl
    return impl(url, method, payload)
  File "/glade/u/apps/ch/opt/cylc/7.8.3/gnu/8.3.0/cylc-7.8.3/lib/cylc/network/httpclient.py", line 477, in _call_server_impl_urllib2
    import ssl
  File "/glade/u/apps/ch/opt/python/2.7.16/gnu/8.3.0/lib/python2.7/ssl.py", line 98, in <module>
    import _ssl # if we can't import it, let the error propagate
ImportError: libssl.so.1.0.0: cannot open shared object file: No such file or directory

```

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [September 27, 2019, 3:22am UTC](https://cylc.discourse.group/t/running-on-remote/150/4 "2019-09-27T03:22:50Z")

</div>

Looks like you don’t have SSL installed.

See [https://cylc.github.io/doc/built-sphinx/installation.html#third-party-software-packages](https://cylc.github.io/doc/built-sphinx/installation.html#third-party-software-packages)

(And hashlib is related, see e.g. [https://stackoverflow.com/questions/20399331/error-importing-hashlib-with-python-2-7-but-not-with-2-6](https://stackoverflow.com/questions/20399331/error-importing-hashlib-with-python-2-7-but-not-with-2-6))

---

<div class="post-metadata">

**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\
**Post date:** [September 27, 2019, 4:03pm UTC](https://cylc.discourse.group/t/running-on-remote/150/5 "2019-09-27T16:03:09Z")

</div>

Okay I now have a proper cylc build on those nodes. How do I let cylc know the correct path to  
that build? I am submitting from localhost, so using some [[remote]] option doesn’t seem to be the correct solution.

---

<div class="post-metadata">

**Author:** ![dpmatthews](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/dpmatthews/32/26_2.png) [@dpmatthews](https://cylc.discourse.group/u/dpmatthews)\
**Post date:** [September 28, 2019, 9:10am UTC](https://cylc.discourse.group/t/running-on-remote/150/6 "2019-09-28T09:10:15Z")

</div>

You need to install the cylc wrapper script in $HOME/bin, see  
[https://cylc.github.io/doc/built-sphinx/installation.html#local-user-installation](https://cylc.github.io/doc/built-sphinx/installation.html#local-user-installation)

---

<div class="post-metadata">

**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\
**Post date:** [September 28, 2019, 4:27pm UTC](https://cylc.discourse.group/t/running-on-remote/150/7 "2019-09-28T16:27:36Z")

</div>

Thanks, that did it. But it doesn’t seem that cylc is sourcing my .bashrc,  
I had to include that explicitly in my job-init-env.sh. The documentation your link points to seems to suggest that that should happen automatically. Any idea why it doesn’t?

---

<div class="post-metadata">

**Author:** ![Jim](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/jim/32/60_2.png) [@Jim](https://cylc.discourse.group/u/Jim)\
**Post date:** [September 28, 2019, 4:34pm UTC](https://cylc.discourse.group/t/running-on-remote/150/8 "2019-09-28T16:34:06Z")

</div>

duh - nevermind, I moved .bashrc to .bash\_profile and now it’s working as expected.

---

<div class="post-metadata">

**Author:** ![hilary.j.oliver](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/hilary.j.oliver/32/4_2.png) [@hilary.j.oliver](https://cylc.discourse.group/u/hilary.j.oliver)\
**Post date:** [September 28, 2019, 11:44pm UTC](https://cylc.discourse.group/t/running-on-remote/150/9 "2019-09-28T23:44:28Z")

</div>

Hi Jim,

That’s right - task job scripts invoke bash login shells, so you need to use .bash\_profile. There are several other ways you can configure the environment for Cylc on job hosts too, but bash login scripts is the best way these days.

Hilary