Can the auto-documenters from Cylc Sphinx Extensions be used to document workflow

Cylc Cylc Support
1

Hi everyone,

The cylc-lang Sphinx extension defines autodocumenters, that are used in Cylc docs for these 3 pages: Data Types, Global Configuration and Workflow Configuration.

Can they be used to auto-document a Cylc workflow (using tasks names, [meta]title/description, [environment]variables etc…), or is it simply not their purpose? Should I simply handwrite what I want to document using the Cylc Sphinx domain defined in cylc-lang ?

Thanks.

-Elliot

2

The auto documenters in cylc-lang are more about documenting the Cylc configuration itself, they won’t work for documenting workflows, however, a similar approach would work for this purpose.

You would need to do something like this to get at the task configs:

from cylc.flow.config import WorkflowConfig
from optparse import Values
workflow_config = WorkflowConfig('workflow_name', '/path/to/flow.cylc', Values())
for task, config in workflow_config.cfg.get('runtime').items():
    document_task(task, config)

Worth mentioning we plan to work on a “metadata view” before long. This would provide access to task’s metadata from within the GUI. Not quite the same as a static reference page, but may fulfil similar requirements.

3

Example of what we’re thinking regards the metadata view:

  • For task definitions:
    • Display task metadata (we’re considering markdown and restructured text).
    • Display task configuration.
  • For specific task instances, also:
    • List relevant broadcasts.
    • Show prerequisites and outputs.

2969×1945 511 KB

Here’s the GH issue: task metadata view · Issue #1071 · cylc/cylc-ui · GitHub

1 Like
4

Do you think there is any simple way to cross the gap between the OrderedDictWithDefault class exposed in your snippet, and the ConfigNode(ContextNode) class used in most of autodocumenters.py ?

I guess json.dumps()ing the runtime dictionary and then feeding it to the auto-cylc-conf:: directive could work? Maybe?

Looking at the code, it seems the directive can indeed take raw json text as content, but it isn’t documented here contrary to auto-cylc-type::.

EDIT: It doesn’t take json. The documentation was correct. :frowning:

5

Hmmm. I don’t think there’s much scope for code reuse between a schema-documenter and a workflow-documenter because they’re using very different attributes and I expect there’ll be lots of special logic to shoehorn into the workflow documenter (metadata, outputs, env, etc).

I think the way to go will be to copy the approach with a new documenter along the lines of:

# untested

from cylc.flow.config import WorkflowConfig
from optparse import Values
from docutils.parsers.rst import Directive
from docutils.statemachine import StringList
from cylc.sphinx_ext.cylc_lang.autodocumenters import directive

class CylcAutoDirective(Directive):
    has_content = True
    option_spec = {}
    required_arguments = 2
    optional_arguments = 0

    def run(self):
        workflow_name = self.arguments[0]
        workflow_def_file_path = self.arguments[1]
        workflow_config = WorkflowConfig(workflow_name, workflow_def_file_path, Values())

        workflow = doc_workflow(workflow_name, workflow_config)

        tasks = [
            document_task(task, config)
            for task, config in workflow_config.cfg.get('runtime').items()
        ]

        node = addnodes.desc_content()
        self.state.nested_parse(
            StringList([workflow, *tasks]),
            self.content_offset,
            node,
        )

        return [node]


def doc_workflow(name, config):
    return directory(
        'cylc:conf',
        [name],
        {},
        {},
        f"{config.get('meta', {}).get('title', '')}\n{config.get('meta', {}).get('description', '')}",
    )


def document_task(name, config):
    return [
        # TODO: implement me!
        # maybe use a "standard domain" directive to avoid the need to implement a special Cylc one
        # (we only use Cylc directives to make them indexable/linkable, likely not needed for workflow docs)
    ]

(see The Standard Domain — Sphinx documentation)

Note, I’ve implemented this Sphinx autodocumenter the lazy way, by writing out restructured text, then parsing it back as I find this much easier to develop / debug, however, most autodocumenters probably import the directives and initiate them via the Python API.

1 Like
6

Thanks you so much! I’m gonna try with the approach you proposed.

On the other hand, I managed to reuse the rose:file:: autoconfig:: directive (here) for documenting my workflow user config.

It was quite easy to add (see PR commit history). Maybe Rose Sphinx Domain could be added to the doctree of Rose’s documentation? I had to find the page with the searchbar. I know it’s supposed to be for you METOMI devs, but I might not be the only one wanting to use it :smiley:

1 Like
7

Thanks you so much! I’m gonna try with the approach you proposed.

Great, good luck! Feel free to open a PR on cylc-sphinx-extensions, I’m sure others would be interested in this. Feel free to post back here, happy to advise as best I’m able.

On the other hand, I managed to reuse the rose:file:: autoconfig:: directive

Nice. The Rose autoconfig directive is a bit different to the Cylc one because it generates its documentation from an example configuration file whereas Cylc generates its documentation from a schema (Rose doesn’t have a schema).

Maybe Rose Sphinx Domain could be added to the doctree of Rose’s documentation?

Wow, well done for finding that!

We hadn’t thought of opening this up to general usage yet, yes, those are our internal pages on documentation writing.

That directive isn’t packaged with Rose ATM, but it could be. Combining the Rose and Cylc directives would result in something very powerful.

8

(found this issue extension to auto-document Cylc workflows from [meta] sections · Issue #75 · cylc/cylc-sphinx-extensions · GitHub)

1 Like