# \[rose\]: metadata file organisation

**URL:** <https://cylc.discourse.group/t/rose-metadata-file-organisation/310>\
**Category:** Rose Support\
**Created:** [February 19, 2021, 4:39pm UTC](https://cylc.discourse.group/t/rose-metadata-file-organisation/310 "2021-02-19T16:39:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![edmundh](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/edmundh/32/70_2.png) [@edmundh](https://cylc.discourse.group/u/edmundh)\
**Post date:** [February 19, 2021, 4:39pm UTC](https://cylc.discourse.group/t/rose-metadata-file-organisation/310/1 "2021-02-19T16:39:24Z")

</div>

Last metadata question (for today anyway!)

Can someone clarify what [the recommendation on metadata organisation in the metadata location appendix](https://metomi.github.io/rose/2019.01.2/html/api/configuration/metadata.html#admonition-12) means please?  
Apologies, I’m probably being rather dim, but I’m not understanding this!

First off, does “system” mean “suite” here? I’m currently assuming it does - maybe I’m wrong, and this whole recommendation is irrelevant for a suite?  
Wondering as I can’t see recommendations on metadata organisation the [suite design guide](https://cylc.github.io/cylc-doc/current/html/suite-design-guide/suite-design-guide-master.html#suite-design-guide) which would fit my interpretation below.

Assuming recommendation is for suites…  
I’ve currently got my suite organised something like this, with a top-level `meta` dir for the suite metadata, and per-app `meta` subdirs for the metadata for individual apps:

```bash
/path/to/mi-su173
me@pc:> tree
.
|-- app
| |-- my-model_config-compile-test_cold
| | |-- bin
| | | `-- config-compile-test_mod.sh
| | |-- meta
| | | `-- rose-meta.conf
| | `-- rose-app.conf
| |-- my-model_install_cold
| | |-- bin
| | | `-- install_mod.sh
| | |-- meta
| | | `-- rose-meta.conf
| | `-- rose-app.conf
| <snip more apps>
<snip other dirs & files>
|-- meta
| `-- rose-meta.conf
|-- rose-suite.conf
|-- rose-suite.info
`-- suite.rc

```

I’m not entirely clear what advice above means for my suite, both for the suite-level metadata, and for the app-level metadata.

I _think_ it means I should rejig my suite, to move the individual apps’ `meta` subdirs up, to live under a `rose-meta` dir (and I assume my suite metadata too).  
And I should be effectively duplicating my apps’ directory structure - would have same subdirs under `/rose-meta` as well as under `/app`. So I’d end up with something like:

```bash
|-- app
| |-- my-model_config-compile-test_cold
| | |-- bin
| | | `-- config-compile-test_mod.sh
| | `-- rose-app.conf
| |-- my-model_install_cold
| | |-- bin
| | | `-- install_mod.sh
| | `-- rose-app.conf
| <snip more apps>
<snip other stuff>
|-- rose-meta
| |-- my-model_config-compile-test_cold
| | `-- rose-meta.conf
| |-- my-model_install_cold
| | `-- rose-meta.conf
| <snip dirs/rose-meta.confs for more apps>
| `-- rose-meta.conf # suite-level metadata
|-- rose-suite.conf
|-- rose-suite.info
`-- suite.rc

```

And I’d let each app/the suite know where to find its metadata by using the [meta setting](https://metomi.github.io/rose/2019.01.2/html/api/configuration/application.html#rose:conf:rose-app.conf%7Cmeta).  
So `rose-suite.conf` would specify: `meta=rose-meta/rose-meta.conf`  
While `app/my-model_copy_cold/rose-app.conf` would specify: `meta=rose-meta/my-model_copy_cold/rose-meta.conf`  
and so on.

Is this right? Wanting to check my understanding on this before I charge off and do this, as this would require updating quite a few extra places if I ever changed the name of a given app! And I’m increasing “separation” in dir structure between my metadata for a given app, and the app itself.

I’m also wondering whether I’ve got wrong end of stick with my “duplicate dir structure” interpretation as I’ve seen other places (e.g. [UM](https://code.metoffice.gov.uk/trac/um/browser/meta/trunk/um-atmos/vn11.8/rose-meta.conf)) where there’s just a single monolithic large rose-meta.conf file.

If recommendation would apply to me, is there a good reason for the recommendation? I.e. what is the benefit of me overhauling current structure to a “duplicated dir structure”? (or a monolithic rose-meta.conf if I’ve got wrong end of stick)?

---

<div class="post-metadata">

**Author:** ![oliver.sanders](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/oliver.sanders/32/110_2.png) [@oliver.sanders](https://cylc.discourse.group/u/oliver.sanders)\
**Post date:** [February 22, 2021, 11:10am UTC](https://cylc.discourse.group/t/rose-metadata-file-organisation/310/2 "2021-02-22T11:10:39Z")

</div>

Hello,

Took me a couple of reads of that passage. This appendix is talking about “centralised” metadata and might not be relevant to your usage of Rose.

> **tldr;** It’s not recommending that you collect all of the metadata written within a suite into a top-level `rose-meta` directory but suggesting that when metadata is written in a system code (e.g. the UM) _rather_ than in the suite it should live in a top-level `rose-meta` dir in the code repository.

Rose apps (and suites) consist of two parts:

1. The metadata which is the bit that really defines the application itself.
2. The configuration which is the bit that is specific to the particular usage of the application.

For a lot of purposes, particularly single-use applications, it can be convenient to develop these two parts in the same place, writing the metadata inside the configurations e.g:

```auto
suite/
  rose-suite.conf
  meta/
    rose-meta.conf
  app/
    foo/
      rose-app.conf
      meta/
        rose-meta.conf

```

This works great, however, poses a barrier to sharing and collaboration. If you update the metadata in an app locally, you need to update all of the other copies of that app that may have been made.

For apps that are widely used and re-used by multiple people it makes sense to collect the metadata all in one place and develop it separately under it’s own version control. That factors out the first part of the application leaving just the configuration that relates to the specific usage of the application in your suite.

That would look something like this:

```none
~/roses/my-suite/
  rose-suite.conf
  app/
    foo/
      rose-suite.conf # no metadata here, it's picked up through the site meta-path

~/foo-code/
  rose-meta/
    foo/
      v1.0.0/
        rose-meta.conf

```

This pattern is used by some of the larger, more regularly shared apps such as the UM.

---

<div class="post-metadata">

**Author:** ![edmundh](https://yyz2.discourse-cdn.com/free1/user_avatar/cylc.discourse.group/edmundh/32/70_2.png) [@edmundh](https://cylc.discourse.group/u/edmundh)\
**Post date:** [February 22, 2021, 12:26pm UTC](https://cylc.discourse.group/t/rose-metadata-file-organisation/310/3 "2021-02-22T12:26:05Z")

</div>

Lovely, thanks Oliver, enlightenment achieved!  
Really helpful - I/others in my team so often work in ~solo code contexts that I don’t encounter/think about pain points relevant to more common shared contexts.  
In particular, I don’t believe we’ve yet started sharing apps within our team.  
This explanation makes perfect sense, and also means I better understand the [upgrading](https://metomi.github.io/rose/2019.01.2/html/tutorial/rose/furthertopics/upgrading.html) use-case, and likely implementation.  
Very useful - means I’ll be better-placed to think about options on how to arrange things if/when we start reusing apps more between suites.