How can i find origin of validation warnings?

Cylc Cylc Support
1

hi there,

is there a way to find the location of errors like this in a workflow?..

> cylc val .
WARNING - Support for integers with leading zeros was dropped in Jinja2 v3. Rose will extend support until
    a future version.
    Please amend your Rose configuration files e.g:
     * 01 => 1
     * 00 => 0
     * 09 => 9

i have lots of rose configuration files and so lookng through all of them would be very time consuming. i know it doesn’t matter for now because i can still run with this warning but it would be great to be able to home in on them a bit better. i’ve tried cylc val -v but that doesn’t seem to help.

thanks,

jonny

2

Technically, I’d say that’s the cylc-rose plugin’s fault, not Cylc itself.

I would just regex grep for integers starting with 0 …

1 Like
3

Rose doesn’t preserve line numbers when it reads in configs so we can’t give you a position.

FYI, Jinja2 completely dropped support for zero-prefixed integers. We thought this was too disruptive so we hacked Jinja2 to maintain support for the short term. This hack is tricky, we won’t be able to keep it going with the next Jinja2 upgrade, so definitely act on this warning, next time it will be an error.

1 Like
4

what is they say?.. something like if you use regex now you have two problems :laughing:

i’ll try and work out how to do this!

thanks

1 Like
5

thanks @oliver.sanders

6

@hilary.j.oliver from the error message i guess this error is only relevant to rose*.conf files?

7

the reason i ask this is that this only show one result which is in the flow.cylc file and seems unrelated…

> grep -rE '=0[0-9]' *
flow.cylc:            -W umask=0022

image
image978×506 37.4 KB

8

Yes, a zero-prefixed integer in your flow.cylc Jinja2 code wlll result in a syntax error, with the bad line highlighted:

{% set foo = 01 %}

result:

$ cylc validate .

Jinja2Error: expected token 'end of statement block', got 'integer'
File /home/oliverh/cylc-src/bug/flow.cylc
   {% set foo = 01 %}    <-- TemplateSyntaxError

(The error message doesn’t seem terribly appropriate unfortunately, but that’s down to Jinja2, not Cylc)

… whereas, these are OK:

{% set foo = 1 %}
{% set bar = "01" %}
1 Like
9

roger that, thanks @hilary.j.oliver

10

ok got it!

i gave up using regex to be honest (too hard for me at the best of times let alone a friday afternoon) and deleted each directory inside the workflow definition until i was left with only a rose-suite.conf and a flow.cylc file left.

once i’d got there i realised that Jinja2 was complaining about this in my rose-suite.conf…

EXPT_BASIS=1978,09,01,00,00

which needed to be changed to…

EXPT_BASIS=1978,9,1,0,0

quite how on earth one would construct a regex to find that i don’t know haha

cheers,

jonny

11

I think that the following works

> sed -E 's@^\.0([0-9]+)@\1@g' ${FILE}

cat > testfile <<__HERE__
# Simple example
> FOO=0043
> # Datetime
> EXPT_BASIS=1978,09,01,00,00
> # Decimal
> BAR=0.004
> __HERE__

> sed -E 's@^\.0([0-9]+)@\1@g' testfile
# Simple example
FOO=0043
# Datetime
EXPT_BASIS=1978,09,01,00,00
# Decimal
BAR=0.004

YMMV - not extensively tested: Given this I’d tend to do

> sed -E 's@^\.0([0-9]+)@\1@g' testfile > testfile.new
> difftool testfile > testfile.new
# Manually inspect.
> mv testfile.new testfile
1 Like
12

Yes, that is unrelated, the 0022 there is just plain old text, not a Jinja2 value.

1 Like
13

Even simpler:

\b0\d
  • \b - word break incl commas etc.
  • 0 - the number 0
  • \d - any number
2 Likes
14

hi @oliver.sanders this doesn’t work for me i’m afraid…

> cat rose-suite.conf |grep EXPT_BASIS
EXPT_BASIS=1978,09,01,00,00

> grep -rE '\b0\d' *
[returns zero matches]

double quotes don’t work either.

fyi grep version is…

> grep --version
grep (GNU grep) 2.20

cheers,

15

The regex works, but grep don’t support it, here’s the same thing adapted for sed:

sed -rn '/\b0[0-9]/p' <file>

(note set doesn’t support \d)

16

ok thanks that works for sed…

> sed -rn '/\b0[0-9]/p' rose-suite.conf
EXPT_BASIS=1978,09,01,00,00

i’ve put in a PR in the cylc-rose repo to amend the warning text.

cheers,