Hi Cylc’ians,
I’m wondering whether anyone had considered using Cylc to build code with dependencies? It seems that a workflow engine could be used for compilation as well as running the code. A potential advantage would be to exploit parallelism when compiling dependencies that don’t depend on each other. I’d be interested to know anyone’s experience in this area.
Thanks. --Alex
Hi Alex,
Interesting idea. On a cluster with a shared FS you could certainly get maximum build concurrency this way. However, you’d need a preprocessing step to analyse the software dependencies and generate the Cylc task graph to do everything in the right order. Do traditional build systems typically allow that information to be written out for external use?
Hilary
Thanks Hilary.
Some knowledge of the dependencies would need to be encoded, e.g.
build_mpi => build_hdf5 => build_netcdf => build_wrf & build_mitgcm => build_coupled_model
I wasn’t thinking, at this point, about generating the graph automatically (although I’d love the idea)
Oh right, yes that would be easy enough and useful.
I guess I was thinking of building a single large model with internal build dependencies among thousands of C++ or Fortran files, say. If make (or similar) could analyze the codebase and spit out the dependencies, you could use a workflow engine to spread the build across your cluster.
Yes, that could be another application. Fortran codes often require modules to be compiled in a certain order. In the case of LFRic, I think they parse the source files to figure out the order of compilation. C++ does not have that issue since headers are ASCII files that don’t require compilation.