Thank you very much for the comprehensive answer @oliver.sanders ! I think this should give me some things to track down to try to figure out the issue. I think @swartn was seeing similar issues on our old system, so its probably related.
RE: the install target, yeah we are still using the hack that is discussed here because all our machines share a home but not where the scratch spaces are, so we make links under cylc-run