What tool should be used to checkout NorESM externals?

[gold2718]

The NorESM 2.5 alpha08 tags introduced git fleximod as a replacement for manage_externals. Both tools were developed at NCAR and both tools aim to manage the correct checkout of NorESM externals and sub-externals based on a documented set of external repositories and tags. Experience with git fleximod has led to this discussion on the properties of a desired solution for the NorESM community and developers. This introduction has two sections:

• Should the NorESM external checkout tool use git submodules
• What features should the NorESM external checkout tool have
  • Required features
  • Desired features
  • Questions

Should the NorESM external checkout tool use git submodules

• git fleximod checks out externals as submodules. Because the tool uses the .gitmodules file to document the desired tags, it also requires that the submodules exist in the commit.
• manage_externals does not use submodules. Each external is checked out as a separate clone.

What features should the NorESM external checkout tool have?

• Required features

  • The tool reads a configuration file (e.g., External.cfg, .gitmodules) and checks out the correct reference of each external
  • The tool is recursive, it has a way to make sure that sub-externals are properly checked out
  • If a new commit is checked out, the tool can switch the checked-out reference for each external
  • The tool will not delete any local (i.e, not committed to a repository) work without a --force option
  • The tool can report the status of the external checkouts in a few easy to read and understand formats
    • If the tool uses git submodules, it must include submodule synchronization information in status reports
  • git submodules in a checked out external will be updated correctly
  • The tool is stable and will work on a broad range of operating systems and versions
  • The tool must support sparse checkouts and sparse cloning
  • The tool must have well defined behavior and error conditions
    • Any behavior not explicitly defined is an error?
  • The configuration must allow for different externals to be checked out if a repository is a sub external or a top-level repository
    • Configuration files (<name>.cfg) manage this by using one file for top-level checkouts and one for sub-externals
    • If submodules are used, there must be some way to control which submodules are checked out for a sub-external.
      • One possibility is an option to disallow duplicate checkouts.

• Desired features

  • The tool can be invoked on a limited set of externals
  • The tool has an exclusion mechanism that can be invoked on a top-level external or a sub-external
  • The tool should provide a way to update the configuration file based on the currently checked out externals
    • This update may or may not commit any changes, possibly determined by a configuration flag
    • If the tool uses git submodules, it should also update the top level repository to match
  • The tool should support shallow clones
  • If a branch is specified instead of a tag or hash, it should be checked out as a branch, not a detached HEAD.

• Questions

  • Should the tool be atomic? That is, if any external update fails, should the tool restore the previous settings?

[oyvindseland]

I must admit that for my part I do not have very strong preferences, so hard to unless I really go through the details. I have a couple of comments though.
-It should write out a log of what your command actually did to your settings / existing checkouts.

Should the tool be atomic? That is, if any external update fails, should the tool restore the previous settings?
I guess that this will be hard, but would a dry run be possible / easier to implement than a full restore, ideally with proper warnings for fails?

Any behavior not explicitly defined is an error?
Depends on how often this is expected to occur. At least I think an error should be defined if the behavior risk doing something irreversible to the system (the --force in example). If it is not critical I think that ideally the behavior should be defined as an error if the error message comes with a suggestion for how the issue can be resolved, syntax, bounds etc.