Requirements

These requirements have been harvested from a number of places. Obviously much is informed by our current build system: FCM.

The bulk come from a previous project to evaluate the existing build systems as potential replacements to FCM. These have been extended and clarified for this document. Furthermore they have been separated into functional and technical requirements, then ranked by priority.

One structural change this project intends to make is to separate out extraction and build into separate tools. Thus separate sets of requirements are presented here.

This is a living document and will be updated periodically by the project team & board to account for emerging or obsolete requirements.

Extract System

The extraction system has the job of preparing a source tree to build. It does this by collating source from a number of locations.

Functional Requirements

Items at the same level are ordered by priority. It may be desirable to implement an item at one level before all items below the previous item are complete.

1. User interface shall be
   1. Programmable API - most suitable for complex situations
   2. Configuration file - most suitable for simple situations
   3. Zero configuration - for very simple situations
      1. If the project is presented in a certain form and does not require complex rules the tool shall infer what to do.
2. A single source tree shall be synthesised from multiple trees
   1. The resulting source tree must be suitable for the build system to operate on.
   2. Filename uniqueness must be preserved using a namespace scheme.
   3. It shall be possible to install the resulting tree to a remote file system.
   4. Extracted source may contain further configuration for the extract system.
3. Extract and merge source from a variety of locations:
   1. Local file system
   2. Subversion VCS
      1. Repository
      2. Working copy
   3. Remote file system
   4. Git VCS
      1. Remote clone
      2. Local clone
   5. Archive files
      1. tarballs
      2. zip files
   6. Nested repositories, e.g. a tarball held by a VCS
4. Extraction should be possible from a sub-tree within a repository.
5. Merge several versions of the same source tree (e.g. branches held in VCS)
   1. Failure to merge automatically is an error condition.
6. Make use of temporary storage (e.g. RAM disk) for temporary files.
7. It should be possible to filter the incoming files based on name.
   1. Both include and exclude filters should be supported.
8. Parallelise activity where possible. e.g. extracts from different locations.
9. Operate incrementally where possible.
   1. Extract only those files which have changed since the last extract.
   2. This may be disabled for a full extract.
10. Schedule parallel tasks such that the longest to complete on the last run go first.
    1. Use arbitrary order for the first run.

Technical Requirements

Items are unordered, they tend to describe things which are always true.

1. Documentation required
   1. API
   2. User interface
   3. User guide
   4. Tutorial
2. Good test coverage
   1. Unit tests
   2. Integration tests (may not be necessary)
   3. System tests
3. Easy to deploy using recognised standard methods: PIP/PyPi
4. Moderate use of 3rd party libraries.
5. Shall be implemented using Python3.
6. The same configuration should always lead to the same source tree, given the same repositories.
7. Sub-systems shall be modular such that they may be used in isolation by other tools.
   1. Shall be easy to re-use components.
8. Support for source repositories should be modular allowing easy extension to support additional options.
9. Local file systems shall include NFS mounts
10. Remote file systems are those accessible explicitly over a network. e.g. SCP/rsync
11. Network communication shall always flow to the compiling system, never from it. e.g. Desktop to HPC, never the other way round.
12. Persistent data shall be compressed to reduce disc usage where it makes sense to do so.

Possible Future Requirements

Sorted into priority order.

1. Extract from FTP server.
2. Extract from HTTP server.

Build System

The buid system's job is to take the provided source tree and produce a number of artifacts from it, be they executable applications, libraries, documentation or other product.

Functional Requirements

Items at the same level are ordered by priority. It may be desirable to implement an item at one level before all items below the previous item are complete.

1. User interface shall be:
   1. Programmable API - most suitable for complex situations
   2. Configuration file - most suitable for simple situations
   3. Zero configuration - for very simple situations
      1. If the project is presented in a certain form and does not require complex rules the tool shall infer what to do.
2. Interpret source trees prior to build
   1. Automatic discovery of source files.
   2. Ability to ignore (filter out) source files based on patterns.
      1. Per tree
      2. Per product?
3. Use dependency analysis in order to determine the directed acyclic graph (DAG) of tasks needed to perform the build.
   1. Reject circular dependencies.
   2. Reject missing dependencies.
   3. Reject duplicate implementations, i.e. same program symbol with multiple implementations.
   4. Visualise the graph for debugging and development
   5. Exclude external dependencies.
   6. Preprocessing which could generate new dependencies (e.g. #include) will need to be done prior to analysis.
   7. Recognise that bind(c) in Fortran creates a dependency on an a symbol in an object or library file. These files may be created by a wide variety of compilers including C/C++.
4. Support the building of all or a subset of end products:
   1. Anything which is the product of a transformation may be an end product.
      1. May be specified individually or as "everything in this directory."
   2. Only source required to build the end products shall be compiled.
   3. Executables
      1. These may be automatically discovered from the source.
   4. Libraries
      1. Static or dynamic (or both!)
      2. Collate additional files needed when compiling against the library. Note that only those relating to the intended API should be exposed.
         1. C header files
         2. Fortran module files
5. Allow intermediate artefacts to be replaced with hand written versions.
   1. This implies that the task which would normally create them is not run.
   2. The user should be aware that this mechanism is not for permanent use. Rather it should be an interim solution to overcome an immediate problem.
6. Support performing additional actions upon completion of the build and/or other intermediate stages of the build.
   1. Installation of final product/s
   2. Preserving intermediates from temporary build locations
      1. This may include files produced by a profiling tool or other debugging/diagnostic features of (e.g.) the compiler
7. Execute compiled products as part of the build, e.g. unit tests, source generators.
8. Support the ability for tasks to output multiple versions. e.g. MPI vs. serial, Static vs dynamic libraries or different build options.
9. Make use of temporary storage (e.g. RAM disc) for intermediate files.
10. Intermediate products shall be retained to speed compilation.
    1. Object files
    2. (Fortran) Module files
    3. Intermediate libraries
    4. Generated source
11. Only sources which have changed or things which depend on rebuilt intermediates (including configuration) should be rebuilt.
    1. This may be disabled for a full rebuild.
12. Support task options
    1. Hierarchy of scope
       1. Global, installation wide
       2. Per Project (single build instance)
       3. Per product (executable, library, etc)
       4. Per directory
       5. Per source file
    2. Options for tasks include
       1. Preprocessor
          1. Include directory for C header files
       2. Compiler
          1. Per compiler version
          2. Directory for Fortran module files
          3. Debug
          4. Optimisation
          5. Features
             1. OpenMP
             2. OpenAcc
       3. Linker
          1. Per linker version
          2. Directories containing external library files
          3. Debug
          4. Optimisation
          5. Features
             1. OpenMP
             2. OpenAcc
13. Parallelise for performance
    1. Preprocessing
    2. Dependency analysis
    3. Compile
14. Support tools (e.g. profiling) which "wrap" the compiler. (e.g. proftool -profargs ifort -compargs)
15. Build Fortran module files only in the first phase obeying dependency information. In the second phase object files are built in parallel in any order.
16. Store intermediate files in a compressed archive.
17. Support cache of intermediate files to speed building.
    1. Only use cached intermediate if its prerequisites have not changed. i.e. source file, module file and configuration
    2. Multiple caches may be available.
18. When dispatching tasks prefer those which took a long time in the previous run.
    1. Only works for parallel tasks
       1. Dependency analysis
       2. Building object files (not module files)
    2. Use arbitrary order for the first run where there is no duration data.

Technical Requirements

Items are unordered, they tend to describe things which are always true.

1. Documentation required
   1. API
   2. User interface
   3. User guide
   4. Tutorial
2. Good test coverage
   1. Unit tests
   2. Integration tests (may not be necessary)
   3. System tests
3. Easy to deploy using recognised standard methods
   1. PIP
   2. PyPi
4. Moderate use of 3rd party libraries.
5. Shall be implemented using Python3.
6. Repeatable behaviour given identical inputs.
7. Source files shall be uniquely identifiable.
8. Support use against a large number of sites
   1. Each site may have many platforms
      1. Each platform may have multiple compilers
9. Support language standards at least as recent as:
   1. Fortran 2008
      1. Support all program units: Program, Subroutine, Function, Module and Submodule.
   2. C 2018
   3. C++ 2017
10. Sub-systems shall be modular such that they may be used in isolation by other tools. e.g. Dependency analyser.

1. Shall be easy to re-use components.

11. It shall be easy to add support for new language source code for both build system developers and users.
12. Different dependency trees may pertain for compilation and linking.
13. A task shall be any transformation which:
    1. Takes zero or more inputs and produces zero or more outputs.
    2. Shall include compilation, linking and various source generators
       1. PSyclone
       2. pFUnit
       3. User defined
14. Libraries may be intermediate as well as end-product.
15. More than one Fortran module may appear in a single source file.
16. When handling inclusions such as C header files or Fortran include, support must be provided for cascading inclusions.
17. Detecting changes to source files should utilise a hash rather than "last write" timestamp.
18. Products should depend on their configuration as well as their source and prerequisites.
19. Dependency analysis must be extendable with new dependency relationships by both build system developers and users.
20. Compile and link have different dependencies. e.g. Fortran source depends on module files while link requires object and library files.
21. Tasks shall be implemented such that it is easy to add additional ones either by build system developers or users.
22. Recently a module system similar to Fortran's was accepted into the next version of the C++ standard. This can not be supported immediately but the implementation must not prevent it in the future.
23. Persistent data shall be stored in a compressed archive where it makes sense to do so.

Possible Future Requirements

Sorted into priority order.

1. Generate code browser during build.
2. For Fortran, only dependency module files should be considered when determining rebuild. A change to the object file does not affect the ABI.
3. Support the use of different compilers for different source trees. How this interacts with incompatible modules needs thought. Requirement from ABOM, consult with them.
4. Trigger incremental rebuild from external dependencies.
5. Integrate with IDEs.
6. Source generation of Fortran interfaces for extern "C" functions and C header files for bind(c) procedures.
7. Support use through GUIs such as Jupyter notebooks.
8. Support for C++20 modules.
9. Support for legacy FCM features
   1. Interface files
   2. "Depends on" comments