Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to fparser 0 2 #373

Merged
merged 58 commits into from
Jan 15, 2025
Merged
Changes from 15 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
282f068
Merge pull request #19 from hiker/linker-lib-flags
hiker Sep 24, 2024
3b6e0bd
Support new and old style of PSyclone command line (no more nemo api …
hiker Sep 26, 2024
16d3ff5
Fix mypy errors.
hiker Sep 26, 2024
71fd1ae
Added missing tests for calling psyclone, and converting old style to…
hiker Sep 30, 2024
ec4c0f6
Updated comment.
hiker Sep 30, 2024
b9aabf8
Removed mixing, use a simple regex instead.
hiker Oct 17, 2024
8ee10e8
Added support for ifx/icx compiler as intel-llvm class.
hiker Oct 18, 2024
d7b2008
Added support for nvidia compiler.
hiker Oct 18, 2024
9005b3b
Add preliminary support for Cray compiler.
hiker Oct 18, 2024
8771e80
Added Cray compiler wrapper ftn and cc.
hiker Oct 18, 2024
0188050
Follow a more consistent naming scheme for crays, even though the nat…
hiker Oct 22, 2024
3c569bd
Changed names again.
hiker Oct 22, 2024
edc5fcd
Renamed cray compiler wrapper to be CrayCcWrapper and CrayFtnWrapper,…
hiker Nov 11, 2024
f6a70c8
Fixed incorrect name in comments.
hiker Nov 11, 2024
4f0e70f
Merge pull request #28 from hiker/additional_compilers
lukehoffmann Nov 11, 2024
58caecf
Merge branch 'compiler_wrapper' into bom_master
hiker Nov 11, 2024
5452d70
Merge branch 'linker-lib-flags' into additional_compilers
hiker Nov 12, 2024
d54d94c
Merge branch 'linker-lib-flags' into bom_master
hiker Nov 12, 2024
605e7e5
Merge branch 'additional_compilers' into bom_master
hiker Nov 12, 2024
70ad4b1
Merge branch 'linker-lib-flags' into additional_compilers
hiker Nov 12, 2024
b70f98f
Merge branch 'linker-lib-flags' into additional_compilers
hiker Nov 12, 2024
2f7e3ba
Merge branch 'linker-lib-flags' into additional_compilers
hiker Nov 12, 2024
7a2eb59
Additional compilers (#349)
hiker Nov 12, 2024
bd1d318
Merge branch 'dev' into additional_compilers
hiker Nov 12, 2024
2148fb7
Merge branch 'compiler_wrapper' into update_psyclone_to_support_next_…
hiker Nov 19, 2024
20fe928
Merge branch 'update_psyclone_to_support_next_release_syntax' into ad…
hiker Nov 19, 2024
f7b49e0
Merge branch 'linker-lib-flags' into additional_compilers
hiker Nov 21, 2024
a493c53
Support new and old style of PSyclone command line (no more nemo api …
hiker Sep 26, 2024
824851d
Fix mypy errors.
hiker Sep 26, 2024
16a125c
Added missing tests for calling psyclone, and converting old style to…
hiker Sep 30, 2024
fc19283
Added shell tool.
hiker Oct 23, 2024
730a824
Try to make mypy happy.
hiker Oct 23, 2024
6e280d9
Removed debug code.
hiker Oct 23, 2024
6c3f1c2
ToolRepository now only returns default that are available. Updated t…
hiker Oct 23, 2024
ae61d4a
Fixed typos and coding style.
hiker Nov 21, 2024
e7c2c83
Support new and old style of PSyclone command line (no more nemo api …
hiker Sep 26, 2024
e2051f2
Fix mypy errors.
hiker Sep 26, 2024
0ad85ee
Added missing tests for calling psyclone, and converting old style to…
hiker Sep 30, 2024
890b50d
Updated comment.
hiker Sep 30, 2024
032ab26
Fixed failing tests.
hiker Nov 21, 2024
7168f42
Merge branch 'additional_compilers_clean' into psyclone_3_support_clean
hiker Nov 21, 2024
70c083e
Merge branch 'psyclone_3_support_clean' into add_shell_tool_clean
hiker Nov 21, 2024
8753d0c
Updated fparser dependency to version 0.2.
hiker Nov 28, 2024
634d28c
Replace old code for handling sentinels with triggering this behaviou…
hiker Nov 29, 2024
78697bf
Fixed tests for latest changes.
hiker Nov 29, 2024
c82cedf
Removed invalid openmp continuation line - since now fparser fails wh…
hiker Nov 29, 2024
652db98
Added test for disabled openmp parsing. Updated test to work with new…
hiker Nov 29, 2024
ea7e428
Coding style changes.
hiker Nov 29, 2024
137d346
Fix flake issues.
hiker Nov 29, 2024
fa0cb5d
Fixed double _.
hiker Nov 29, 2024
63e77e5
Merge branch 'psyclone_3_support_clean' into add_shell_tool_clean
hiker Nov 29, 2024
810da77
Merge branch 'add_shell_tool_clean' into update_to_fparser_0_2
hiker Nov 29, 2024
bbdb380
Merge branch 'develop' into add_shell_tool_clean
hiker Dec 2, 2024
1335878
Merge branch 'add_shell_tool_clean' into update_to_fparser_0_2
hiker Dec 2, 2024
d032d8a
Merge branch 'develop' into update_to_fparser_0_2
hiker Jan 9, 2025
ccc8a39
Removed more accesses to private members.
hiker Jan 9, 2025
34f4985
Added missing type hint.
hiker Jan 9, 2025
686f990
Make flake8 happy.
hiker Jan 9, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion Documentation/source/site-specific-config.rst
Original file line number Diff line number Diff line change
@@ -148,6 +148,20 @@ rsync, ar, ...).
tool_box = ToolBox()
default_c_compiler = tool_box.get_tool(Category.C_COMPILER)

There is special handling for compilers and linkers: the build
configuration stores the information if an MPI and/or OpenMP build
is requested. So when a default tool is requested by the ToolBox
from the ToolRepository (i.e. when the user has not added specific
compilers or linkers), this information is taken into account, and
only a compiler that will fulfil the requirements is returned. For
example, if you have `gfortran` and `mpif90-gfortran` defined in this
order in the ToolRepository, and request the default compiler for an
MPI build, the `mpif90-gfortran` instance is returned, not `gfortran`.
On the other hand, if no MPI is requested, an MPI-enabled compiler
might be returned, which does not affect the final result, since
an MPI compiler just adds include- and library-paths.


Compiler Wrapper
================
Fab supports the concept of a compiler wrapper, which is typically
@@ -205,7 +219,6 @@ applied to the wrapper as well:
assert my_mpicc.flags == ["-a", "-b"]



TODO
====
At this stage compiler flags are still set in the corresponding Fab
2 changes: 1 addition & 1 deletion run_configs/gcom/build_gcom_ar.py
Original file line number Diff line number Diff line change
@@ -15,7 +15,7 @@
if __name__ == '__main__':

with BuildConfig(project_label='gcom object archive $compiler',
mpi=False, openmp=False, tool_box=ToolBox()) as state:
mpi=True, openmp=False, tool_box=ToolBox()) as state:
common_build_steps(state)
archive_objects(state, output_fpath='$output/libgcom.a')
cleanup_prebuilds(state, all_unused=True)
2 changes: 1 addition & 1 deletion run_configs/gcom/build_gcom_so.py
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@
parsed_args = arg_parser.parse_args()

with BuildConfig(project_label='gcom shared library $compiler',
mpi=False, openmp=False, tool_box=ToolBox()) as state:
mpi=True, openmp=False, tool_box=ToolBox()) as state:
common_build_steps(state, fpic=True)
link_shared_object(state, output_fpath='$output/libgcom.so')
cleanup_prebuilds(state, all_unused=True)
2 changes: 1 addition & 1 deletion run_configs/gcom/grab_gcom.py
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@

# we put this here so the two build configs can read its source_root
grab_config = BuildConfig(project_label=f'gcom_source {revision}',
mpi=False, openmp=False, tool_box=ToolBox())
tool_box=ToolBox())


if __name__ == '__main__':
4 changes: 2 additions & 2 deletions run_configs/lfric/grab_lfric.py
Original file line number Diff line number Diff line change
@@ -18,10 +18,10 @@
tool_box = ToolBox()
lfric_source_config = BuildConfig(
project_label=f'lfric source {LFRIC_REVISION}',
mpi=False, openmp=False, tool_box=tool_box)
tool_box=tool_box)
gpl_utils_source_config = BuildConfig(
project_label=f'lfric source {LFRIC_REVISION}',
mpi=False, openmp=False, tool_box=tool_box)
tool_box=tool_box)


if __name__ == '__main__':
6 changes: 2 additions & 4 deletions run_configs/lfric/gungho.py
Original file line number Diff line number Diff line change
@@ -33,7 +33,7 @@
gpl_utils_source = gpl_utils_source_config.source_root / 'gpl_utils'

with BuildConfig(project_label='gungho $compiler $two_stage',
mpi=False, openmp=False, tool_box=ToolBox()) as state:
mpi=True, openmp=True, tool_box=ToolBox()) as state:
grab_folder(state, src=lfric_source / 'infrastructure/source/', dst_label='')
grab_folder(state, src=lfric_source / 'components/driver/source/', dst_label='')
grab_folder(state, src=lfric_source / 'components' / 'inventory' / 'source', dst_label='')
@@ -87,7 +87,7 @@
state,
common_flags=[
'-c',
'-ffree-line-length-none', '-fopenmp',
'-ffree-line-length-none',
'-g',
'-std=f2008',

@@ -104,8 +104,6 @@
link_exe(
state,
flags=[
'-fopenmp',

'-lyaxt', '-lyaxt_c', '-lnetcdff', '-lnetcdf', '-lhdf5', # EXTERNAL_DYNAMIC_LIBRARIES
'-lxios', # EXTERNAL_STATIC_LIBRARIES
'-lstdc++',
2 changes: 1 addition & 1 deletion run_configs/lfric/mesh_tools.py
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@
psyclone_overrides = Path(__file__).parent / 'mesh_tools_overrides'

with BuildConfig(project_label='mesh tools $compiler $two_stage',
mpi=False, openmp=False, tool_box=ToolBox()) as state:
mpi=True, openmp=False, tool_box=ToolBox()) as state:
grab_folder(state, src=lfric_source / 'infrastructure/source/', dst_label='')
grab_folder(state, src=lfric_source / 'mesh_tools/source/', dst_label='')
grab_folder(state, src=lfric_source / 'components/science/source/', dst_label='')
2 changes: 1 addition & 1 deletion run_configs/tiny_fortran/build_tiny_fortran.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ def __init__(self):
tool_box.add_tool(Linker(compiler=fc))

with BuildConfig(project_label='tiny_fortran $compiler',
mpi=False, openmp=False, tool_box=tool_box) as state:
tool_box=tool_box) as state:
git_checkout(state, src='https://github.com/metomi/fab-test-data.git',
revision='main', dst_label='src')

4 changes: 3 additions & 1 deletion run_configs/um/build_um.py
Original file line number Diff line number Diff line change
@@ -124,9 +124,11 @@ def replace_in_file(inpath, outpath, find, replace):
revision = 'vn12.1'
um_revision = revision.replace('vn', 'um')

# The original build script disabled openmp, so for now
# we keep this disabled.
state = BuildConfig(
project_label=f'um atmos safe {revision} $compiler $two_stage',
mpi=False, openmp=False, tool_box=ToolBox())
mpi=True, openmp=False, tool_box=ToolBox())

# compiler-specific flags
compiler = state.tool_box[Category.FORTRAN_COMPILER]
17 changes: 11 additions & 6 deletions source/fab/build_config.py
Original file line number Diff line number Diff line change
@@ -42,8 +42,8 @@ class BuildConfig():
"""
def __init__(self, project_label: str,
tool_box: ToolBox,
mpi: bool,
openmp: bool,
mpi: bool = False,
openmp: bool = False,
multiprocessing: bool = True,
n_procs: Optional[int] = None,
reuse_artefacts: bool = False,
@@ -56,9 +56,13 @@ def __init__(self, project_label: str,
created from this name, with spaces replaced by underscores.
:param tool_box: The ToolBox with all tools to use in the build.
:param mpi: whether the project uses MPI or not. This is used to
pick a default compiler (if not explicitly set in the ToolBox),
and controls PSyclone parameters.
:param openmp: whether the project should use OpenMP or not.
pick a default compiler (if none is explicitly set in the
ToolBox), and controls PSyclone parameters.
:param openmp: as with `mpi`, this controls whether the project is
using OpenMP or not. This is used to pick a default compiler
(if none is explicitly set in the ToolBox). The compiler-specific
flag to enable OpenMP will automatically be added when compiling
and linking.
:param multiprocessing:
An option to disable multiprocessing to aid debugging.
:param n_procs:
@@ -85,7 +89,8 @@ def __init__(self, project_label: str,
self._openmp = openmp
self.two_stage = two_stage
self.verbose = verbose
compiler = tool_box.get_tool(Category.FORTRAN_COMPILER, mpi=mpi)
compiler = tool_box.get_tool(Category.FORTRAN_COMPILER, mpi=mpi,
openmp=openmp)
project_label = Template(project_label).safe_substitute(
compiler=compiler.name,
two_stage=f'{int(two_stage)+1}stage')
2 changes: 1 addition & 1 deletion source/fab/cli.py
Original file line number Diff line number Diff line change
@@ -34,7 +34,7 @@ def _generic_build_config(folder: Path, kwargs=None) -> BuildConfig:
# Set the default Fortran compiler as linker (otherwise e.g. the
# C compiler might be used in linking, requiring additional flags)
tr = ToolRepository()
fc = tr.get_default(Category.FORTRAN_COMPILER, mpi=False)
fc = tr.get_default(Category.FORTRAN_COMPILER, mpi=False, openmp=False)
# TODO: This assumes a mapping of compiler name to the corresponding
# linker name (i.e. `linker-gfortran` or `linker-ifort`). Still, that's
# better than hard-coding gnu here.
5 changes: 3 additions & 2 deletions source/fab/steps/compile_c.py
Original file line number Diff line number Diff line change
@@ -81,7 +81,8 @@ def compile_c(config, common_flags: Optional[List[str]] = None,
# No need to look for compiler etc if there is nothing to do
return

compiler = config.tool_box.get_tool(Category.C_COMPILER, config.mpi)
compiler = config.tool_box.get_tool(Category.C_COMPILER, mpi=config.mpi,
openmp=config.openmp)
logger.info(f'C compiler is {compiler}')

mp_payload = MpCommonArgs(config=config, flags=flags)
@@ -147,7 +148,7 @@ def _compile_file(arg: Tuple[AnalysedC, MpCommonArgs]):
compiler.compile_file(analysed_file.fpath, obj_file_prebuild,
openmp=config.openmp,
add_flags=flags)
except Exception as err:
except RuntimeError as err:
return FabException(f"error compiling "
f"{analysed_file.fpath}:\n{err}")

3 changes: 2 additions & 1 deletion source/fab/steps/link.py
Original file line number Diff line number Diff line change
@@ -61,7 +61,8 @@ def link_exe(config,
output from compiler steps, which typically is the expected behaviour.

"""
linker = config.tool_box.get_tool(Category.LINKER, config.mpi)
linker = config.tool_box.get_tool(Category.LINKER, mpi=config.mpi,
openmp=config.openmp)
logger.info(f'Linker is {linker.name}')

libs = libs or []
15 changes: 13 additions & 2 deletions source/fab/tools/compiler.py
Original file line number Diff line number Diff line change
@@ -31,12 +31,13 @@ class Compiler(CompilerSuiteTool):
:param exec_name: name of the executable to start.
:param suite: name of the compiler suite this tool belongs to.
:param category: the Category (C_COMPILER or FORTRAN_COMPILER).
:param mpi: whether the compiler or linker support MPI.
:param compile_flag: the compilation flag to use when only requesting
compilation (not linking).
:param mpi: whether MPI is supported by this compiler or not.
:param output_flag: the compilation flag to use to indicate the name
of the output file
:param openmp_flag: the flag to use to enable OpenMP
:param openmp_flag: the flag to use to enable OpenMP. If no flag is
specified, it is assumed that the compiler does not support OpenMP.
:param availability_option: a command line option for the tool to test
if the tool is available on the current system. Defaults to
`--version`.
@@ -66,6 +67,12 @@ def mpi(self) -> bool:
'''Returns whether this compiler supports MPI or not.'''
return self._mpi

@property
def openmp(self) -> bool:
''':returns: if the compiler supports openmp or not
'''
return self._openmp_flag != ""

@property
def openmp_flag(self) -> str:
'''Returns the flag to enable OpenMP.'''
@@ -370,6 +377,7 @@ class Gcc(GnuVersionHandling, CCompiler):

:param name: name of this compiler.
:param exec_name: name of the executable.
:param mpi: whether the compiler supports MPI.
'''
def __init__(self,
name: str = "gcc",
@@ -385,6 +393,7 @@ class Gfortran(GnuVersionHandling, FortranCompiler):

:param name: name of this compiler.
:param exec_name: name of the executable.
:param mpi: whether the compiler supports MPI.
'''

def __init__(self, name: str = "gfortran",
@@ -435,6 +444,7 @@ class Icc(IntelVersionHandling, CCompiler):

:param name: name of this compiler.
:param exec_name: name of the executable.
:param mpi: whether the compiler supports MPI.
'''

def __init__(self, name: str = "icc", exec_name: str = "icc"):
@@ -448,6 +458,7 @@ class Ifort(IntelVersionHandling, FortranCompiler):

:param name: name of this compiler.
:param exec_name: name of the executable.
:param mpi: whether the compiler supports MPI.
'''

def __init__(self, name: str = "ifort", exec_name: str = "ifort"):
13 changes: 9 additions & 4 deletions source/fab/tools/tool_box.py
Original file line number Diff line number Diff line change
@@ -46,13 +46,18 @@ def add_tool(self, tool: Tool,
f"'{tool}'.")
self._all_tools[tool.category] = tool

def get_tool(self, category: Category, mpi: Optional[bool] = None) -> Tool:
def get_tool(self, category: Category, mpi: Optional[bool] = None,
openmp: Optional[bool] = None) -> Tool:
'''Returns the tool for the specified category.

:param category: the name of the category in which to look
for the tool.
:param mpi: if no compiler or linker is specified when requesting one,
use the MPI setting to find an appropriate default.
:param mpi: if no compiler or linker is explicitly specified in this
tool box, use the MPI and OpenMP setting to find an appropriate
default from the tool repository.
:param mpi: if no compiler or linker is explicitly specified in this
tool box, use the MPI and OpenMP setting to find an appropriate
default from the tool repository.

:raises KeyError: if the category is not known.
'''
@@ -69,6 +74,6 @@ def get_tool(self, category: Category, mpi: Optional[bool] = None) -> Tool:
# from the ToolRepository, and add it, so we don't need to look
# it up again later.
tr = ToolRepository()
tool = tr.get_default(category, mpi=mpi)
tool = tr.get_default(category, mpi=mpi, openmp=openmp)
self._all_tools[category] = tool
return tool
26 changes: 23 additions & 3 deletions source/fab/tools/tool_repository.py
Original file line number Diff line number Diff line change
@@ -138,7 +138,8 @@ def set_default_compiler_suite(self, suite: str):
f"in the suite '{suite}'.")

def get_default(self, category: Category,
mpi: Optional[bool] = None):
mpi: Optional[bool] = None,
openmp: Optional[bool] = None):
'''Returns the default tool for a given category. For most tools
that will be the first entry in the list of tools. The exception
are compilers and linker: in this case it must be specified if
@@ -147,6 +148,7 @@ def get_default(self, category: Category,

:param category: the category for which to return the default tool.
:param mpi: if a compiler or linker is required that supports MPI.
:param open: if a compiler or linker is required that supports OpenMP.

:raises KeyError: if the category does not exist.
:raises RuntimeError: if no compiler/linker is found with the
@@ -165,11 +167,29 @@ def get_default(self, category: Category,
raise RuntimeError(f"Invalid or missing mpi specification "
f"for '{category}'.")

if not isinstance(openmp, bool):
raise RuntimeError(f"Invalid or missing openmp specification "
f"for '{category}'.")

for tool in self[category]:
# If the tool supports/does not support MPI, return the first one
# If OpenMP is request, but the tool does not support openmp,
# ignore it.
if openmp and not tool.openmp:
continue
# If the tool supports/does not support MPI, return it.
if mpi == tool.mpi:
return tool

# Don't bother returning an MPI enabled tool if no-MPI is requested -
# that seems to be an unlikely scenario.
raise RuntimeError(f"Could not find '{category}' that supports MPI.")
if mpi:
if openmp:
raise RuntimeError(f"Could not find '{category}' that "
f"supports MPI and OpenMP.")
raise RuntimeError(f"Could not find '{category}' that "
f"supports MPI.")

if openmp:
raise RuntimeError(f"Could not find '{category}' that "
f"supports OpenMP.")
raise RuntimeError(f"Could not find any '{category}'.")
Loading