Python / Numpy compilation and cross compatibility #13

tomsail · 2023-11-17T16:37:11Z

EDIT (after applying patch provided by @nicogodet, see details below):

Recapitulative table:

python \ numpy	1.19	1.20	1.21	1.22	1.23	1.24	1.25	1.26
3.8	✔️	✔️	✔️	✔️	✔️	✔️	❌	❌
3.9	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️
3.10	❌	❌	✔️	✔️	✔️	✔️	✔️	✔️
3.11	❌	❌	❌	❌	✔️	✔️	✔️	✔️
3.12	❌	❌	❌	❌	❌	❌	❌	❌

❌ Compiling fails
🔶 API fails
✔️ everything works

TODO:

check the triangulation mtri.triangulation.Triangulation/mtri.Triangulation to apply depending on the numpy version

Note:
for the API test, a simple test was done :

mamba create -n "$env_name" opentelemac python="$py" numpy="$np" -y
mamba activate "$env_name"
TELEMAC_DIR="<LOCAL_DIR_THAT_CONTAINS_EXAMPLES>"
example_dir="$TELEMAC_DIR/examples/python3/TelApy_api"
destination_dir="$CONDA_PREFIX/opentelemac/examples/telemac2d"
mkdir -p "$destination_dir"
cp -r "$TELEMAC_DIR/examples/telemac2d/gouttedo/" "$destination_dir/"
cd "$example_dir" || exit
python t2d.py

@nicogodet validate_telemac.py does not work because package does not contain the examples. I f you have a fix for this, I'm keen to hear.

The text was updated successfully, but these errors were encountered:

nicogodet · 2023-11-17T17:03:02Z

What's the error message when it fails ?

Compiling main is PIA (sha256 will change multiple time a day), I would stick to released version with patch.

To generate a patch, you can

git checkout v8p4 branch
create a new branch from v8p4 : v8p4-patch-f2py
cherry pick the merge commit
git diff v8p4..v8p4-patch-f2py > f2py.patch

f2py.patch

tomsail · 2023-11-17T17:11:57Z

What's the error message when it fails ?

Mainly during one of the API steps, i.e.


Building the Python API 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    ~> Wrap_api built
    ~> Compiling Modules api
    ~> First part of f2py for api passed

It can be during the wrap, the first or second step, it depends.. I haven't printed out the whole compilation process, just the outcome of it.

Compiling main is PIA (sha256 will change multiple time a day), I would stick to released version with patch.

To generate a patch, you can

git checkout v8p4 branch

create a new branch from v8p4 : v8p4-patch-f2py

cherry pick the merge commit

git diff v8p4..v8p4-patch-f2py > f2py.patch

f2py.patch

Thanks that's a great hack! I had seen in one of your meta.yaml that you could include the patch in the conda build process, like so correct?

source:
  url: <source-url>
  sha256: <source-sha256>
  patches:
    - f2py.patch

I will finish the first matrix build I started and add a second table with the patch fix

nicogodet · 2023-11-18T17:10:24Z

I opened conda-forge/staged-recipes#24529
We need to manually specify python and numpy version in conda_build_config.yaml and add condition on patch depending numpy version.

nicogodet · 2023-11-18T17:11:45Z

With patch in PR, il was able to build on py3.11 and numpy 1.25 locally but it fails with same recipe on azure pipeline

tomsail · 2023-11-19T16:34:17Z

what is the error on azure?
With the patch locally, doing the conda build, I have (with python 3.10, numpy 1.25)

  - completed: .../opentelemac/sources/api/api_run_t3d.f                     
    - completed: .../opentelemac/sources/api/api_run_wac.f                     
    - completed: .../opentelemac/sources/api/api_interface.f                   
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:146:30:

  146 |      &                      0,0,IERR)
      |                              1
Error: Non-variable expression in variable definition context (actual argument to INTENT = OUT/INOUT) at (1)
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:146:72:

  146 |      &                      0,0,IERR)
      |                                                                        1
Error: More actual than formal arguments in procedure call at (1)
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:164:30:

  164 |      &                      0,0,IERR)
      |                              1
Error: Non-variable expression in variable definition context (actual argument to INTENT = OUT/INOUT) at (1)
/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f:164:72:

  164 |      &                      0,0,IERR)
      |                                                                        1
Error: More actual than formal arguments in procedure call at (1)
    - completed: .../opentelemac/sources/api/homere_api.F                      
Traceback (most recent call last):\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ ]  99%  | 1s
  File "/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/scripts/python3/compile_telemac.py", line 142, in <module>
    main()
  File "/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/scripts/python3/compile_telemac.py", line 129, in main
    compile_cmdf(options.ncsize, modules, options.verbose)
  File "/home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/scripts/python3/compilation/compil_tools.py", line 1298, in compile_cmdf
    raise TelemacException(\
utils.exceptions.TelemacException: 
        +> failed: .../opentelemac/sources/api/example_cpl_t2d_sis.f
... The following command failed for the reason in the listing
mpif90 -c -O2 -fPIC -cpp -fconvert=big-endian -frecord-marker=4 -DHAVE_I16 -DHAVE_MPI -DHAVE_MUMPS -DHAVE_GOTM -DHAVE_AED2  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/special  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/damocles  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/parallel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/hermes  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/bief  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/nestor  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/ad  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/tomawac  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/sisyphe  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/waqtel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/khione  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/gaia  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/telemac2d  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/gretel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/utils/partel  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/telemac3d  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/artemis  -I /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/builds/gnu.dynamic/obj/api    /home/tomsail/miniconda3/conda-bld/opentelemac_1700411180981/work/opentelemac/sources/api/example_cpl_t2d_sis.f

nicogodet · 2023-11-19T16:37:12Z

Retry with patch from https://github.com/conda-forge/staged-recipes/blob/65a32bc7fdb6112b6c841249e5eed27642f212cd/recipes/opentelemac/patches/f2py.patch

tomsail · 2023-11-19T17:32:15Z

I have this error, would you know why?

Using :

{% set name = "opentelemac" %}
{% set version = "v8p4r1" %}
{% set build = 0 %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  # - url: https://gitlab.pam-retd.fr/otm/telemac-mascaret/-/archive/{{ version }}/telemac-mascaret-{{ version }}.tar.gz
  - url: https://github.com/nicogodet/otm-sources/releases/download/{{ version }}/telemac-mascaret-{{ version }}.tar.gz
    folder: opentelemac
    # sha256: b48a0ee5cb43a68bc7255357544810ea67ea918a5ffd7fd61eeed41be4cf7b71
    sha256: 4dc0fdf59c7befff277b51ef8bec1eaac9b4b12db071b8914739ea2156532ee5
    patches:
      - patches/f2py.patch
  - path: ../src

INFO:conda_build.source:Found source in cache: telemac-mascaret-v8p4r1_b48a0ee5cb.tar.gz
Extracting download
Applying patch: /home/tomsail/work/python/feedstocks/telemac-mascaret-feedstock/recipe/f2py.patch
Applying patch: /home/tomsail/work/python/feedstocks/telemac-mascaret-feedstock/recipe/f2py.patch with args:
['-Np1', '-i', '/tmp/tmp7sw0rlpr/f2py.patch.native', '--binary']
checking file scripts/python3/compilation/compil_tools.py
checking file scripts/python3/execution/get.py
checking file scripts/python3/telapy/api/api_module.py
checking file scripts/python3/telapy/api/hermes.py
checking file sources/api/api.cmdf
checking file sources/api/api_interface.f
checking file sources/api/example_cpl_t2d_sis.f
patch unexpectedly ends in middle of line
/home/tomsail/miniconda3/bin/patch: **** malformed patch at line 769:  

Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['/home/tomsail/miniconda3/bin/patch', '--no-backup-if-mismatch', '--batch', '-Np1', '-i', '/tmp/tmp7sw0rlpr/f2py.patch.native', '--binary', '--dry-run']' returned non-zero exit status 2.
Failed to download or patch source. Please see build log for info.```

nicogodet · 2023-11-19T17:49:53Z

Check you have an empty line at end of patch file

tomsail · 2023-11-19T18:04:33Z

It worked! I'll matrix build overnight and update the table tomorrow.
I'll also test all versions with validate_telemac.py or some basic tests so to see how the compilation and API performed.

This will give us insight for the constraints to put on numpy on conda-forge.
Bravo @nicogodet, great stuff

tomsail · 2023-11-20T12:01:52Z

@sebourban @nicogodet
please have a look at the recap table above (first post)
basic API runs fine, even 1.22/1.23 version ! (see test run above)

full linux versions (MPI/MUMPS/API/AED/GOTM) are available on my anaconda channel for the python x numpy versions in the table above.

nicogodet · 2023-11-20T13:48:17Z

@nicogodet validate_telemac.py does not work because package does not contain the examples. I f you have a fix for this, I'm keen to hear.

You can add a test folder inside you recipe dir with examples and refere to this folder during test phase
8f744a1#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9a

tomsail changed the title ~~Python / Numpy compilation and cross~~ Python / Numpy compilation and cross compatibility Nov 17, 2023

tomsail mentioned this issue Mar 19, 2024

latest linux build setup #16

Draft

tomsail mentioned this issue May 24, 2024

Add opentelemac conda-forge/staged-recipes#24529

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python / Numpy compilation and cross compatibility #13

Python / Numpy compilation and cross compatibility #13

tomsail commented Nov 17, 2023 •

edited

Loading

nicogodet commented Nov 17, 2023

tomsail commented Nov 17, 2023 •

edited

Loading

nicogodet commented Nov 18, 2023

nicogodet commented Nov 18, 2023

tomsail commented Nov 19, 2023 •

edited

Loading

nicogodet commented Nov 19, 2023

tomsail commented Nov 19, 2023

nicogodet commented Nov 19, 2023 •

edited

Loading

tomsail commented Nov 19, 2023

tomsail commented Nov 20, 2023

nicogodet commented Nov 20, 2023

Python / Numpy compilation and cross compatibility #13

Python / Numpy compilation and cross compatibility #13

Comments

tomsail commented Nov 17, 2023 • edited Loading

nicogodet commented Nov 17, 2023

tomsail commented Nov 17, 2023 • edited Loading

nicogodet commented Nov 18, 2023

nicogodet commented Nov 18, 2023

tomsail commented Nov 19, 2023 • edited Loading

nicogodet commented Nov 19, 2023

tomsail commented Nov 19, 2023

nicogodet commented Nov 19, 2023 • edited Loading

tomsail commented Nov 19, 2023

tomsail commented Nov 20, 2023

nicogodet commented Nov 20, 2023

tomsail commented Nov 17, 2023 •

edited

Loading

tomsail commented Nov 17, 2023 •

edited

Loading

tomsail commented Nov 19, 2023 •

edited

Loading

nicogodet commented Nov 19, 2023 •

edited

Loading