ocrd_all (2.45)- make all does not work

[stefanCCS]

Hi,

I get an error during make all for "making" ocrd-all.
Version: Made a "pull" at 13.01.2023 --> expecting to have version 2.45.
I have created a new venv with: `python3.7 -m venv ocrd-3.7

Error I have got (--> please clarify):

...

Requirement already satisfied: certifi>=2017.4.17 in /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages (from requests->ocrd==2.45.0) (2022.12.7)
Requirement already satisfied: zipp>=0.5 in /home/ocrdadmin/ocrd-3.7/lib/python3.7/site-packages (from importlib-metadata->ocrd_utils==2.45.0->ocrd==2.45.0) (3.11.0)
Collecting MarkupSafe>=2.0
  Using cached MarkupSafe-2.1.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Building wheels for collected packages: ocrd, sparklines, future
  Building wheel for ocrd (setup.py): started
  Building wheel for ocrd (setup.py): finished with status 'done'
  Created wheel for ocrd: filename=ocrd-2.45.0-py3-none-any.whl size=69890 sha256=ad2048fa420e45a8ee8ab0de7cad64208b585c4252f097ea08e8467c968ed2b5
  Stored in directory: /tmp/pip-ephem-wheel-cache-val5i0h4/wheels/9a/2f/72/c4c786024d7050ab443f6466193268537ec3fb052054ca10f7
  Building wheel for sparklines (setup.py): started
  Building wheel for sparklines (setup.py): finished with status 'done'
  Created wheel for sparklines: filename=sparklines-0.4.2-py3-none-any.whl size=224586 sha256=8bb3dd3072417661e23c61f5033260a55c35c6f0b7b6dc8209972923423ba9e1
  Stored in directory: /home/ocrdadmin/.cache/pip/wheels/5d/23/75/0415daad215b30bd5f49bc434a0f21f63804b70b98648a9997
  Building wheel for future (setup.py): started
  Building wheel for future (setup.py): finished with status 'done'
  Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492025 sha256=173fa491f777fe210f4681d9521210e95a3aa2cd0a04b416d080e16a86c9102a
  Stored in directory: /home/ocrdadmin/.cache/pip/wheels/52/2a/fc/520209cfa6448febd490720a0b09036cb367628f7c4e9cc172
Successfully built ocrd sparklines future
Installing collected packages: wrapt, python-magic, psutil, opencv-python-headless, MarkupSafe, itsdangerous, future, Werkzeug, sparklines, memory-profiler, Jinja2, Deprecated, Flask, ocrd
Successfully installed Deprecated-1.2.0 Flask-2.2.2 Jinja2-3.1.2 MarkupSafe-2.1.1 Werkzeug-2.2.2 future-0.18.3 itsdangerous-2.1.2 memory-profiler-0.61.0 ocrd-2.45.0 opencv-python-headless-4.7.0.68 psutil-5.9.4 python-magic-0.4.27 sparklines-0.4.2 wrapt-1.14.1
sem -q --will-cite --fg --id ocrd_all_pipocrd-3.7 pip install --no-binary shapely --force-reinstall shapely
Collecting shapely
  Downloading shapely-2.0.0.tar.gz (274 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 274.5/274.5 kB 4.8 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Collecting numpy>=1.14
  Using cached numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
Building wheels for collected packages: shapely
  Building wheel for shapely (pyproject.toml): started
  Building wheel for shapely (pyproject.toml): finished with status 'error'
Failed to build shapely
make[1]: Leaving directory '/home/ocrdadmin/ocrd_all/core'

[bertsky]

sem -q --will-cite --fg --id ocrd_all_pipocrd-3.7 pip install --no-binary shapely --force-reinstall shapely
Collecting shapely
Downloading shapely-2.0.0.tar.gz (274 kB)

oh my...

Thanks @stefanCCS for the report!

It seems we are leaving the frying pan straight for the fire. We needed --no-binary shapely due to this. But now shapely 2.0 is released, and it seems it comes with new dependencies which we cannot fulfill.

So should we remove the --no-binary shapely in hope of accommodating every user with the wheels published for Shapely? Or try to hold at 1.x for now? Or try to identify the extra system dependencies needed for compilation?

[kba]

Is there any way we can fix this upsteam? shapely/shapely#1598 did not lead to a proper workaround, did it?

new dependencies which we cannot fulfill

How so? Or rather what additional dependencies? How can I test (fail) this to debug?

So should we remove the --no-binary shapely in hope of accommodating every user with the wheels published for Shapely?

If that is working, then it would be preferable of course.

Or try to hold at 1.x for now?

Since I doubt any fix, should it arise, would be backported, that is likely a dead-end? We could pin it for now at 1.x to make ocrd_all work again in the short term.

Or try to identify the extra system dependencies needed for compilation?

As long as we need to build from source, yes.

[bertsky]

Is there any way we can fix this upsteam? shapely/shapely#1598 did not lead to a proper workaround, did it?

no, unfortunately not.

The only outcome was that (for whatever reason) we need to import torch prior to shapely. In every module of every processor. It's doable for now with a number of PRs, but still someone might fall into the same trap again in the future. Our short-termed workaround was simply to install from source (hence --no-binary). And we had hopes that this would automatically disappear with Shapely 2.x (but were hesitant to force the update due to other incompatibilities this might entail). In fact, the segfault does disappear there (just tested again).

new dependencies which we cannot fulfill

How so? Or rather what additional dependencies? How can I test (fail) this to debug?

I believe it's libgeos-dev again (even if the OP's output does not say it for some reason).

@stefanCCS can you confirm you did not do make deps-ubuntu (which would have installed that package)?

That would make it a non-issue for the moment.

[bertsky]

@kba since the prebuilt Shapely 2.x is only available for Python >= 3.7 and our default on ocrd_all is still 3.6, and I do think that source compilation should work with libgeos-dev, and 2.x seems to cooperate with our code, I recommend we keep everything as is for now and close here.

[stefanCCS]

@stefanCCS can you confirm you did not do make deps-ubuntu (which would have installed that package)?

That would make it a non-issue for the moment.

Short answer: Confirmed, I have NOT made a make deps-ubuntu.
In detail:
I made a deactive of old venv - moved this old venv to another place (to have some backup, even if I do not know, if this would work as backup a all...).
Afterwards just made a git pull in ~/ocrd_all , created a new venv and finally, made a make all.

[kba]

since the prebuilt Shapely 2.x is only available for Python >= 3.7

On that note: Can you have a look at #956 so we can proceed with the base image/python update soon-ish?

and 2.x seems to cooperate with our code, I recommend we keep everything as is for now and close here.

Agreed. Just so I understand correctly: updating to Shapely 2.0 might break existing code but we value non-segfault-behavior over possible incompatibility. Updating python to 3.7 will allow us to use shapely wheels again without the segfault with torch?

(to have some backup, even if I do not know, if this would work as backup a all...).

As long as you move/copy/symlink the backed up directory back to the original place (so absolute paths still work), it will work same as before.

[stefanCCS]

As long as you move/copy/symlink the backed up directory back to the original place (so absolute paths still work), it will work same as before.
--> nice - many thanks!

[bertsky]

@kba

Just so I understand correctly: updating to Shapely 2.0 might break existing code but we value non-segfault-behavior over possible incompatibility. Updating python to 3.7 will allow us to use shapely wheels again without the segfault with torch?

yes, precisely. And we need to move forward to 2.x with our code anyway (just have not found the time to test on a wide set of data with known outcome yet).

[kba]

OK, then let's keep the shapely --no-binary/compile-from-source solution until #956 is merged and drop it afterwards.

[bertsky]

OK, then let's keep the shapely --no-binary/compile-from-source solution until #956 is merged and drop it afterwards.

yes, either that or do everything in one step (moving to python 3.7 and ubuntu 20.4, dropping forced compilation for shapely, dropping gcc dependency)

[bertsky]

For the record: it did turn out that even with Shapely 2.x, we need source compilation to avoid the segfault with torch.

Since we have since kept --no-binary as part of ocrd_all machinery, this can be closed.