pyrevolve installation issue: fatal error: 'iostream' file not found [when creating anaconda environment]

[angelsaj]

We have a problem with pyrevolve when trying to install devito on a MacBook Pro (os: MacOS Mojave 10.14.3). The error first appears when we create the environment with anaconda. The error message is:

In file included from pyrevolve/crevolve.cpp:628:

  ./pyrevolve/../include/revolve_c.h:5:10: fatal error: 'iostream' file not found

  #include <iostream>

           ^~~~~~~~~~

  1 warning and 1 error generated.

  error: command 'gcc' failed with exit status 1

  

  ----------------------------------------

  Failed building wheel for pyrevolve

Please find attached a file with the complete output of :
conda env create -f environment.yml

We have installed the last version of anaconda3, Xcode, and we took the most recent files of devito from the git repository
devito_pyrevolve_error.pdf

[navjotk]

Please post the output of gcc --version and g++ --version

[angelsaj]

(base) host-131-114-235-97:~ MacMat$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

(base) host-131-114-235-97:~ MacMat$ g++ --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[angelsaj]

In case it can be helpful:

(base) host-131-114-235-97:~ MacMat$ printenv
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/_n/mh4xvbhs7sz8tz1dqh752lyw0000gn/T/
CONDA_SHLVL=1
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.A5Unpt67Pl/Render
CONDA_PROMPT_MODIFIER=(base)
TERM_PROGRAM_VERSION=421.1
TERM_SESSION_ID=1CED67F9-0BD5-459A-B14E-AD3D849339DD
USER=MacMat
CONDA_EXE=/anaconda3/bin/conda
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.uEL1Jyc457/Listeners
PATH=/anaconda3/bin:/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
CONDA_PREFIX=/anaconda3
PWD=/Users/MacMat
LANG=it_IT.UTF-8
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
SHLVL=1
HOME=/Users/MacMat
CONDA_PYTHON_EXE=/anaconda3/bin/python
LOGNAME=MacMat
CONDA_DEFAULT_ENV=base
_=/usr/bin/printenv

[navjotk]

I think the problem is because you're using clang, not gcc. Can you install gcc with conda install -c anaconda gcc and try again?

[angelsaj]

We installed gcc with conda as you suggested. Next, we reinstalled devito from the repository. But, unfortunately, we get exactly the same error!

I was guessing, since the error message says "iostream" file not found, but the iostream must be somewhere. Maybe are we missing some folders in the bash path?

Furthermore the warning message says:
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
do you think that it could be helpful to follow the warning suggestion? In case, where should we add the '-std=libc++' flag?

[navjotk]

Can you post the results of gcc --version and g++ --version again please (i.e. after installing gcc with conda). I would first like to be sure that you are really using gcc and not Apple's fake gcc which is actually clang. Since the error message hasn't changed, I'm inclined to believe that it is still using Apple's gcc/clang.

[angelsaj]

Please find the result below, it looks like bash is using the correct compiler

gcc --version:

(base) host-131-114-235-224:~ MacMat$ gcc --version
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

g++ --version:

(base) host-131-114-235-224:~ MacMat$ g++ --version
g++ (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[navjotk]

Have you had any progress with this @angelsaj ?

[angelsaj]

Nope... do you have suggestions?

[navjotk]

Can you please try the following:
conda create -n pyrevolve_test python=3.7
pip install pyrevolve

[angelsaj]

That is the error message. Pip cannot find the header file "limits.h", how is it possible? do we need to add the path somewhere or can it be that the file is missing??

building 'pyrevolve.crevolve' extension
  creating build/temp.macosx-10.7-x86_64-3.7
  creating build/temp.macosx-10.7-x86_64-3.7/pyrevolve
  creating build/temp.macosx-10.7-x86_64-3.7/src
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/include -arch x86_64 -I/anaconda3/include -arch x86_64 -I./pyrevolve -I. -Ipyrevolve -I/anaconda3/include/python3.7m -c pyrevolve/crevolve.cpp -o build/temp.macosx-10.7-x86_64-3.7/pyrevolve/crevolve.o
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  In file included from /anaconda3/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/syslimits.h:7:0,
                   from /anaconda3/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/limits.h:34,
                   from /anaconda3/include/python3.7m/Python.h:11,
                   from pyrevolve/crevolve.cpp:27:
  /anaconda3/lib/gcc/x86_64-apple-darwin11.4.2/4.8.5/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
   #include_next <limits.h>  /* recurse down to the real one */
                                                               ^
  compilation terminated.
  error: command 'gcc' failed with exit status 
  

  ----------------------------------------
  Failed building wheel for pyrevolve
  Running setup.py clean for pyrevolve
Failed to build pyrevolve
Installing collected packages: pyrevolve
  Running setup.py install for pyrevolve ... error

Command "/anaconda3/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/_n/mh4xvbhs7sz8tz1dqh752lyw0000gn/T/pip-install-lns7cwu1/pyrevolve/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/var/folders/_n/mh4xvbhs7sz8tz1dqh752lyw0000gn/T/pip-record-9fskyz71/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/_n/mh4xvbhs7sz8tz1dqh752lyw0000gn/T/pip-install-lns7cwu1/pyrevolve/

please find attached the complete outcome here:
pyrevolve_install_error.pdf

[navjotk]

Can you try:
xcode-select --install
See here for more info

[angelsaj]

xcode-select: error: command line tools are already installed, use "Software Update" to install updates

[navjotk]

Have you tried the other solutions on the page I linked above? Especially open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

[angelsaj]

we are going to try also this:

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

we will let you know the outcome

[angelsaj]

yeah I was reading the same, unfortunately we cannot test it right now, we will let you know as soon as we will have again access to the machine

[angelsaj]

We have news. We typed

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

and next we followed the guided procedure. This installed pyrevolve

Successfully installed pyrevolve-1.0.2

Next, we tried again:

conda env create -f environment.yml

but unfortunately we obtained the same error message. So the problem is still unsolved.
Nevertheless, now we have a pyrevolve_test environment available. Do we want to use it?

 conda info --envs
# conda environments:
#
base                  *  /anaconda3
pyrevolve_test           /anaconda3/envs/pyrevolve_test

[navjotk]

If you do pip install -e . in the devito directory, while having the pyrevolve environment loaded, you should be able to install and use devito fine.

[angelsaj]

We tried the following:

conda activate pyrevolve_test
pip install -e .

It run with no error messages. So, next, we tried to see if devito is working correctly using a python editor (spyder). We imported a devito function in a script:

from examples.seismic import plot_velocity

When we run it, we got the following error:

ModuleNotFoundError: No module named 'cpuinfo'

Here the complete error message
cpuinfo_error.pdf

So something is still not working.

Are we following the correct procedure? Does it mean that devito is not installed correctly?

We googled the error, here they suggest to try
pip install py-cpuinfo, nevertheless we should have already cpuinfo installed since it is in the environment.yml file, so I don't get it. Suggestions?

[navjotk]

Did you try pip install py-cpuinfo?
You were not able to follow the installation procedure that uses environment.yml because the default compiler on your system was not able to compile pyrevolve. We fixed that by installing a specific gcc version in a conda environment and installing pyrevolve separately to that environment. Once that was successful, we installed devito with pip install -e .. At no point during this, did we use environment.yml, so something being listed there does not affect your system because of the procedure you followed.

[angelsaj]

Your suggestion was right, installing cpuinfo with pip pip install py-cpuinfo successfully solved the error! But..

when importing the devito modules in spyder, we get a new error:

 File "/Users/MacMat/Applications/devito/devito/tools/utils.py", line 9, in <module>
    from cgen import dtype_to_ctype as cgen_dtype_to_ctype

ModuleNotFoundError: No module named 'cgen'

To solve this, we tried

pip install cgen
pip install -e .

from shell, in devito folder, having the environment pyrevolve_test active. But, unfortunately this does not solve the issue. Suggestions? Sorry for the large number of problems!

[navjotk]

Could you please confirm that the following commands succeeded with no errors?
pip install -e .
pip install cgen

If pip install -e . succeeds, you shouldn't have the need for pip install cgen etc.

Assuming pip install -e . in the devito folder succeeded, can you run python examples/seismic/acoustic/acoustic_example.py and see if the example runs fine?

[angelsaj]

We confirm that

pip install -e .
pip install cgen

succeded with no error. As you suggested, we run
python examples/seismic/acoustic/acoustic_example.py
from shell. We get the error ModuleNotFoundError: No module named 'crevolve'
We googled the error without success. It seems that no one had problems with a module named crevolve before. Is crevolve like a wrapper of C functions for pyrevolve (python)?

Here below the complete output:

(pyrevolve_test) host-131-114-235-20:devito MacMat$ python examples/seismic/acoustic/acoustic_example.py
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Traceback (most recent call last):
  File "/anaconda3/envs/pyrevolve_test/lib/python3.7/site-packages/pyrevolve/pyrevolve.py", line 2, in <module>
    import pyrevolve.crevolve as cr
ImportError: dlopen(/anaconda3/envs/pyrevolve_test/lib/python3.7/site-packages/pyrevolve/crevolve.cpython-37m-darwin.so, 2): Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
  Referenced from: /anaconda3/envs/pyrevolve_test/lib/python3.7/site-packages/pyrevolve/crevolve.cpython-37m-darwin.so
  Expected in: /usr/lib/libstdc++.6.dylib
 in /anaconda3/envs/pyrevolve_test/lib/python3.7/site-packages/pyrevolve/crevolve.cpython-37m-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "examples/seismic/acoustic/acoustic_example.py", line 6, in <module>
    from examples.seismic.acoustic import AcousticWaveSolver
  File "/Users/MacMat/Applications/devito/examples/seismic/acoustic/__init__.py", line 2, in <module>
    from .wavesolver import *  # noqa
  File "/Users/MacMat/Applications/devito/examples/seismic/acoustic/wavesolver.py", line 6, in <module>
    from examples.checkpointing.checkpoint import DevitoCheckpoint, CheckpointOperator
  File "/Users/MacMat/Applications/devito/examples/checkpointing/checkpoint.py", line 1, in <module>
    from pyrevolve import Checkpoint, Operator
  File "/anaconda3/envs/pyrevolve_test/lib/python3.7/site-packages/pyrevolve/__init__.py", line 3, in <module>
    from pyrevolve.pyrevolve import * # noqa
  File "/anaconda3/envs/pyrevolve_test/lib/python3.7/site-packages/pyrevolve/pyrevolve.py", line 4, in <module>
    import crevolve as cr
ModuleNotFoundError: No module named 'crevolve'

[navjotk]

If you just want the quickest way to run devito, may I suggest the docker installation method at: https://github.com/opesci/devito/blob/master/README.md#quickstart

This method should bypass the problems that are leading to the errors you see.

I recommend this as the quickest way to get started with Devito.

---

If you do want to explore the error above, here is some information I could gather:

Firstly: the real error above is actually Symbol not found: __ZNKSt5ctypeIcE13_M_widen_initEv
Googling for this leads me to this and this.

TLDR: This is happening because the environment has changed between the time you installed pyrevolve and the time that you ran it. So the shared libaries being loaded are different from the ones pyrevolve was linked against.

It's not strictly necessary but if you want, we can work together to find the source of this problem if you don't like the docker method for some reason.

[angelsaj]

Thank you for your response. You are right the effective error is Symbol not found we got confused.

We installed docker. We followed the installation procedure of devito with docker. We prompted:
docker-compose run devito /tests
and everything went fine, apart from the warning which we got at the end of the output message, which is as follows:

  /venv/lib/python3.6/site-packages/numpy/lib/type_check.py:546: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
    'a.item() instead', DeprecationWarning, stacklevel=1)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

====== 681 passed, 62 skipped, 2 xfailed, 337 warnings in 2100.14 seconds ======

Unfortunately this does not solve our problems. That is, we still got errors when we try to import devito functions. For instance, when we run

from examples.seismic import plot_velocity

from a python script (named prova2.py) using spyder, we got the error:

runfile('/Users/MacMat/Applications/devito/prova2.py', wdir='/Users/MacMat/Applications/devito')
Traceback (most recent call last):

  File "<ipython-input-8-cad081d1d859>", line 1, in <module>
    runfile('/Users/MacMat/Applications/devito/prova2.py', wdir='/Users/MacMat/Applications/devito')

  File "/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 704, in runfile
    execfile(filename, namespace)

  File "/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 108, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/Users/MacMat/Applications/devito/prova2.py", line 14, in <module>
    from examples.seismic import plot_velocity

  File "/Users/MacMat/Applications/devito/examples/seismic/__init__.py", line 1, in <module>
    from .model import *  # noqa

  File "/Users/MacMat/Applications/devito/examples/seismic/model.py", line 5, in <module>
    from examples.seismic.utils import scipy_smooth

  File "/Users/MacMat/Applications/devito/examples/seismic/utils.py", line 3, in <module>
    from devito import error

  File "/Users/MacMat/Applications/devito/devito/__init__.py", line 8, in <module>
    from devito.base import *  # noqa

  File "/Users/MacMat/Applications/devito/devito/base.py", line 5, in <module>
    from devito.backends import _BackendSelector

  File "/Users/MacMat/Applications/devito/devito/backends.py", line 11, in <module>
    from devito.logger import warning

  File "/Users/MacMat/Applications/devito/devito/logger.py", line 7, in <module>
    from devito.parameters import configuration

  File "/Users/MacMat/Applications/devito/devito/parameters.py", line 7, in <module>
    from devito.tools import Signer, filter_ordered

  File "/Users/MacMat/Applications/devito/devito/tools/__init__.py", line 1, in <module>
    from devito.tools.utils import *  # noqa

  File "/Users/MacMat/Applications/devito/devito/tools/utils.py", line 9, in <module>
    from cgen import dtype_to_ctype as cgen_dtype_to_ctype

ModuleNotFoundError: No module named 'cgen'

we tried also to run the code from shell. When we write:

python prova2.py

(which imports plot_velocity) we got the error:

(pyrevolve_test) host-131-114-235-18:devito MacMat$ pythonw prova2.py

Traceback (most recent call last):

  File "prova2.py", line 14, in <module>

    from examples.seismic import plot_velocity

  File "/Users/MacMat/Applications/devito/examples/seismic/__init__.py", line 1, in <module>

    from .model import *  # noqa

  File "/Users/MacMat/Applications/devito/examples/seismic/model.py", line 5, in <module>

    from examples.seismic.utils import scipy_smooth

  File "/Users/MacMat/Applications/devito/examples/seismic/utils.py", line 3, in <module>

    from devito import error

  File "/Users/MacMat/Applications/devito/devito/__init__.py", line 6, in <module>

    import cpuinfo

ImportError: No module named cpuinfo

We are sorry for the large number of errors, but, at this point, we have absolutely no clue on how to successfully install devito in our framework.

[navjotk]

The output from docker-compose run devito /tests above tells me that you managed to successfully install and test devito. You might want to familiarise yourself with docker - devito will only be available to code running inside the docker container containing devito. From devito's instructions:

# start a bash shell with devito
docker-compose run devito /bin/bash

Any code you run in this shell will see devito, conversely, a regular shell terminal won't see devito.

With the above, you should be able to run code that references devito in the shell you started.

Spyder isn't officially supported by devito so I doubt I or the rest of the devito team can help you with that. Now that you have devito running in a shell, maybe you can read the Spyder documentation to get it to play nicely with docker.

Most people getting started with Devito find it easy to start with jupyter notebook before moving on to executing scripts directly in the terminal. You can start a jupyter notebook server by doing the following (from the same instructions):

# start a jupyter notebook server on port 8888
docker-compose up devito

In any case, I don't think this is a pyrevolve installation issue any more. I believe you might find it beneficial to join Devito's Slack Channel and engage with the team (and other users) there.

[zhangwenwen]

I think the problem is because you're using clang, not gcc. Can you install gcc with conda install -c anaconda gcc and try again?

When I install mmcv in Mac OS, your suggestion is helpful, thanks.

[navjotk]

Closing as resolved