Skip to content

Commit

Permalink
1) update setup.py to be able to find python in OSX
Browse files Browse the repository at this point in the history 
2) add warning about colander version mismatch
  • Loading branch information
@suntzu86
suntzu86 committed Nov 5, 2014
1 parent afcad34 commit b22c608
Show file tree
Hide file tree
Showing 4 changed files with 116 additions and 57 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

* Switched from `testify` to `py.test` - http://pytest.org/ (#36)
* [cleanup] Moved bandits into their own sub directories (#375)
* Supply PYTHON_LIBRARY and PYTHON_INCLUDE_DIR vars to cmake automatically in ``setup.py`` (#412)
* Added warning when colander version is out of date (#413)

* Bugs

Expand Down
96 changes: 49 additions & 47 deletions docs/install.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,55 +73,10 @@ Requires:
$ pip install -r requirements.txt
$ python setup.py install

.. Note:: MOE's ``setup.py`` invokes cmake. Users can pass command line arguments to cmake via the ``MOE_CMAKE_OPTS`` environment variable. Other sections (e.g., `Python Tips`_, `CMake Tips`_) detail additional environment variables that may be needed to customize cmake's behavior.
.. Note:: MOE's ``setup.py`` invokes cmake. ``setup.py`` installs MOE with the python installation used to run it; so be sure to invoke ``setup.py`` with the Python that you want to use to run MOE. If this fails, then consult `Python Tips`_. Users can pass command line arguments to cmake via the ``MOE_CMAKE_OPTS`` environment variable. Other sections (e.g., `Python Tips`_, `CMake Tips`_) detail additional environment variables that may be needed to customize cmake's behavior.

.. Warning:: Boost, MOE, and the virtualenv must be built with the same python. (OS X users: we recommend using the MacPorts Python: ``/opt/local/bin/python``.)

Python Tips
^^^^^^^^^^^

Sometimes cmake will fail to find your Python installation or you will want to specify an alternate Python. To specify Python, add:

::

-D MOE_PYTHON_INCLUDE_DIR=/path/to/where/Python.h/is/found
-D MOE_PYTHON_LIBRARY=/path/to/python/shared/library/object

to the ``MOE_CMAKE_OPTS`` environment variable. For example, an OS X user might have:

::

export MOE_CMAKE_OPTS='-D CMAKE_FIND_ROOT_PATH=/opt/local -D MOE_PYTHON_INCLUDE_DIR=/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/ -D MOE_PYTHON_LIBRARY=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib'

In OS X, the python dynamic library will be a ``.dylib`` file; in Linux, it will be a ``.so`` file.

.. WARNING:: Mis-matched Python versions between your virtual environment, Boost, and/or MOE's installer can lead to a plethora of strange bugs. Anywhere from ``Fatal Python error: PyThreadState_Get: no current thread`` to segmentation faults and beyond. (You are hitting a binary incompatibility so it is hard to predict the specific error.) You may need to instruct your package manager to build Boost against a particular version of Python, indicate a different Python to MOE, etc. to make these versions line up.

Here are some ways to check/ensure that Python was found and linked correctly:

1. You can verify that cmake found the correct version by checking the values of ``PYTHON_INCLUDE_DIR`` and ``PYTHON_LIBRARY`` in ``moe/build/CMakeCache.txt``.
2. In `General MacPorts Tips`_, *notice* that Boost is built against ``python27``. Checking ``port installed "python*"``, you should see (amongst others) ``python27 @2.7.6_0 (active)``.
3. ``python --version`` will show you what version of Python is called by default.
4. Outside of a virtual environment, running ``which python`` (and tracking through the symlinks; the first level should be in ``/opt/local/...`` if you are using MacPorts in OS X) will show you specifically which Python is being used.
5. Inside of a virtual environment, ``yolk -l`` will show you what software versions are in use. The path to Python should match the Python used to install Boost and MOE. (Running ``which python`` still works here if you trace through the symlinks.) Get ``yolk`` via ``pip install yolk``.
6. Check binary shared library dependencies (only works if you are not linking statically). ``locate libboost_python`` and run ``ldd`` (Linux) or ``otool -L`` (OS X) on the dynamic library. (Note: ``ldd`` in Linux may not show the Python dependency since this linkage may be delayed till actual use.) Similarly, running those commands on ``moe/build/GPP.so`` should show you the same Python as above; for example:

::

LINUX:
$ ldd moe/build/GPP.so
yields lines like:
libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007f7d7a9fc000)

OS X:
$ otool -L moe/build/GPP.so
yields:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)

This should be the same Python that you see in the other steps.

If you linked statically, you need to check your link lines manually. Since MOE links dynamically by default, we assume that you know what you are doing if you changed it.

OSX Tips
--------

Expand Down Expand Up @@ -158,7 +113,7 @@ OS X 10.9 users beware: do not install boost with MacPorts. You *must* install i
The previous assumes that you want to use ``gcc 4.7`` and ``Python 2.7``; modify the ``install`` and ``set`` invocations if you want other versions.

7. Using ``port select --list``, check that the active versions of gcc, python, etc. are correct. In particular, OS X users want to see ``python27 (active)``, not ``python27-apple (active)``. See `port select information`_.
8. If you are having strange errors (no current thread, segfault, etc.), check `Python Tips`_.
8. Continue with the installation instructions. If you are having strange errors (no current thread, segfault, etc.), check `Python Tips`_.

General MacPorts Tips
^^^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -285,3 +240,50 @@ CMake Tips
1. Do you have dependencies installed in non-standard places? e.g., did you build your own boost? Set the env var: ``export MOE_CMAKE_OPTS=-DCMAKE_FIND_ROOT_PATH=/path/to/your/dependencies ...`` (OS X users with MacPorts should set ``/opt/local``.) This can be used to set any number of cmake arguments.
2. Have you checked `Connecting Boost to MOE`_ and `Python Tips`_?
3. Are you using the right compiler? e.g., for ``gcc``, run ``export MOE_CC_PATH=/path/to/your/gcc && export MOE_CXX_PATH=/path/to/your/g++`` (OS X users need to explicitly set this.)

Python Tips
-----------

.. Note:: This is an advanced-user section. ``setup.py`` should be able to identify the correct Python automatically (i.e., it tries to find the Python it was launched with). Examples of why you might need to keep reading: 1) ``setup.py`` failed to find the correct Python paths; 2) you building manually and not using ``setup.py``; 3) you are doing something "weird" like building MOE with a different version of Python than the one you intend to run MOE with.

Sometimes cmake and/or ``setup.py`` will fail to find your Python installation or you will want to specify an alternate Python. To specify Python, add:

::

-D MOE_PYTHON_INCLUDE_DIR=/path/to/where/Python.h/is/found
-D MOE_PYTHON_LIBRARY=/path/to/python/shared/library/object

to the ``MOE_CMAKE_OPTS`` environment variable. Note that options added to this environment variable *supersede* options set by ``setup.py``; so if ``setup.py`` failed, manually specifying the right paths will solve the problem. For example, an OS X user might have:

::

export MOE_CMAKE_OPTS='-D CMAKE_FIND_ROOT_PATH=/opt/local -D MOE_PYTHON_INCLUDE_DIR=/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/ -D MOE_PYTHON_LIBRARY=/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib'

In OS X, the python dynamic library will be a ``.dylib`` file; in Linux, it will be a ``.so`` file.

.. WARNING:: Mis-matched Python versions between your virtual environment, Boost, and/or MOE's installer can lead to a plethora of strange bugs. Anywhere from ``Fatal Python error: PyThreadState_Get: no current thread`` to segmentation faults and beyond. (You are hitting a binary incompatibility so it is hard to predict the specific error.) You may need to instruct your package manager to build Boost against a particular version of Python, indicate a different Python to MOE, etc. to make these versions line up.

Here are some ways to check/ensure that Python was found and linked correctly:

1. You can verify that cmake found the correct version by checking the values of ``PYTHON_INCLUDE_DIR`` and ``PYTHON_LIBRARY`` in ``moe/build/CMakeCache.txt``.
2. In `General MacPorts Tips`_, *notice* that Boost is built against ``python27``. Checking ``port installed "python*"``, you should see (amongst others) ``python27 @2.7.6_0 (active)``.
3. ``python --version`` will show you what version of Python is called by default.
4. Outside of a virtual environment, running ``which python`` (and tracking through the symlinks; the first level should be in ``/opt/local/...`` if you are using MacPorts in OS X) will show you specifically which Python is being used.
5. Inside of a virtual environment, ``yolk -l`` will show you what software versions are in use. The path to Python should match the Python used to install Boost and MOE. (Running ``which python`` still works here if you trace through the symlinks.) Get ``yolk`` via ``pip install yolk``.
6. Check binary shared library dependencies (only works if you are not linking statically). ``locate libboost_python`` and run ``ldd`` (Linux) or ``otool -L`` (OS X) on the dynamic library. (Note: ``ldd`` in Linux may not show the Python dependency since this linkage may be delayed till actual use.) Similarly, running those commands on ``moe/build/GPP.so`` should show you the same Python as above; for example:

::

LINUX:
$ ldd moe/build/GPP.so
yields lines like:
libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007f7d7a9fc000)

OS X:
$ otool -L moe/build/GPP.so
yields:
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)

This should be the same Python that you see in the other steps.

If you linked statically, you need to check your link lines manually. Since MOE links dynamically by default, we assume that you know what you are doing if you changed it.
12 changes: 12 additions & 0 deletions moe/views/schemas/base_schemas.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,24 @@
Thus changing missing/default fields in the output dict can modify the schema!
"""
import inspect
import warnings

import colander

from moe.optimal_learning.python.constant import GRADIENT_DESCENT_OPTIMIZER, L_BFGS_B_OPTIMIZER, TENSOR_PRODUCT_DOMAIN_TYPE, SQUARE_EXPONENTIAL_COVARIANCE_TYPE, NULL_OPTIMIZER, NEWTON_OPTIMIZER, DOMAIN_TYPES, OPTIMIZER_TYPES, COVARIANCE_TYPES
from moe.optimal_learning.python.python_version.expected_improvement import DEFAULT_MVNDST_PARAMS


# Several users have forgotten to run ``pip install -r requirements.txt`` and end up with
# and old colander version. The error message looks like a syntax error (vs version mismatch),
# so let's be more explicit.
if len(inspect.getargspec(colander.SchemaNode.__init__).args) > 1:
warnings.warn("It looks like your version of colander is out of date.\n"
"You will probably see the error quoted here: https://github.com/Yelp/MOE/issues/413\n"
"Did you run 'pip install -r requirements.txt'?.")


class StrictMappingSchema(colander.MappingSchema):

"""A ``colander.MappingSchema`` that raises exceptions when asked to serialize/deserialize unknown keys.
Expand Down
63 changes: 53 additions & 10 deletions setup.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
import shlex
import shutil
import subprocess
import sys
import warnings
from collections import namedtuple

from moe import __version__
Expand Down Expand Up @@ -101,11 +103,11 @@ def run(self):
build_dir = os.path.join(package_dir, 'build')

cmake_path = find_path(
MoeExecutable(
env_var='MOE_CMAKE_PATH',
exe_name='cmake',
)
MoeExecutable(
env_var='MOE_CMAKE_PATH',
exe_name='cmake',
)
)

cmake_options = env.get('MOE_CMAKE_OPTS', '')
if cmake_options == '':
Expand Down Expand Up @@ -140,22 +142,63 @@ def run(self):
# must be passed to subprocess.Popen in separate list elements.
cmake_options_split = shlex.split(cmake_options)

# Get info on Python paths (for the currently running Python) that we need to discover
# header/library locations for MOE
includepy, libdir, instsoname = sysconfig.get_config_vars('INCLUDEPY', 'LIBDIR', 'INSTSONAME')

# The meaning of 'LIBDIR' and 'INSTSONAME' is platform-specific. Handle the various cases.
# Initial value: 'LIBDIR' usually contains what we want, but one OSX case requires a different path.
moe_python_library_base = libdir
if not sys.platform.startswith('darwin'):
# For other platforms (only Linux is tested!) libdir + instsoname yields the shared object location.
if not sys.platform.startswith('linux'):
warnings.warn("Automatic path discovery untested outside of OSX and Linux. Taking our best guess.\n"
"Please check PYTHON_INCLUDE_DIR and PYTHON_LIBRARY paths below.")
else:
# on OSX versions that we tested against, LIBDIR looks like:
# /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib
# and INSTSONAME looks like:
# Python.framework/Versions/2.7/Python
# What we want is:
# /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python

# This is easy in Linux b/c INSTSONAME is the name of a dynamic library file
# in foo/bar/lib whereas in OSX it's a partial path.
# Instead, remove the overlapping part of both paths to construct the result.
instsoname_components = instsoname.rsplit('/', 1)
framework_subpath = instsoname_components[0] # Probably is similar to 'Python.framework/Versions/2.7'
libdir_base, libdir_framework_subpath, libdir_subpath = libdir.rpartition(framework_subpath)

if not libdir_base and not libdir_framework_subpath:
# Did not find framework_subpath; the expected overlap isn't there so just join libdir + instsoname
warnings.warn("Unexpected OSX library paths.\n"
"Please check PYTHON_INCLUDE_DIR and PYTHON_LIBRARY paths below.")
else:
# Found framework_subpath overlap so skip the overlapped components
moe_python_library_base = libdir_base

moe_python_include_dir = includepy
moe_python_library = os.path.join(moe_python_library_base, instsoname)

# Print the Python paths we found so that the user can verify them if something goes wrong.
print 'PYTHON_INCLUDE_DIR (Expected full path to where Python.h is found): {0:s}'.format(moe_python_include_dir)
print 'PYTHON_LIBRARY (Expected path to Python shared object; e.g., libpython2.7.so or .dylib): {0:s}'.format(moe_python_library)

# Build the full cmake command using properly tokenized options
cmake_full_command = [
cmake_path,
'-DMOE_PYTHON_INCLUDE_DIR=' + includepy,
'-DMOE_PYTHON_LIBRARY=' + os.path.join(libdir, instsoname)]
'-DMOE_PYTHON_INCLUDE_DIR=' + moe_python_include_dir,
'-DMOE_PYTHON_LIBRARY=' + moe_python_library,
]
cmake_full_command.extend(cmake_options_split)
cmake_full_command.append(cpp_location)

# Run cmake
proc = subprocess.Popen(
cmake_full_command,
cwd=local_build_dir,
env=env,
)
cmake_full_command,
cwd=local_build_dir,
env=env,
)
proc.wait()

# Compile everything
Expand Down

0 comments on commit b22c608

Please sign in to comment.