Skip to content

Commit

Permalink
Improve setup.py to make gcc/cython compilation optional
Browse files Browse the repository at this point in the history
  • Loading branch information
ronny-rentner committed Sep 20, 2024
1 parent 7d5a399 commit 5893ae8
Show file tree
Hide file tree
Showing 3 changed files with 40 additions and 14 deletions.
6 changes: 3 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,10 @@ tests = ["pytest", "ultraimport", "redis"]
[tool.pylint.'MESSAGES CONTROL']
max-line-length = 140

# Many false positives and wrong assumptoins with pylint unfortuantely and
# Many false positives and wrong assumptions with pylint unfortuantely and
# also really wrong things like "unidiomatic-typecheck" or "import-outside-toplevel"
disable = ["missing-module-docstring", "missing-class-docstring", "missing-function-docstring",
"unidiomatic-typecheck", "multiple-imports", "fixme", "no-else-return", "unused-argument",
disable = ["missing-module-docstring", "missing-class-docstring", "missing-function-docstring",
"unidiomatic-typecheck", "multiple-imports", "fixme", "no-else-return", "unused-argument",
"invalid-name", "import-outside-toplevel", "no-self-use", "method-hidden", "line-too-long",
"raise-missing-from", "redefined-builtin", "logging-too-many-args", "logging-fstring-interpolation",
"deprecated-method", "inconsistent-return-statements", "too-many-instance-attributes"]
Expand Down
8 changes: 4 additions & 4 deletions readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -265,15 +265,15 @@ dict.

## Memory management

`UltraDict` uses shared memory buffers and those usually live is RAM. `UltraDict` does not use any management processes to keep track of buffers. Also it cannot know when to free those shared memory buffers again because you might want the buffers to outlive the process that has created them.
`UltraDict` uses shared memory buffers and those buffers usually 'live' in RAM, but `UltraDict` does not use any management processes to keep track of the buffers. Furthermore, it cannot know when to free those shared memory buffers because you might want the buffers to outlive the process that has created them.

By convention you should set the parameter `auto_unlink` to True for exactly one of the processes that is using the `UltraDict`. The first process
that is creating a certain `UltraDict` will automatically get the flag `auto_unlink=True` unless you explicitly set it to `False`.
When this process with the `auto_unlink=True` flag ends, it will try to unlink (free) all shared memory buffers.

A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal `UltraDict` to keep
track of recursively nested `UltraDict` instances. All child `UltraDicts` will write to this register the names of the shared memory buffers
they are creating. This allows the buffers to outlive the processes and still being correctly cleanup up by at the end of the program.
A special case is the recursive mode using `recurse=True` parameter. This mode will use an additional internal register to keep
track of recursively nested `UltraDict` instances. All child `UltraDicts` will write to this internal register the names of the shared memory buffers
they are creating. This allows the buffers to outlive the processes and still being correctly cleaned up at the end of the program.

**Buffer sizes and read performance:**

Expand Down
40 changes: 33 additions & 7 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,41 @@
from pathlib import Path
from setuptools import setup, Extension
import Cython.Build
import os
import shutil

# read the contents of your README file
# Read the contents of your README file
this_directory = Path(__file__).parent
long_description = (this_directory / "readme.md").read_text()

version = '0.0.6'
version = '0.0.7'

ext = Extension(name="UltraDict", sources=["UltraDict.py"])
# Function to detect if a C compiler is available
def has_compiler():
# Check if gcc or clang is available
return shutil.which('gcc') or shutil.which('clang')

# Decide whether to build C extensions or skip based on the availability of a compiler
def get_extension_modules():
# If a C compiler is not available, skip the extension
if not has_compiler():
print("No C compiler detected. Skipping C extension building.")
return []

# C compiler available, attempt to use Cython or pre-generated C code
try:
from Cython.Build import cythonize
ext = Extension(name="UltraDict", sources=["UltraDict.py"])
return cythonize(ext, compiler_directives={'language_level': "3"})
except ImportError:
# Fall back to pre-generated C file
if os.path.exists("UltraDict.c"):
ext = Extension(name="UltraDict", sources=["UltraDict.c"])
return [ext]
print("No Cython or C source file available. Skipping C extension building.")
return []

# Get the extension modules (empty list if no compilation)
ext_modules = get_extension_modules()

setup(
name='UltraDict',
Expand All @@ -19,11 +46,10 @@
author='Ronny Rentner',
author_email='[email protected]',
url='https://github.com/ronny-rentner/UltraDict',
cmdclass={'build_ext': Cython.Build.build_ext},
package_dir={'UltraDict': '.', 'UltraDict.pymutex': 'pymutex'},
packages=['UltraDict', 'UltraDict.pymutex'],
zip_safe=False,
ext_modules=Cython.Build.cythonize(ext, compiler_directives={'language_level' : "3"}),
setup_requires=['cython>=0.24.1'],
ext_modules=ext_modules, # Only add extensions if a compiler is available
setup_requires=['cython>=0.24.1'] if ext_modules else [],
python_requires=">=3.8",
)

0 comments on commit 5893ae8

Please sign in to comment.