-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[NO MRG] Test CI #675
[NO MRG] Test CI #675
Conversation
Am seeing build failures with wheels. In particular am seeing the failure below with CUDA 12 wheels on CI: ---------------------------------------------------
--- JIT compile log for /tmp/tmp74v6a1_0/6908e4b23643780851108ea49e8b868d7148d761.cubin.cu ---
Traceback (most recent call last):
---------------------------------------------------
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 250, in _jitify_prep
/tmp/tmp74v6a1_0/6908e4b23643780851108ea49e8b868d7148d761.cubin.cu(11): error: namespace "std" has no actual member "is_signed"
name, options, headers, include_names = jitify.jitify(source, options)
/tmp/tmp74v6a1_0/6908e4b23643780851108ea49e8b868d7148d761.cubin.cu(33): warning #177-D: variable "BLOCK_DIM_Z" was declared but never referenced
File "cupy/cuda/jitify.pyx", line 240, in cupy.cuda.jitify.jitify
File "cupy/cuda/jitify.pyx", line 264, in cupy.cuda.jitify.jitify
Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"
1 error detected in the compilation of "/tmp/tmp74v6a1_0/6908e4b23643780851108ea49e8b868d7148d761.cubin.cu".
RuntimeError: Runtime compilation failed
---------------------------------------------------
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/__w/cucim/cucim/./ci/wheel_smoke_test.py", line 14, in <module>
img = cucim.skimage.data.binary_blobs(length=512, n_dim=2)
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_shared/utils.py", line 293, in fixed_func
return func(*args, **kwargs)
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/data/_binary_blobs.py", line 67, in binary_blobs
mask = gaussian(mask, sigma=0.25 * length * blob_size_fraction)
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_shared/filters.py", line 174, in gaussian
ndi.gaussian_filter(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_vendored/_ndimage_filters.py", line 527, in gaussian_filter
return _run_1d_correlates(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_vendored/_ndimage_filters.py", line 309, in _run_1d_correlates
return _filters_core._run_1d_filters(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_vendored/_ndimage_filters_core.py", line 120, in _run_1d_filters
fltr(input, arg, axis, temp, mode, cval, origin, **filter_kwargs)
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_vendored/_ndimage_filters.py", line 135, in correlate1d
return _correlate_or_convolve1d(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/_vendored/_ndimage_filters.py", line 249, in _correlate_or_convolve1d
out = _shmem_convolve1d(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/filters/_separable_filtering.py", line 1044, in _shmem_convolve1d
conv_axis_kernel, block, patch_per_block = _get_separable_conv_kernel(
File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cucim/skimage/filters/_separable_filtering.py", line 922, in _get_separable_conv_kernel
return m.get_function(func_name), block, patch_per_block
File "cupy/_core/raw.pyx", line 487, in cupy._core.raw.RawModule.get_function
File "cupy/_core/raw.pyx", line 100, in cupy._core.raw.RawKernel.kernel.__get__
File "cupy/_core/raw.pyx", line 117, in cupy._core.raw.RawKernel._kernel
File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
File "cupy/_core/raw.pyx", line 538, in cupy._core.raw._get_raw_module
File "cupy/_core/core.pyx", line 2236, in cupy._core.core.compile_with_cache
File "cupy/_core/core.pyx", line 2254, in cupy._core.core.compile_with_cache
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 484, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 562, in _compile_with_cache_cuda
ptx, mapping = compile_using_nvrtc(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 319, in compile_using_nvrtc
return _compile(source, options, cu_path,
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 284, in _compile
options, headers, include_names = _jitify_prep(
File "/pyenv/versions/3.9.18/lib/python3.9/site-packages/cupy/cuda/compiler.py", line 257, in _jitify_prep
raise JitifyException(str(cex)) from e
cupy.cuda.compiler.JitifyException: Runtime compilation failed |
This issue wasn't present in PR ( #672 ), which was opened a week ago (though it is building for CUDA 12.2). So presumably the issue is due to something that changed in the last week (either in this repo or in its dependencies) |
It is most likely related to the release of CuPY 13.0 (I see cupy-cuda12x-13.0.0 listed in the log for the failing case). |
Reopening to test again |
Seeing the following error with just the Conda packages on CI: /tmp/tmp61vjho6p/74f00af9c74cf1e0abee33a25043645b6c0323ad.cubin.cu(11): error: namespace "std" has no actual member "is_signed"
/tmp/tmp61vjho6p/74f00af9c74cf1e0abee33a25043645b6c0323ad.cubin.cu(33): warning #177-D: variable "BLOCK_DIM_Z" was declared but never referenced
Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"
1 error detected in the compilation of "/tmp/tmp61vjho6p/74f00af9c74cf1e0abee33a25043645b6c0323ad.cubin.cu". This is odd as PR ( #676 ) fixed a similar issue. However it seems to be back |
Try to get more info on CuPy compilation failures.
Closing as this should be resolved with PR ( #680 ) |
No description provided.