Possible bug in symbol version lookup? "GLIBC_PRIVATE"

[dralley]

Filing this after a discussion with @njsmith on the wheel-builders mailing list.

I'm trying to build a manylinux2014-based Python wheel for the "createrepo_c" library. It compiles properly in the docker container (and every other platform I've tried) but "auditwheel repair" always fails.

Here is the output from auditwheel repair and auditwheel show:

+ auditwheel show wheelhouse/createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl

createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libz.so.1 with versions {'ZLIB_1.2.3.5',
'ZLIB_1.2.3.3', 'ZLIB_1.2.2.3'}, liblzma.so.5 with versions
{'XZ_5.1.2alpha', 'XZ_5.0'}, libxml2.so.2 with versions
{'LIBXML2_2.4.30'}, libcrypto.so.10 with versions {'libcrypto.so.10',
'OPENSSL_1.0.1_EC'}, libc.so.6 with versions {'GLIBC_2.4',
'GLIBC_2.15', 'GLIBC_2.3.3', 'GLIBC_2.8', 'GLIBC_PRIVATE',
'GLIBC_2.7', 'GLIBC_2.14', 'GLIBC_2.3', 'GLIBC_2.2.5', 'GLIBC_2.16',
'GLIBC_2.3.2', 'GLIBC_2.17', 'GLIBC_2.3.4', 'GLIBC_2.11',
'GLIBC_2.12'}, libpthread.so.0 with versions {'GLIBC_2.3.3',
'GLIBC_2.2.5', 'GLIBC_2.3.2'}, libdl.so.2 with versions
{'GLIBC_2.2.5'}, libgssapi_krb5.so.2 with versions
{'gssapi_krb5_2_MIT'}, libidn.so.11 with versions {'LIBIDN_1.0'},
libnss3.so with versions {'NSS_3.5', 'NSS_3.21', 'NSS_3.11.2',
'NSS_3.6', 'NSS_3.13', 'NSS_3.34', 'NSS_3.22', 'NSS_3.11.1',
'NSS_3.10', 'NSS_3.12', 'NSS_3.12.5', 'NSS_3.12.6', 'NSS_3.3.1',
'NSS_3.14.3', 'NSS_3.2', 'NSS_3.14', 'NSS_3.11', 'NSS_3.8',
'NSS_3.9.3', 'NSS_3.7', 'NSS_3.4', 'NSS_3.15', 'NSS_3.9',
'NSS_3.12.1', 'NSS_3.19.1', 'NSS_3.3', 'NSS_3.9.2'}, libssl3.so with
versions {'NSS_3.2', 'NSS_3.11.4', 'NSS_3.4', 'NSS_3.14'},
libk5crypto.so.3 with versions {'k5crypto_3_MIT'}, libkrb5.so.3 with
versions {'krb5_3_MIT'}, librt.so.1 with versions {'GLIBC_2.2.5'},
libnssutil3.so with versions {'NSSUTIL_3.31', 'NSSUTIL_3.21',
'NSSUTIL_3.15', 'NSSUTIL_3.13', 'NSSUTIL_3.14', 'NSSUTIL_3.12.3',
'NSSUTIL_3.17.1', 'NSSUTIL_3.24', 'NSSUTIL_3.38', 'NSSUTIL_3.12',
'NSSUTIL_3.12.5', 'NSSUTIL_3.39'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libfreebl3.so with versions {'NSSRAWHASH_3.12.3'},
libresolv.so.2 with versions {'GLIBC_2.2.5'}, libkrb5support.so.0 with
versions {'krb5support_0_MIT'}, libkeyutils.so.1 with versions
{'KEYUTILS_0.3', 'KEYUTILS_1.0', 'KEYUTILS_1.5'}, libacl.so.1 with
versions {'ACL_1.0'}, libpopt.so.0 with versions {'LIBPOPT_0'},
libgcc_s.so.1 with versions {'GCC_3.3.1', 'GCC_3.0'}, libssl.so.10
with versions {'libssl.so.10'}, libattr.so.1 with versions
{'ATTR_1.0'}, libelf.so.1 with versions {'ELFUTILS_1.0'}

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would need to recompile a new wheel
from source on a system with earlier versions of these libraries, such
as a recent manylinux image.
+ auditwheel repair wheelhouse/createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl --plat manylinux2014_x86_64 -w /io/wheelhouse/
INFO:auditwheel.main_repair:Repairing createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "wheelhouse/createrepo_c-0.15.7-cp35-cp35m-linux_x86_64.whl" to "manylinux2014_x86_64" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.
(.env) [vagrant@pulp3-source-fedora31 createrepo_c]$

I went through all of those symbols and compared them against the symbol version whitelist here

All of the versions of GCC and Glib that are listed exist in the whitelist, except for one which is rather strange.

libc.so.6 with versions { ...., 'GLIBC_PRIVATE', ....},

Is this a bug? I don't know what GLIBC_PRIVATE is, but it doesn't match the general pattern, and I don't think this library is depending on any private glibc interfaces or anything like that.

[ehashman]

I think GLIBC_PRIVATE is definitely your issue. Somewhere, you have a dependency on functions that use symbols from the private GLIBC ABI. Can you use readelf on your wheel to try to figure out what those are and how you might have picked up the dep? It might be getting pulled in by one of your dependencies...

[dralley]

@ehashman edit: needed to extract the .whl to get the .so not just run readelf on it directly.

What arguments should I use, and what information should I be looking for? Case-insensitive search on the output of readelf -a doesn't find any usages of "private". Searching the dynamic symbol table for "glibc" shows this:

[dalley@localhost createrepo_c]$ readelf --dyn-syms _createrepo_c.so | rg -i glibc
     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __errno_location@GLIBC_2.2.5 (2)
    11: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strptime@GLIBC_2.2.5 (2)
    14: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strtoll@GLIBC_2.2.5 (2)
    15: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __ctype_tolower_loc@GLIBC_2.3 (3)
    16: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sprintf@GLIBC_2.2.5 (2)
    23: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strstr@GLIBC_2.2.5 (2)
    38: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strchr@GLIBC_2.2.5 (2)
    45: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (2)
    46: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strlen@GLIBC_2.2.5 (2)
    49: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND mkdir@GLIBC_2.2.5 (2)
    66: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND utime@GLIBC_2.2.5 (2)
    77: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND rename@GLIBC_2.2.5 (2)
    91: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND localtime@GLIBC_2.2.5 (2)
   100: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strftime@GLIBC_2.2.5 (2)
   102: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy@GLIBC_2.14 (8)
   104: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND perror@GLIBC_2.2.5 (2)
   117: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND time@GLIBC_2.2.5 (2)
   126: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fclose@GLIBC_2.2.5 (2)
   129: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __xstat@GLIBC_2.2.5 (2)
   134: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fileno@GLIBC_2.2.5 (2)
   142: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fseek@GLIBC_2.2.5 (2)
   154: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fopen@GLIBC_2.2.5 (2)
   155: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND free@GLIBC_2.2.5 (2)
   163: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND nftw@GLIBC_2.3.3 (9)
   170: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND malloc@GLIBC_2.2.5 (2)
   171: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND vfprintf@GLIBC_2.2.5 (2)
   172: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strtoull@GLIBC_2.2.5 (2)
   181: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strcmp@GLIBC_2.2.5 (2)
   184: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strtol@GLIBC_2.2.5 (2)
   196: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getpid@GLIBC_2.2.5 (2)
   200: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fread@GLIBC_2.2.5 (2)
   212: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND puts@GLIBC_2.2.5 (2)
   213: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND feof@GLIBC_2.2.5 (2)
   220: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND snprintf@GLIBC_2.2.5 (2)
   224: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND gettimeofday@GLIBC_2.2.5 (2)
   225: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND realloc@GLIBC_2.2.5 (2)
   256: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND remove@GLIBC_2.2.5 (2)
   264: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND mkdtemp@GLIBC_2.2.5 (2)
   269: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND ferror@GLIBC_2.2.5 (2)
   281: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND fwrite@GLIBC_2.2.5 (2)

[ehashman]

@dralley it would likely be helpful if you uploaded a copy of your built wheel as well as sharing the line-by-line steps you used to build it.

[dralley]

createrepo_c.zip

Wheels and build script in the zip file

CMD: docker run --rm -e PLAT=manylinux2014_x86_64 -v `pwd`:/io quay.io/pypa/manylinux2014_x86_64 /io/build_wheels.sh

[dralley]

I thought I had linked the full code already but apparently not: https://github.com/dralley/createrepo_c

[dralley]

The script itself is not in the source code because I want it to be done by CI eventually, just debugging the process at the moment.

I should mention that this came straight from the manylinux demo script https://github.com/pypa/python-manylinux-demo/tree/master/travis and I used the same script for 2 other packages successfully.

[lkollar]

The issue is definitely GLIBC_PRIVATE:

DEBUG:auditwheel.policy.versioned_symbols:Package requires GLIBC_PRIVATE, incompatible with policy manylinux2014_x86_64 which requires {'GLIBC_2.14', 'GLIBC_2.15', 'GLIBC_2.8', 'GLIBC_2.17', 'GLIBC_2.3.2', 'GLIBC_2.3.4', 'GLIBC_2.2', 'GLIBC_2.1.1', 'GLIBC_2.0', 'GLIBC_2.1.2', 'GLIBC_2.3.3', 'GLIBC_2.12', 'GLIBC_2.5', 'GLIBC_2.6', 'GLIBC_2.7', 'GLIBC_2.1.3', 'GLIBC_2.2.4', 'GLIBC_2.11', 'GLIBC_2.4', 'GLIBC_2.13', 'GLIBC_2.16', 'GLIBC_2.2.1', 'GLIBC_2.10', 'GLIBC_2.9', 'GLIBC_2.2.5', 'GLIBC_2.2.3', 'GLIBC_2.3', 'GLIBC_2.1', 'GLIBC_2.2.2', 'GLIBC_2.2.6'}

Scanning through the runtime dependencies of _createrepo_c.so I found a dependency on libcrypt.so.1 which does need the GLIBC_PRIVATE version. This library was removed from the white list and the manylinux images have been upgraded to libcrypt.so.2 a while back. See #182.

[lkollar]

Looks like project's build picks up the wrong libcrypt.so.1. As an experiment I created a symlink from /usr/local/lib/libcrypt.so.2.0.0 to /lib64/libcrypt.so.1 and after compiling the extension I was able to run auditwheel repair on it.

Not sure why it is building with the wrong library though.

[dralley]

@lkollar Thank you for the help! That is strange indeed.

[dralley]

@lkollar I think this issue is a duplicate of pypa/manylinux#411 (comment)? Or they are at least possibly talking about the same issue.

They claim they avoided the issue by avoiding libcurl, which we can't do.

[dralley]

Apologize for all the edits and comments I've made and then deleted. It's my first time dealing with any of these tools or even messing around with the concepts of symbols and linking (I know of them, but I'm a Python dev, not a C dev). Such is the process of learning 😄

I'm able to confirm your results -- I'll test the packages tomorrow and verify that everything works as expected.

Strangely, there's no reference to libcrypt.so.2 in /lib64/ at all, it's just in /usr/lib64/. And making a link under the name of libcrypt.so.2 (instead of .1) doesn't work, it has to be .1

[njsmith]

So the issue is definitely that the build is picking up the wrong version of libcrypt. The .so.1 version is the old version that's part of libc (so we can't vendor it), but has been deprecated and isn't available on all systems anymore (so we can't assume the system will provide it, either). So there's just no way for a manylinux wheel to link to the .so.1 version.

The .so.2 version is the new standalone package, which acts like a normal shared library. So that's the version that you need to use.

There are two ways you might be picking up the wrong version:

• Option 1: you're building some C code, and the build system (autoconf or whatever) somehow finds the wrong version. This isn't supposed to happen; the manylinux images are supposed to be set up in a way that prevents that. But it's possible that there's some bug somewhere and it's not working right.

• Option 2: you're using pre-built libraries provided by the distribution, and those have been built to use the distro-provided libcrypt.so.1.

I think the second option is more likely. For example, maybe when you dnf install libcurl, you get a library that uses libcrypt.so.1. If that's what's going on, then what it means is that you can't use the distro-provided libcurl; instead you need to download the libcurl source and build it yourself, making sure that it uses libcrypt.so.2.

(Libcurl is just an example here. Replace with "any libraries that use libcrypt.so.1.)

[myselfhimself]

@lkollar I think this issue is a duplicate of pypa/manylinux#411 (comment)? Or they are at least possibly talking about the same issue.

They claim they avoided the issue by avoiding libcurl, which we can't do.

Hello... I filed that very #411 issue, and in the end in our case we fall back on using the platform's curl executable when libcurl is not linked... I am following the present issue, in case some solution finally were landed, but have no more clue on a workaround sorry.

[dralley]

I believe @njsmith is right, and it is option 2. libcurl links against libcrypt.so.1 inside the environment. That is why I get different results for ldd on my Fedora host (libcrypt.so.2), and why readelf doesn't mention libcrypt at all -- because it's not a direct dependency.

This is an environmental problem so I will close this issue in favor of @myselfhimself 's issue.
pypa/manylinux#411 (comment)