Can't build on macOS

[beaumartinez]

Hello, I'm having issues trying to build on macOS.

I use the latest version of bash (not the outdated version macOS comes with, so it should support bash plugins)

$ bash --version
GNU bash, version 5.0.11(1)-release (x86_64-apple-darwin18.6.0)

Running ./configure gives me

configure: WARNING: elfutils is not available, struct support will not be available
[...]
checking whether bash symbols are exported... no
configure: error: in `~/code/ctypes.sh':
configure: error: unable to build a test extension
See `config.log' for more details

In config.log I see

configure:13864: checking whether bash symbols are exported
configure:13891: gcc -o conftest -g -O2  -I./include  -shared -fPIC conftest.c -lelf  >&5
Undefined symbols for architecture x86_64:
  "_num_shell_builtins", referenced from:
      _conftest_function in conftest-efa94e.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
configure:13891: $? = 1
configure: failed program was:
| /* confdefs.h */
[...]
|     #include "builtins.h"
| 
|     int conftest_function(WORD_LIST *list)
|     {
|         exit(num_shell_builtins
|                 ? EXIT_SUCCESS
|                 : EXIT_FAILURE);
|     }
| 
|     struct builtin conftest_struct = {
|         .name       = "conftest",
|         .function   = conftest_function,
|         .flags      = BUILTIN_ENABLED,
|         .long_doc   = NULL,
|         .short_doc  = NULL,
|         .handle     = NULL,
|     };
[...]
configure:13911: result: no
configure:13913: error: in `~/code/ctypes.sh':
configure:13915: error: unable to build a test extension
See `config.log' for more details

I don't know if this has something to do with include/builtins.h since that what defines the extern of num_shell_builtins. The only other thing I can think of is not having struct support since I don't have elfutils installed (I don't think it builds for macOS), and the code it tries to compile has a struct. I might be barking up the wrong tree though; I don't know how to resolve this myself.

Thank you for your time.

[cemeyer]

You must configure bash to support extensions; it's not the default. I believe the option to bash's ./configure is --enable-disabled-builtins, but I may be mistaken. The end result is that bash must be linked with ld --export-dynamic (or cc -Wl,--export-dynamic, or s/--export-dynamic/-E/).

[cemeyer]

That said, I also do not see the symbol we're looking for in a recent ish bash (5.0.11). Instead I see the similarly named num_shell_builtins. @beaumartinez , can you confirm:

nm -D $(which bash) | grep num_shell
0000000000319198 D num_shell_builtins

If the output is empty, you're missing the linker flag above; if the output matches, but does not have the leading underscore, we need to fix ctypes.sh configure. Thanks for the report!

[beaumartinez]

Thank you so much @cemeyer, that's very helpful. I have just rebuilt my bash from source with the --enable-disabled-builtins flag; before I didn't have any such num_shell_builtins symbol but now I do (with a leading underscore)

$ nm -D $(which bash) | grep num_shell
00000001000bde98 D _num_shell_builtins

[beaumartinez]

Since I'm on macOS, ld uses different flags and it expects -export_dynamic instead of --export-dynamic. I'm setting it in my LDFLAGS, but it still doesn't build

$ LDFLAGS=-Wl,-export_dynamic ./configure

config.log contains

configure:13891: gcc -o conftest -g -O2  -I./include -Wl,-export_dynamic -shared -fPIC conftest.c -lelf  >&5
Undefined symbols for architecture x86_64:
  "_num_shell_builtins", referenced from:
      _conftest_function in conftest-c185bc.o
ld: symbol(s) not found for architecture x86_64

[cemeyer]

Follow-up question, is ctypes.sh finding the correct bash (your 5.x) and not the ancient one OSX ships with?

[taviso]

I'm a bit confused, it doesn't even get to trying to load it, that error is just from trying to compile it...

(If it compiles successfully, then it tries to load it like bash -c 'enable -f xxx')

That seems like just standard C to me, what happens if you do this on the command-line?

extern int num_shell_builtins;
int test()
{
    return num_shell_builtins;
}

$ gcc -fPIC -shared -o test test.c

The flags look right to me in your output, -shared and -fPIC being the important ones... very confusing.

[taviso]

Here's the test that it's running:

https://github.com/taviso/ctypes.sh/blob/master/configure.ac#L105

[beaumartinez]

Would you know how to verify that @cemeyer? I've got the newer bash set in my PATH and /etc/shells; unless ./configure ignores those then I assume it would use the correct one.

I've managed to get a bit further; adding an additional linker flag -undefined dynamic_lookup seems to allow us to bypass the error for now. I'm now running

LDFLAGS=-Wl,-export_dynamic,-undefined,dynamic_lookup ./configure

However after running make and make install, when running source ctypes.sh I get

$ source ctypes.sh
-bash: enable: cannot open shared object /usr/local/lib/ctypes.dylib: dlopen(/usr/local/lib/ctypes.dylib, 0x0001): dlopen(): file not found: /usr/local/lib/ctypes.dylib
can't find the ctypes.dylib library, run make install?

There are however ctypes.la and ctypes.so in /usr/local/lib.

[beaumartinez]

@taviso I tried what you've suggested, thank you.

$ gcc -fPIC -shared -o test test.c
Undefined symbols for architecture x86_64:
  "_num_shell_builtins", referenced from:
      _test in test-b9c1fb.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[beaumartinez]

@taviso Trying different flags allows me to compile it successfully though.

$ gcc -fPIC -bundle -bundle_loader $(which bash) test.c -o test

[cemeyer]

@beaumartinez Does gcc -fPIC -shared -Wl,--allow-shlib-undefined -o test test.c also work? It’s possible the macos linker strictly requires shlib symbols to be resolved at link time by default.

[cemeyer]

Re ctypes.so in usr/local/lib, can you try setting LD_LIBRARY_PATH=“$LD_LIBRARY_PATH:/usr/local/lib” bash before sourcing ctypes.sh?

[cemeyer]

(MacOS is a weird ass unix and I’m totally unfamiliar with it, sorry. Just guessing in the dark here.)

[beaumartinez]

@cemeyer Unfortunately --allow-shlib-undefined doesn't work for me; I think for clang the equivalent flag is -undefined dynamic_lookup which I was setting in my LDFLAGS above.

I tried setting LD_LIBRARY_PATH and calling bash again before sourcing ctypes.sh but I get the same error as before.

I'm not that familiar with building C on macOS so we're both guessing in the dark :)

[beaumartinez]

@cemeyer Just to clarify this also compiles successfully.

$ gcc -fPIC -shared -Wl,-undefined,dynamic_lookup -o test test.c

[cemeyer]

Sorry, I’m out of ideas. 🥶