Sorry, ReBench runs everything in an empty environment...

So, this means you'll need to configure the PATH variable in your configuration.

The one important immediate improvement would be to fix the docs to point to the semantics of environment variables: you'll need to manage them yourself. All in an effort to have configurations are reproducible as possible. But perhaps we went a little far with it...

See for previous discussions and details:

• Making environment handling more user-friendly #221
• Environment variable handling #174

I see.

Yes, that is unexpected and it did catch me off guard a bit.

I can work around it by requiring the user to symlink their interpreter binaries into (e.g.) bin/ and adding path: bin into the config file.

If you don't want to pass $PATH through, one thing you could do to fix this is: before any experiments are run, resolve absolute paths for all the executors that don't have a path directive using the existing PATH (or crash if you can't) and then run the interpreters using absolute paths and an empty environment. (In addition, if you can resolve all executors to an absolute path, you could invoke the interpreters without a shell).

I dunno, just an idea.

How about something like this:

executors:
    Lua:
        executable: lua
        env:
           PATH: /opt/local/bin

Also note that we try to resolve ~ in the commands (not in env variables though) to avoid having configurations to be too specific to a machine: #240

IIRC, you can already acheive that effect with:

executors:
    Lua:
        executable: lua
        path: /opt/local/bin

I was looking for a way to do it without requiring hardcoding (potentially) machine-specific paths.

I'll leave the decision up to you, because I can certainly work around this.

If you don't want to pass $PATH through, one thing you could do to fix this is: before any experiments are run, resolve absolute paths for all the executors that don't have a path directive using the existing PATH (or crash if you can't) and then run the interpreters using absolute paths and an empty environment. (In addition, if you can resolve all executors to an absolute path, you could invoke the interpreters without a shell).

While this might be more user-friendly, I don't think this is in the spirit of making configurations self-contained.
This then is still depending on the undocumented environment, and one may pick up different binaries based without this being visible in the config.

Right, so actually what you have now is probably sufficient, just the documentation needs to be updated.

Path to the executable. If not given, it's up to the shell to find the executable.

^ The second sentence is misleading

IIRC, you can already achieve that effect with

Yeah, the trivial example doesn't really illustrate why that's useful.
Though, when the executable itself relies on the PATH being set for something, then this is more important.

For your specific case, why not this?

executors:
    Lua:
        executable: lua
        path: yklua/src
    YkLua:
        executable: yklua
        path: yklua/src

This way, you don't rely on PATH at all.
And that's really the style we are using, too.
Assuming you don't need it in anything these things execute.

Right, so actually what you have now is probably sufficient, just the documentation needs to be updated.

Path to the executable. If not given, it's up to the shell to find the executable.

^ The second sentence is misleading

Yes, that's what I meant earlier with "The one important immediate improvement would be to fix the docs "

I don't think this is in the spirit of making configurations self-contained.

One other thing that might be a little misleading is that although rebench doesn't pass down $PATH from the parent environment, it will still search the system's default paths.

For example, on my system this does work:

executors:
    Lua:
        executable: lua53

because lua53 is in /usr/local/bin, which is in the system's default $PATH.

ReBench doesn't do anything magic, I think. So, I'd assume the system is doing this for you.

So, I am trying this very basic config here:

default_experiment: benchmarks
default_data_file: 'rebench.data'

benchmark_suites:
    test-suite:
        gauge_adapter: Time
        command: " "
        benchmarks:
            - TestBench

executors:
    exec1:
        executable: env

experiments:
    benchmarks:
        description: All benchmarks
        suites:
          - test-suite
        executions:
          - exec1

When I run it like this: rebench -d rebench-test.conf

I get:

Executing run:TestBench (exec1, test-suite) 1 None None None
    cmd: /opt/local/bin/gtime -f "max rss (kb): %M
wall-time (secounds): %e
" env
    cwd: /Users/smarr/tmp
    Starting run
PWD=/usr/bin
SHLVL=1
_=/opt/local/bin/gtime
max rss (kb): 1040
wall-time (secounds): 0.00

So, we see the environment is entirely empty, and the shell is asked to run /opt/local/bin/gtime -f "..." env" in the path /Users/smarr/tmp.

Interesting. On my BSD box I get:

Executing run:TestBench (exec1, test-suite) 1 None None None
    cmd: /usr/bin/time -p env
    cwd: /home/edd/research/yk-benchmarks
    Starting run
PWD=/home/edd/research/yk-benchmarks
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/local/bin:/usr/local/sbin

On my debian box I get:

Executing run:TestBench (exec1, test-suite) 1 None None None
    cmd: /usr/bin/time -f "max rss (kb): %M
wall-time (secounds): %e
" env
    cwd: /home/vext01/research/yk-benchmarks
    Starting run
PWD=/home/vext01/research/yk-benchmarks

So I guess different systems do different things.