Ginkgo slower than go test for same suite (resurrect #644)

[stroiman]

This is a resurrection of #644 from 4 years ago, which was closed, as far as I could tell, due to lack of input on reproducability? An automated message suggested a new issue, rather than commenting on the old.

I got the same issue in my small project (so far it's small)

My small hobby project was to build a headless browser. But when I added the v8 engine to my project, I noticed a significant delay added to my TDD loop.

Then comparing with go test, I saw an approximately x2 execution time increase.

> time go test
> # ...
> Ran 3 of 3 Specs in 0.009 seconds
> SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped
> go test  1.05s user 1.52s system 192% cpu 1.339 total

> time ginkgo
> # ...
> ginkgo  2.16s user 1.65s system 147% cpu 2.581 total

So the primary diff is about ~1 sec of user time, and as visible, it's not test execution time, which is 9millisconds.

This is measured on MacOS, on a MacBook M2 Max

Setting the -compilers or --compilers to any value doesn't have any significant effect.

My project is here: https://github.com/stroiman/go-dom

The main branch doesn't include v8 - it's in the branch embed-v8-engine. Main branch is still slower (about x2), but it wasn't until I added v8 that I felt the delay was becoming annoying.

But I noticed something else

Features were missing from v8go, so I use an experimental branch of my own fork of the v8go project; and has a result, I have a replace directive in the go.mod file to point to my version. While working on those features on v8go, it would point to a local folder:

replace github.com/tommie/v8go v0.22.0 => /Users ... /go/src/github/stroiman/v8go

So, in short, my directory structure is

$GOPATH/
  src/
    github/
      stroiman/
        v8go/ # separate github repo with go.mod file
        go-dom/ # separate github repo with go.mod file pointing to sibling folder
          lexer/
          parser/

When I would run ginkgo watch -r from go/src/github/stroiman/go-dom it would actually detect when I made modifications to files in v8go. So not only files in a subfolder, but subfolders of a sibling

While this was really helpful while working on the v8go code, it doesn't seem like it should do that? That -r should only look in subfolders.

So this could indicate that ginkgo is looking at too many directories?

[onsi]

hey thanks for this and for including a reproducer. That’s fantastic! i’ll dig in and take a look - I’m perpetually annoyed at the performance difference between the ginkgo cli and go test. the only difference is that the ginkgo cli (for reasons that still make sense) first runs go test -c to compile the binary and then invokes it.

but you may be bumping into something to do with how ginkgo walks the directory structure to find test suites. i’ll dig in and let you know, but it might be a bit before I get back to you.

[stroiman]

Hey, I made another observation; which does seem to support the idea that there's something about walking the directory structure/dependency analyses.

I restructured my code, so right now the root has no code, but there are two folder, browser/, which is the DOM implementation and parser, and scripting/ which hosts the v8 engine. scripting has a dependency to browser.

This was mostly done, as the v8 added some delay, so I wanted to just separate the code; so I could dive into one part of the system and run test tests with no unnecessary dependencies.

But my observation is that when I run ginkgo watch -r from the root folder, the whole process hangs

Identified 2 test suites.  Locating dependencies to a depth of 1 (this may take a while)...
Watching 2 suites:
  ./browser [0 dependencies]
  ./scripting [0 dependencies]

When going into one of the subfolders, there is some delay to the initial startup, but it doesn't hang.

commit where this behavior is observed: stroiman/go-dom@5368cf1

[myzhan]

Guys, I'm aslo annoyed at the performance difference. And I notice that the Apple's XProtect will slow down the launch speed of each test executables. Ginkgo produces more test executables than go test. So I run the same testcases on Ubuntu, the performance difference is gone.