The "Results of running suite" tab only shows a tiny fraction of the actual tests executed. What am i doing wrong?

[RichMacDonald]

This is a tough question to ask: I've been using testing in eclipse for a very long time. I consider myself highly experienced. But ... the test results tab that shows the summary information (All tests, Failed tests, Summary) has always been untrustworthy. It only occasionally shows the correct results. More often than not it shows a handful of tests, while the console shows a completely different number.

To prove I am not crazy, here is a screenshot of the last test suite I ran. The console shows 666 tests run with 135 failures. The tab shows 31 tests passing and 0 failures. The console is correct.

I don't see anyone else complaining about this, so what could I possibly be doing wrong?

Rich MacDonald

[ahr-huber]

We ran into the same issue.
I can reproduce it with tests that use data providers. I could not reproduce it with 'normal' test methods.
When you execute the minimal example (code below) you can see that the UI is lagging behind the test execution. Once the test process terminates the UI stops updating.

Workaround:
Hide the TestNG view in Eclipse. This way the rendering is skipped and the Eclipse code is fast enough to gather all results.

Minimal example:

import org.testng.Assert;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class MinimalExample
{
  @DataProvider
  Object[][] provider()
  {
    final Object[][] result = new Object[1000][];
    for (int i = 0; i < result.length; i++)
    {
      result[i] = new Object[] { i };
    }
    return result;
  }

  @Test(dataProvider = "provider")
  public void test(final int i)
  {
    Assert.assertTrue(i >= 0);
  }
}

[missedone]

@RichMacDonald @ahr-huber, thanks for reporting the issue and trying to narrow down the root cause. let me check the code.

[missedone]

the limit of the current IRemoteTestListener implementation in testng eclipse plugin causes the issue. it posts the events to the UI viewer rendering thread which is falling behind the testng main thread, when all test completed, the rendering has completed yet.

https://github.com/testng-team/testng-eclipse/blob/master/testng-eclipse-plugin/src/main/org/testng/eclipse/ui/TestRunnerViewPart.java#L1421-L1437

You will easily see the issue if your tests are completed too fast.
for now, I will leave this issue open until a simple workaround or a proper fix comes out.