Unable to Retrieve Benchmarks Using Coreruns

[AvishaiDotan]

Environment Setup

I attempted to set up the environment for running benchmarks to complete this issue by building two CoreRun executables:

windows.x64.Debug
windows.x64.Release
The builds were successful, and no errors were encountered during the process. I followed the instructions provided in the documentation for runtime repository.

Benchmark Example

Here is the benchmark code I used:

> [DisassemblyDiagnoser]
> public class IntroDisassembly
> {
>     private const int Iterations = 10;
> 
>     [Benchmark]
>     public string StringConcatenationWithPlus()
>     {
>         string result = "";
>         for (int i = 0; i < Iterations; i++)
>         {
>             result += i.ToString();
>         }
>         return result;
>     }
> 
>     [Benchmark]
>     public string StringConcatenationWithStringBuilder()
>     {
>         var sb = new System.Text.StringBuilder();
>         for (int i = 0; i < Iterations; i++)
>         {
>             sb.Append(i);
>         }
>         return sb.ToString();
>     }
> }
> 
> class Program
> {
>     static void Main(string[] args)
>     {
>         var summary = BenchmarkRunner.Run<IntroDisassembly>(null, args);
>     }
> }

Command Used

I ran the benchmark using the following command:
dotnet run -c Release --corerun some-path\windows.x64.Release\corerun.exe some-path\windows.x64.Debug\corerun.exe

Issue Observed

Despite the setup being seemingly correct, the benchmarks did not execute successfully. Here is the output:

// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 4 benchmark(s) in total *****
// ***** Building 2 exe(s) in Parallel: Start   *****
// ***** Done, took 00:01:26 (86.92 sec)   *****
// Found 4 benchmarks:
//   IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
//   IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
//   IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
//   IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)

Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\b53bf288-4775-4a8f-a902-6d613279c5a9\corerun.exe 5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d.dll --anonymousPipes 1496 1756 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithPlus --job Toolchain=\windows.x64.Debug\corerun.exe --benchmarkId 0 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 35920 has exited with code -532462766.

// ** Remained 3 (75.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\b53bf288-4775-4a8f-a902-6d613279c5a9\corerun.exe 5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d.dll --anonymousPipes 2000 1996 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithStringBuilder --job Toolchain=\windows.x64.Debug\corerun.exe --benchmarkId 1 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\5ed8e74d-4c51-43a3-b724-0dbe9f39ca6d\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 1604 has exited with code -532462766.

// ** Remained 2 (50.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\6f8b4d58-36dd-4e1f-b34a-3b9edee9857a\corerun.exe 571de9ef-b65b-4415-8329-ba2a6c33caec.dll --anonymousPipes 2028 1784 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithPlus --job Toolchain=\windows.x64.Release\corerun.exe --benchmarkId 0 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\571de9ef-b65b-4415-8329-ba2a6c33caec\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 36464 has exited with code -532462766.

// ** Remained 1 (25.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// **************************
// Benchmark: IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
// *** Execute ***
// Launch: 1 / 1
// Execute: F:\OpenSource\runtime\artifacts\bin\coreclr\6f8b4d58-36dd-4e1f-b34a-3b9edee9857a\corerun.exe 571de9ef-b65b-4415-8329-ba2a6c33caec.dll --anonymousPipes 1496 2004 --benchmarkName BenchmarkDemo.IntroDisassembly.StringConcatenationWithStringBuilder --job Toolchain=\windows.x64.Release\corerun.exe --benchmarkId 1 in F:\OpenSource\Tests\BenchmarkDemo\bin\Release\net8.0\571de9ef-b65b-4415-8329-ba2a6c33caec\bin\Release\net8.0\publish
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 35880 has exited with code -532462766.

// ** Remained 0 (0.0%) benchmark(s) to run. Estimated finish 2024-12-21 20:34 (0h 0m from now) **
Successfully reverted power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)
// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-report.csv
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-report-github.md
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-report.html
  BenchmarkDotNet.Artifacts\results\BenchmarkDemo.IntroDisassembly-asm.md

// * Detailed results *
IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
Runtime = ; GC =
There are not any results runs

IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
Runtime = ; GC =
There are not any results runs

IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
Runtime = ; GC =
There are not any results runs

IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
Runtime = ; GC =
There are not any results runs

// * Summary *

BenchmarkDotNet v0.14.0, Windows 10 (10.0.19045.5247/22H2/2022Update)
AMD Ryzen 5 3600, 1 CPU, 12 logical and 6 physical cores
.NET SDK 9.0.101
  [Host] : .NET 8.0.11 (8.0.1124.51707), X64 RyuJIT AVX2


| Method                               | Job        | Toolchain                        | Mean | Error | Ratio | RatioSD |
|------------------------------------- |----------- |--------------------------------- |-----:|------:|------:|--------:|
| StringConcatenationWithPlus          | Job-YCGGKX | \windows.x64.Debug\corerun.exe   |   NA |    NA |     ? |       ? |
| StringConcatenationWithPlus          | Job-GKFKQX | \windows.x64.Release\corerun.exe |   NA |    NA |     ? |       ? |
|                                      |            |                                  |      |       |       |         |
| StringConcatenationWithStringBuilder | Job-YCGGKX | \windows.x64.Debug\corerun.exe   |   NA |    NA |     ? |       ? |
| StringConcatenationWithStringBuilder | Job-GKFKQX | \windows.x64.Release\corerun.exe |   NA |    NA |     ? |       ? |

Benchmarks with issues:
  IntroDisassembly.StringConcatenationWithPlus: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
  IntroDisassembly.StringConcatenationWithPlus: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)
  IntroDisassembly.StringConcatenationWithStringBuilder: Job-YCGGKX(Toolchain=\windows.x64.Debug\corerun.exe)
  IntroDisassembly.StringConcatenationWithStringBuilder: Job-GKFKQX(Toolchain=\windows.x64.Release\corerun.exe)

// * Warnings *
BaselineCustomAnalyzer
  Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline value is too close to zero.
Environment
  Summary -> Detected error exit code from one of the benchmarks. It might be caused by following antivirus software:
        - Windows Defender (windowsdefender://)
        - Kaspersky Total Security (C:\Program Files (x86)\Kaspersky Lab\Kaspersky Total Security 21.3\wmiav.exe)
Use InProcessEmitToolchain or InProcessNoEmitToolchain to avoid new process creation.


// * Legends *
  Mean    : Arithmetic mean of all measurements
  Error   : Half of 99.9% confidence interval
  Ratio   : Mean of the ratio distribution ([Current]/[Baseline])
  RatioSD : Standard deviation of the ratio distribution ([Current]/[Baseline])
  1 ns    : 1 Nanosecond (0.000000001 sec)

// * Diagnostic Output - DisassemblyDiagnoser *
No benchmarks were disassembled

// ***** BenchmarkRunner: End *****
Run time: 00:00:17 (17.43 sec), executed benchmarks: 4

Global total time: 00:01:44 (104.67 sec), executed benchmarks: 4
// * Artifacts cleanup *
Artifacts cleanup is finished

[kg]

I'm also experiencing this.

E:\Documents\Projects\performance\src\benchmarks\micro>E:\Documents\Projects\runtime\dotnet.cmd run -c Release -f net10.0 --artifacts "E:\Documents\Projects\benchmark-results\runtime" --coreRun "E:\Documents\Projects\runtime\artifacts\bin\testhost\net10.0-windows-Release-x64\shared\Microsoft.NETCore.App\10.0.0\CoreRun.exe" --filter "*CtorGivenSize<String>*"
E:\Documents\Projects\runtime\.dotnet
E:\Documents\Projects\runtime\.dotnet\sdk\10.0.100-alpha.1.24610.7\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.EolTargetFrameworks.targets(32,5): warning NETSDK1138: The target framework 'net7.0' is out of support and will not receive security updates in the future. Please refer to https://aka.ms/dotnet-core-support for more information about the support policy.
// Validating benchmarks:
// ***** BenchmarkRunner: Start   *****
// ***** Found 8 benchmark(s) in total *****
// ***** Building 1 exe(s) in Parallel: Start   *****
// start dotnet  restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\obj\Release\net10.0/" /p:OutDir="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" /p:OutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" in E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4
// command took 0.02 sec and exited with -2147450735
// start dotnet  publish -c Release --no-restore /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true /p:IntermediateOutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\obj\Release\net10.0/" /p:OutDir="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" /p:OutputPath="E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" --output "E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish/" in E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4
// command took 0.01 sec and exited with -2147450735
// ***** Done, took 00:00:00 (0.29 sec)   *****
// Found 8 benchmarks:
//   CtorGivenSize<String>.Array: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.List: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.HashSet: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.Dictionary: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.Queue: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.Stack: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.SortedList: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
//   CtorGivenSize<String>.ConcurrentDictionary: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]

// **************************
// Benchmark: CtorGivenSize<String>.Array: Job-WWTTVM(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Size=512]
// *** Execute ***
// Launch: 1 / 1
Did not find E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish\d84672c5-5a3d-4073-8e51-757c5b0f4da4.dll, but the folder contained:
d84672c5-5a3d-4073-8e51-757c5b0f4da4.dll.config
Executable E:\Documents\Projects\performance\artifacts\bin\MicroBenchmarks\Release\net10.0\d84672c5-5a3d-4073-8e51-757c5b0f4da4\bin\Release\net10.0\publish\d84672c5-5a3d-4073-8e51-757c5b0f4da4.dll not found
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.

It seems like the restore and publish are both failing with nonzero exit codes, but BDN ignores that and sails along trying to run the benchmarks anyway. I'm not sure what's going on since there's no stdout/stderr.

[timcassell]

It seems like the restore and publish are both failing with nonzero exit codes, but BDN ignores that and sails along trying to run the benchmarks anyway.

I'm not sure what's the issue with corerun (I've never used it myself), but that particular issue looks similar to #2533 (should be fixed by #2535).