All unit test library checks returns true for default vs 2015 unit test library

[AdmiringWorm]

I just created a new default Visual Studio 2015 Unit Test Library (I believe it is a MSTest compatible project) for testing purposes.

I found at that this project was detected as pretty much any unit test library that Cake.Incubator have added checks for.

Basically, Cake.Incubator things the project is a XUnit, NUnit, Fixie and a MSTest project (probably Expecto and FsUnit as well, but didn't check those).

I believe the false positives are caused by an incorrect? check in the following helper method: https://github.com/cake-contrib/Cake.Incubator/blob/develop/src/Cake.Incubator/Project/ProjectParserExtensions.cs#L1152

From the look of it, the helper method would return true for any place this in used as long as the project in a Unit Test library, even if it do not have the necessary references/packages.

Project csproj

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="14.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
    <ProjectGuid>{5C71F804-526F-42BC-AACF-917BC24DDB04}</ProjectGuid>
    <OutputType>Library</OutputType>
    <AppDesignerFolder>Properties</AppDesignerFolder>
    <RootNamespace>MSUnitTests</RootNamespace>
    <AssemblyName>MSUnitTests</AssemblyName>
    <TargetFrameworkVersion>v4.5.2</TargetFrameworkVersion>
    <FileAlignment>512</FileAlignment>
    <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>
    <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>
    <VSToolsPath Condition="'$(VSToolsPath)' == ''">$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)</VSToolsPath>
    <ReferencePath>$(ProgramFiles)\Common Files\microsoft shared\VSTT\$(VisualStudioVersion)\UITestExtensionPackages</ReferencePath>
    <IsCodedUITest>False</IsCodedUITest>
    <TestProjectType>UnitTest</TestProjectType>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <DebugSymbols>true</DebugSymbols>
    <DebugType>full</DebugType>
    <Optimize>false</Optimize>
    <OutputPath>bin\Debug\</OutputPath>
    <DefineConstants>DEBUG;TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <DebugType>pdbonly</DebugType>
    <Optimize>true</Optimize>
    <OutputPath>bin\Release\</OutputPath>
    <DefineConstants>TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
  </PropertyGroup>
  <ItemGroup>
    <Reference Include="System" />
  </ItemGroup>
  <Choose>
    <When Condition="('$(VisualStudioVersion)' == '10.0' or '$(VisualStudioVersion)' == '') and '$(TargetFrameworkVersion)' == 'v3.5'">
      <ItemGroup>
        <Reference Include="Microsoft.VisualStudio.QualityTools.UnitTestFramework, Version=10.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL" />
      </ItemGroup>
    </When>
    <Otherwise>
      <ItemGroup>
        <Reference Include="Microsoft.VisualStudio.QualityTools.UnitTestFramework" />
      </ItemGroup>
    </Otherwise>
  </Choose>
  <ItemGroup>
    <Compile Include="UnitTest1.cs" />
    <Compile Include="Properties\AssemblyInfo.cs" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\ClassLibrary\ClassLibrary.csproj">
      <Project>{1791b52a-64e8-426c-8d73-167afcf42f17}</Project>
      <Name>ClassLibrary</Name>
    </ProjectReference>
  </ItemGroup>
  <Choose>
    <When Condition="'$(VisualStudioVersion)' == '10.0' And '$(IsCodedUITest)' == 'True'">
      <ItemGroup>
        <Reference Include="Microsoft.VisualStudio.QualityTools.CodedUITestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
          <Private>False</Private>
        </Reference>
        <Reference Include="Microsoft.VisualStudio.TestTools.UITest.Common, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
          <Private>False</Private>
        </Reference>
        <Reference Include="Microsoft.VisualStudio.TestTools.UITest.Extension, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
          <Private>False</Private>
        </Reference>
        <Reference Include="Microsoft.VisualStudio.TestTools.UITesting, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL">
          <Private>False</Private>
        </Reference>
      </ItemGroup>
    </When>
  </Choose>
  <Import Project="$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets" Condition="Exists('$(VSToolsPath)\TeamTest\Microsoft.TestTools.targets')" />
  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
  <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
       Other similar extension points exist, see Microsoft.Common.targets.
  <Target Name="BeforeBuild">
  </Target>
  <Target Name="AfterBuild">
  </Target>
  -->
</Project>

[nils-a]

Looking at the current implementation, I would mentally "group" the testing-related extensions in two categories:

• Determine test projects
  • IsDotNetCliTestProject looks fine: It only checks if dotnet test is possible.
  • IsFrameworkTestProject I'm a bit torn here. On the one hand, I'd like to say that the check for the test project-type-id suffices. On the other hand, I'm fairly certain that most of my old .NET framework test projects where classlibs from a project-id view. I think I would "reduce" it to checking the project type id.
  • IsTestProject determines if it's any kind of test or test-like project. Looks fine, but implementation would need to change after my proposed change for IsFrameworkTestProject. (See below)
• Determine a specific test framework
  This is IsXUnitTestProject, IsFsUnitTestProject, IsNUnitTestProject, IsExpectoTestProject, IsFixieTestProject and IsMSTestProject which all lead to IsTestProjectOfType. And here, I think, for determining the test framework in use the fact that a project is a IsFrameworkTestProject or IsDotNetCliTestProject does not matter. I would remove that and check only for reference/package.

So.. I think the testing-related extensions should be re-worked as follows:

• IsDotNetCliTestProject: Stays as it is - It checks if dotnet test should work, free from any test frameworks.
• IsFrameworkTestProject: Remove the checks for the test framework, so that a .NET framework test project is a project that has the "correct" project type id. (Again, free from the test framework in use.)
• Is[whatever framwork]TestProject: Check only for the test framework references. This would mean removing the check for IsNetStandard and IsType(ProjectType.Test) from IsTestProjectOfType.
• IsTestProject - this would essentially keep it's old meaning but since IsFrameworkTestProject has changed a bit, this would need to be modified to IsDotNetCliTestProject or IsFrameworkTestProject or Is[any of the known frameworks]TestProject

@AdmiringWorm I think the sketched out changes should fix the problem at hand, what do you think?

I'd be willing to start working on this, but I'd love to hear what everyone thinks of the proposed changes.

/cc @wwwlicious