Sort out CD

[dazinator]

1. Can anyone help with the following tasks:

• We need to get cake build script to build the gem package as it isn't built at present.
• We need to get cake build script to build the tfs task as it isn't build at present.

2. We need to get all of the build assets being pushed automatically to their respective deployment channels, on every commit push. Can we agree on the following strategy:

• In order to issue a "stable" release, we will tag a commit on master with a stable number.
• Every other commit not tagged with a stable number, and irrespective of branch, will produce an "Unstable" release.

If so we need to:

• Configure gitversion to produce stable / unstable version numbers as per the above.
• Amend the build so that stable assets (i.e of the back of a stable build) are pushed to their stable deployment channels (i.e nuget.org etc)
• Ament the build so that "unstable" assets (everything that isn't a stable build are pushed to their unstable / pre-release deployment channels (i.e appveyor ci nuget feed etc)

Note: There appears to be a cake script for deployments` that does a lot of what we need already, but it's not clear to me when this runs or if someone executes it manually or something. Perhaps someone could clarify whether its still used and how.

We have PR's waiting in the wings, and issues because people can't find the latest packages where they are expecting, and this is because we haven't been able to get a handle on the release process or are too reliant on manual intervention.

See several of the bottom comments here for background.

The assets we need to be pushing (Please amend appropriately) are as seen here:

• NuGetCommandLineBuild --> NuGet.org
• NuGetRefBuild --> NuGet.org
• NuGetExeDotNetCoreBuild --> NuGet.org
• NuGetTaskBuild --> NuGet.org
• NuGetExeBuild --> Chocolatey.org
• GemBuild --> ?
• GitVersionTfsTaskBuild--> ?
• zip --> Docker Hub
• zip-dotnetcore --> Docker Hub

[dazinator]

@JakeGinnivan @asbjornu @gep13 @pascalberger please feel free to chime in

[gep13]

Not in a position to help directly, but I can review/answer questions if there are any.

There are addin's Cake.Gem and Cake.Tfx that might be able to help with the work that needs to be done.

[asbjornu]

I think #1338 might have some information on how the current release process works. Missing from the release list is Homebrew, but since that requires a pull request, I'm not sure we can fully automate that? Or can we?

[gep13]

Or can we?

Automatically creating a PR is something that certainly "can" be done. We have thought about doing this for Cake, as that is also pushed to homebrew, but for now, we do this as a manual step.

[dazinator]

@gep13

There are addin's Cake.Gem and Cake.Tfx that might be able to help with the work that needs to be done.

That sounds like an excellent place to start, thanks

[jnevins-gcm]

The support for netstandard has been underway for about 8 months and we are at the point of evaluating whether GitVersion(Task) is a viable solution for use in a new project. Is there any expectation of when a release may come or any advice that can be offered by the maintainers of this project? I realize and appreciate that there is a lot of pre-work required for this support - I'm just trying to evaluate whether it makes sense to use this tool right now.

[dazinator]

@jnevins-gcm afaik there is no reason why you cant consume the gitversiontask nuget package from the netstandard PR. You just need to add the appveyor nuget feed because its not released to nuget.org yet. Instructions should be at the top of the netstandard PR. If you hit problems let us know.

[jnevins-gcm]

That's what we're doing (with mostly success). The problem is redistributing the custom NuGet.config to all of our repos and build agents. Also, there is an issue with that PR that generates TONS of warnings in our builds and slow down build times by 4x (presumably due to the NuGet package restores that occur on every single project that gets versioned due to the newly introduced use of the MSBuild helper library for resolving dependent packages instead of including dependent dlls directly). Our build output looks like this:

    10>GetVersion:
         INFO [08/10/18 9:37:22:56] IsDynamicGitRepository: False
         INFO [08/10/18 9:37:22:56] Returning Project Root from DotGitDirectory: D:\gitrepo\.git - D:\gitrepo
         INFO [08/10/18 9:37:22:56] Project root is: D:\gitrepo
         INFO [08/10/18 9:37:22:56] DotGit directory is: D:\gitrepo\.git
         INFO [08/10/18 9:37:22:56] IsDynamicGitRepository: False
         INFO [08/10/18 9:37:22:56] Returning Project Root from DotGitDirectory: D:\gitrepo\.git - D:\gitrepo
    13>WriteVersionInfoToBuildLog:
         INFO [08/10/18 9:37:22:57] IsDynamicGitRepository: False
         INFO [08/10/18 9:37:22:58] Returning Project Root from DotGitDirectory: D:\gitrepo\.git - D:\gitrepo
         INFO [08/10/18 9:37:22:58] Project root is: D:\gitrepo
         INFO [08/10/18 9:37:22:58] DotGit directory is: D:\gitrepo\.git
         INFO [08/10/18 9:37:22:58] IsDynamicGitRepository: False
         INFO [08/10/18 9:37:22:58] Returning Project Root from DotGitDirectory: D:\gitrepo\.git - D:\gitrepo
    10>GetVersion:
         INFO [08/10/18 9:37:22:60] Begin: Loading version variables from disk cache
           INFO [08/10/18 9:37:22:60] Begin: Deserializing version variables from cache file D:\gitrepo\.git\gitversion_cache\DB52FF94A455E845DE32CE7BD871A0ABE8FC594C.yml
    13>WriteVersionInfoToBuildLog:
         INFO [08/10/18 9:37:22:62] Begin: Loading version variables from disk cache
           INFO [08/10/18 9:37:22:62] Begin: Deserializing version variables from cache file D:\gitrepo\.git\gitversion_cache\DB52FF94A455E845DE32CE7BD871A0ABE8FC594C.yml
    10>GetVersion:
           INFO [08/10/18 9:37:22:68] End: Deserializing version variables from cache file D:\gitrepo\.git\gitversion_cache\DB52FF94A455E845DE32CE7BD871A0ABE8FC594C.yml (Took: 77.13ms)
         INFO [08/10/18 9:37:22:68] End: Loading version variables from disk cache (Took: 77.93ms)

with tons of these

10>MSBUILD : Task factory warning NMSBT010: There is a mismatch between SDK NuGet version (4.7.0) and the NuGet version the task factory was compiled against (4.6.0). There might occur some exotic errors. [D:\gitrepo\src\Web\Web.csproj]

Also, our builds sporadically fail due to concurrency issues (which we having seen previously with other builds of GitVersionTask). MSBuild is building in parallel and it seems there are locking issues:
WARN [08/10/18 7:40:00:51] Could not determine assembly version: LibGit2Sharp.LockedFileException: failed to lock file 'D:/gitrepo/.git/config.lock' for writing:

[dazinator]

@jnevins-gcm
Could the parrallel build / locking issue you are seeing be anything related to: #1381

In terms of the additional restores slowing builds down, thats certainly something we could investigate, feel free to open a more specific issue, with more details about the csproj etc or a repro or link to a build etc. I would expect nuget packages to hit the cache when restored for the first time, so wouldn't expect too much of a slow down in this area.

With respect to the MSBUILD : Task factory warnings - what version of the dotnet sdk are you building with and what gitversiontask nuget package are you referencing?

Lastly, I tend not to use gitversiontask much these days. I usually just execute gitversion at the start of my build either the .exe directly, or if on a dotnet core platform, you can now also run it via the dotnet command, or if you have access to a docker host, you can invoke it via a docker image. This will give you the version number back, and I usually pass that in to msbuild via /p:PackageVersion. Not sure if this kind of approach would work better for you than the GitVersionTask so just mentioning it just in case.

[jnevins-gcm]

This is great help - thanks!

We will switch to the CLI for GitVersion. How is this runnable via the dotnet command?

Also, any reason you are passing in /p:PackageVersion vs. /p:Version?

Are there any other pieces of functionality that would be missing? (aside from appveyor/build server support which could be implemented pretty easily using the command line for their build worker api)?

[dazinator]

RE the cli, there are instructions at the top of here: #1269

And Nope package version was sufficient for my case. Let me know how /p:Version works out though.

[jnevins-gcm]

Thanks.

And how do you distribute the command line version with your builds since the PackageReference isn't the one installing GitVersion anymore?

Also - tried the command line - blows up here:
https://github.com/GitTools/GitVersion/blob/master/src/GitVersionCore/OutputVariables/VersionVariables.cs

due to

#if NETDESKTOP
                return typeof(VersionVariables).GetProperty(variable).GetValue(this, null) as string;
#else
                throw new NotImplementedException();
                //  return typeof(VersionVariables).GetTypeInfo().GetProperty(variable).GetValue(this, null) as string;
#endif

[dazinator]

RE distribution: You could add the cli package to a tools folder in your repo. Or you could have a build step to nuget install the tools to a directory?

Re: command line exception: what command are you executing? Probably worth raising an issue for that. When running under a build server such as teamcity, appveyor etc etc gitversion should be setting the build number for the build without you having to index into any json variables or anything.

[jnevins-gcm]

gitversion /listvariable semver

Without listvariable it works fine printing the whole json. Seems like it’s just not implemented for netstandard?

[dazinator]

Yeah probably the usage of reflection made that code path an endeavour and it wasn't implemented, or just forgotten about :-/

[JakeGinnivan]

Ok, so https://github.com/GitTools/GitVersion/releases/tag/untagged-e9734003373a071c8f29 is a draft release of the v4.0.0-beta.14 binaries.

Can people please test in their environments, it has been ages since a release so I want to ensure everything is good to go.

I am going to start working through this. There may be a few failed deployments and issues as we go, but I think we are going to have to endure it to move forward. Any issues with me pushing beta.14 and then fixing the errors (if any)?

[asbjornu]

@JakeGinnivan: Awesome! Given the time since last release, I think it’s totally fine that there’s a bit of stumbling before we have a ship-shape CD pipeline.

My vote is for automated pre-releases for every (merge) commit to master and as automated as possible stable releases for tagged commits on master.

[JakeGinnivan]

Ok, so deploy fails with a Cake error. Does someone know how to sort this out:

https://ci.appveyor.com/project/GitTools/gitversion-jdi4p

Have moved back to a draft release at https://github.com/GitTools/GitVersion/releases/tag/untagged-7b1747225ba6f2c21c05 until we sort the deploy script out

[gjonespf]

@JakeGinnivan Cake.Json doesn't seem to be particularly active, and there's an issue currently for new Cake version support:
cake-contrib/Cake.Json#35

YMMV, but to get over the hump I've just been using Newtonsoft.Json, with a quick wrapper doing pretty much same thing.

#addin "nuget:https://www.nuget.org/api/v2?package=Newtonsoft.Json"
using Newtonsoft.Json;

[JakeGinnivan]

New issues:

https://ci.appveyor.com/project/GitTools/gitversion-jdi4p/build/1.0.29

Will work through them

[JakeGinnivan]

beta-14 is out. The NuGet package is still missing, working on sorting that out.

https://github.com/GitTools/GitVersion/releases/tag/v4.0.0-beta.14

[JakeGinnivan]

So the problem is https://www.nuget.org/packages/GitVersion/4.0.0-beta0012

Which I am unsure is being built anymore? It's now GitVersionCore. Do we actually need the library version of GitVersion, or just stick to CommandLine and Task?

[JakeGinnivan]

Chocolatey is failing with:
Response status code does not indicate success: 409 (This package had an issue pushing: An error occurred while executing the command definition. See the inner exception for details.).

VSIX blows up the rest. I am going to disable (comment out) the NuGet package, Chocolatey, VSIX until we can resolve their deployments.

[arturcic]

@ajbjornu , I kept the format for pre-release version like this as 4.0.0-beta0015-1644 as chocolatey and rubygem does not support the + and nuget is showing warnings.

[arturcic]

@dazinator @gep13 @asbjornu @JakeGinnivan . I have updated the scripts to support the stable release as well as pre-release (there were some issues with the versioning format). I guess we should close this issue as @dazinator suggested and open a new issue for the VSTS publishing.

As for this issue, we should just tag the master branch with 4.0.0 and run a new build. Any other suggestions?

[dazinator]

@arturcic agreed. Let's tag first, and check the stable channels. I'll leave this issue open until we have verified everything goes to stable channels as 4.0.0!

[dazinator]

Actually.. just looking at releases:

https://github.com/GitTools/GitVersion/releases

It seems historically, releases have been created in Github as "draft" status first - I believe this creates a tag no?
Against the draft release is the opportunity to put release notes etc.

Do we want the ability to create a draft release first, do the release notes, but then only push the actual assets to the stable channels when the draft release on GitHub is "released" ?

[arturcic]

Does this mean it's a 2-step release process? Does this mean when you we "release" from GitHub we should somehow trigger a publish to all channels? I guess that was the way it worked before. That's why we had a build.cake and a deploy.cake scripts.

I think when we have a stable release we should create a Github release as well, as part of the build, similar with what I saw in the https://github.com/cake-build/cake/blob/develop/build.cake#L594

This way we can still do stable releases in one step and have a Github release as well.

[asbjornu]

I reckon we should release 4.0.0, followed by a 4.1.0 shortly after for that netstandard baseline increment.

Yep! 👍

I think when we have a stable release we should create a Github release as well, as part of the build

That's what I would like. A git tag should be enough to get 99% there. It's more important to get new releases out the door easily than them being with perfect release documentation.

If we can get the GitHub release draft to include automatically created release documentation with GitReleaseManager, GitReleaseNotes or similar, that would be supreme.

[dazinator]

I concur. We can always add the auto creation of a github release (in draft status, with generated release notes) at a later stage too, rather than conflate this any further, We can create a seperate issue for that addition.

Ok then so for now, let's just tag, check the stable build and deployment channels work, then we can manually create a github release for now. I don't envy the person who has to create the github release notes for v4.0.0 :-)

[arturcic]

So I suggest we open an issue related to GitHub release/release notes, so that we are able to release this version and unblock the release process. Creating a Github release from script will be next step to improve the build pipeline. As for now, for stable releases the build script creates the release notes and upload to AppVeyor, so we could manually create the release notes on Github till we have it automatically created.

[dazinator]

Release created: https://github.com/GitTools/GitVersion/releases/tag/v4.0.0

I was curious to know if the act of publishing a tag would automatically trigger a build. Checked AppVeyor and it currently shows a build pending form 2 hours ago: https://ci.appveyor.com/project/GitTools/gitversion/history

Makes me think something maybe broken on AppVeyor as surely builds shouldn't queue for that long?

[arturcic]

Actually it was 2 hours ago when I pushed the latest changes. The build is working fine now

[dazinator]

@arturcic watching it in anticipation! :-)

[dazinator]

I have added some general stuff to the 4.0.0 release notes on github, anyone please feel free to amend or completely change as necessary, it'ls just a bit better to have something over nothing :-)

[arturcic]

Looks like all supported channels have been updated.

For chocolatey we'll have to fix the requirements for approval https://chocolatey.org/packages/GitVersion.Portable/4.0.0

[asbjornu]

I kept the format for pre-release version like this as 4.0.0-beta0015-1644 as chocolatey and rubygem does not support the + and nuget is showing warnings.

Ok, so neither Chocolatey nor RubyGems supports SemVer 2.0, then. That's a shame.

[arturcic]

Ok, so neither Chocolatey nor RubyGems supports SemVer 2.0, then. That's a shame.

and nuget clients < 4.3 I think

[asbjornu]

OK.

For chocolatey we'll have to fix the requirements for approval

How do we fix this? I've got the Chocolatey credentials, but I can't find anywhere to adjust this in ther interface. If I try to "Edit Package", I'm only presented with this error message:

To edit the metadata for a package, please upload an updated version of the package.

NuGet currently does not allow updating package metadata on the website. This helps ensure that the package itself (and the source used to build the package) remains the one true source of package metadata.

This does require that you increment the package version.

[arturcic]

I could run manually the pack for chocolatey and push it from local as I have the API key.

[asbjornu]

I could run manually the pack for chocolatey and push it from local as I have the API key.

So you can fix the approval requirements locally and push them somehow? Or did you have something else in mind?

[arturcic]

That's exactly what I mean

[asbjornu]

[arturcic]

I have pushed the nuget package with the missing files - VERIFICATION.txt and LICENSE.txt. Waiting for moderation

[arturcic]

Looks like the chocolatey was approved.
Hooray!! We have version 4.0.0 published to all supported channels, except Vsts. We can close this issue

[jmezach]

What would need to happen in order to push this to VSTS? Op vr 12 okt. 2018 17:17 schreef Artur <[email protected]>:

…

Looks like the chocolatey was approved. Hooray!! We have version 4.0.0 published to all supported channels, except Vsts — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1445 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKzEShE-2d9jd5uhpfPFxMJD-YNPPu_ks5ukLKIgaJpZM4Vbtg5> .

[arturcic]

Actually we can manually update to latest as we already have an issue registered for build script vsts publish

[dazinator]

@arturcic excellent news. I'll close this, and we can track VSTS under the issue you created :-)
Thank you!

[arturcic]

@jmezach I haved updated to latest version on VSTS