Go and Cloud Native User Group Leipzig https://golangleipzig.space
2019-04-12, 19:00 at Basislager.co
Repository with slides will be:
- dependency management has been a pain point for years
$GOPATHwas (or is) unintuitive
Many project (at least 15) developed over time.
The official Go proposal is at https://golang.org/issue/24301, filed on March 20, 2018 and accepted on May 21, 2018.
Focused on simplicity in a couple of ways:
- single code tree
- unified go command and simple fetching of libraries and executables with
go get
Go and versioning series from Russ Cox:
67 pages. XXX: Link to concatenated version.
Although we must add versioning, we also must not remove the best parts of the current go command: its simplicity, speed, and understandability. Today, many programmers mostly don't pay attention to versioning, and everything mostly works fine.
https://research.swtch.com/vgo-intro
This proposal keeps the best parts of go get, adds reproducible builds, adopts semantic versioning, eliminates vendoring, deprecates GOPATH in favor of a project-based workflow, and provides for a smooth migration from dep and its predecessors.
In March 2014, Gustavo Niemeyer created gopkg.in, advertising “stable APIs for the Go language.” The domain is a version-aware GitHub redirector, allowing import paths like gopkg.in/yaml.v1 and gopkg.in/yaml.v2 to refer to different commits (perhaps on different branches) of a single Git repository.
If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.)
Proliferation of tools.
- godep
- glide
- gb
- and many more
Also: GOVENDOREXPERIMENT in Go 1.5.
- 2017, GopherCon talk, The New Era of Go Package Management
- import compatibility rule hinted at by the Go FAQ and gopkg.in; that is, establish the expectation that newer versions of a package with a given import path should be backwards-compatible with older versions
- minimal version selection, to choose which package versions are used in a given build
- introduce the concept of a Go module, a group of packages versioned as a single unit and that declare the minimum requirements that must be satisfied by their dependencies
- define how to retrofit all this into the existing go command, so that basic workflows do not change significantly from today
For me, design means building, tearing down, and building again, over and over.
https://research.swtch.com/vgo-tour
Example of vgo usage.
I personally did not try it out.
A year ago, I believed that putting versions in import paths like this was ugly, undesirable, and probably avoidable. But over the past year, I've come to understand just how much clarity and simplicity they bring to the system.
https://research.swtch.com/vgo-import
The import compatibility rule:
If an old package and a new package have the same import path, the new package must be backwards compatible with the old package.
The import compatibility rule dramatically simplifies the experience of using incompatible versions of a package. When each different version has a different import path, there is no ambiguity about the intended semantics of a given import statement. This makes it easier for both developers and tools to understand Go programs.
The problem here is that the semver spec is really not much more than a way to choose and compare version strings. It says nothing else.
The most valuable part of semver is the encouragement to make backwards-compatible changes when possible.
Ok, we use a new name, but which?
But choosing a new name at every backwards-incompatible change is difficult and unhelpful to users.
We can use semantic versioning and follow the import compatibility rule by including the major version in the import path. Instead of needing to invent a cute but unrelated new name like Pocoauth, Alice can call her new API OAuth2 2.0.0, with the new import path "oauth2/v2".
Hello, Plan 9.
Twenty years ago, Rob Pike and I were modifying the internals of a Plan 9 C library, and Rob taught me the rule of thumb that when you change a function’s behavior, you also change its name.
Enter: Rich Hickey.
Rich Hickey made the point in his “Spec-ulation” talk in 2016 that this approach of only adding new names and behaviors, never removing old names or redefining their meanings, is exactly what functional programming encourages with respect to individual variables or data structures.
- The singleton problem
By including the major version into the import path, it should be clearer to authors that my/thing and my/thing/v2 are different and need to be able to coexist.
Gradual code repair.
One of the key reasons to allow both v1 and v2 of a package to coexist in a large program is to make it possible to upgrade the clients of that package one at a time and still have a buildable result. This is specific instance of the more general problem of gradual code repair.
Automatic API updates via go fix.
To be clear, this is not implemented today, but it seems promising, and this kind of tooling is made possible by giving different things different names. These examples demonstrate the power of having durable, immutable names attached to specific code behavior.
More work for library authors.
Semantic import versioning is more work for authors of packages. They can't just decide to issue v2, walk away from v1, and leave users [like Ugo] to deal with the fallout.
Let's commit to compatibility.
A versioned Go command must decide which module versions to use in each build. I call this list of modules and versions for use in a given build the build list.
For stable development, today's build list must also be tomorrow's build list. But then developers must also be allowed to change the build list: to upgrade all modules, to upgrade one module, or to downgrade one module.
This post presents minimal version selection, a new, simple approach to the version selection problem. Minimal version selection is easy to understand and predict, which should make it easy to work with. It also produces high-fidelity builds, in which the dependencies a user builds are as close as possible to the ones the author developed against.
It is also efficient to implement, using nothing more complex than recursive graph traversals, so that a complete minimal version selection implementation in Go is only a few hundred lines of code.
Minimal version selection assumes that each module declares its own dependency requirements: a list of minimum versions of other modules.
Modules are assumed to follow the import compatibility rule—packages in any newer version should work as well as older ones—so a dependency requirement gives only a minimum version, never a maximum version or a list of incompatible later versions.
Go's current version selection algorithm is simplistic, providing two different version selection algorithms, neither of which is right.
-
The first algorithm is the default behavior of go get: if you have a local version, use that one, or else download and use the latest version.
-
The second algorithm is the behavior of go get -u: download and use the latest version of everything.
- recursive version
- graph traversal
Details at:
An equivalent, more efficient construction is based on graph reachability.
In contrast, minimal version selection prefers the minimum allowed version, which is the exact version requested by some go.mod in the project.
Theory plus some helpers for the real world.
- exclusions
- replacements
In contrast, most other systems, including Cargo and Dep, use the newest version available that meets requirements listed in a “manifest file.” The release of a new version changes their build decisions. To get reproducible builds, these systems add a second mechanism, the “lock file,” which lists the specific versions a build should use.
I realized that both manifest and lock exist for the same purpose: to work around the “upgrade everything to latest” default behavior.
- https://research.swtch.com/vgo-repro (Part 5)
A reproducible build is one that, when repeated, produces the same result.
There is a movement around this, too, e.g. in Debian.
A verifiable build is one that records enough information to be precise about exactly how to repeat it. A verified build is one that checks that it is using the expected source code.
How would you find out the version used to build a Go binary?
$ go get -u rsc.io/goversion
$ goversion /usr/bin | shuf -n 3
/usr/bin/containerd-shim go1.12.1
/usr/bin/yay go1.11.5
/usr/bin/gofmt go1.12.3
Now that the vgo prototype understands module versions, it includes that information in the final binary too, and the new goversion -m flag prints it back out.
Example:
$ goversion -m goimports
Was called go.modverify, now go.sum.
The go.modverify file is a log of the hash of all versions ever encountered: lines are only added, never removed.
The go.modverify feature helps detect unexpected mismatches between downloaded dependencies on different machines. It compares the hashes in go.modverify against hashes computed and saved at module download time.
A Go module is a collection of packages versioned as a unit, along with a go.mod file listing other required modules.
The move to modules is an opportunity for us to revisit and fix many details of how the go command manages source code. The current go get model will be about ten years old when we retire it in favor of modules. We need to make sure that the module design will serve us well for the next decade.
We want to encourage more developers to tag releases of their packages, instead of expecting that users will just pick a commit hash that looks good to them. Tagging explicit releases makes clear what is expected to be useful to others and what is still under development. At the same time, it must still be possible—although maybe not convenient—to request specific commits.
We want to move away from invoking version control tools such as bzr, fossil, git, hg, and svn to download source code. These fragment the ecosystem.
We want to make it possible, at some future point, to introduce a shared proxy for use by the Go community, similar in spirit to those used by Rust, Node, and other languages. At the same time, the design must work well without assuming such a proxy or registry.
We want to eliminate vendor directories. They were introduced for reproducibility and availability, but we now have better mechanisms. Reproducibility is handled by proper versioning, and availability is handled by caching proxies.
Since not all code will be tagged, pseudo versions are possible.
A module version is defined by a tree of source files.
The file format is line-oriented, with // comments only. Each line holds a single directive, which is a single verb (module, require, exclude, or replace, as defined by minimum version selection), followed by arguments
module "my/thing"
require "other/thing" v1.0.2
require "new/thing" v2.3.4
exclude "old/thing" v1.2.3
replace "bad/thing" v1.4.5 => "good/thing" v1.4.5
My goals for the file format were that it be (1) clear and simple, (2) easy for people to read, edit, manipulate, and diff, (3) easy for programs like vgo to read, modify, and write back, preserving comments and general structure, and (4) have room for limited future growth. I looked at JSON, TOML, XML, and YAML but none of them seemed to have those four properties all at once.
The leading v is required, and having three numbers is also required.
All commands (go build, go run, and so on) will download imported source code automatically, if the necessary version is not already present in the download cache on the local system.
The go list command will add access to module information.
The all pattern is redefined to make sense in the world of modules.
Developers can and will be encouraged to work in directories outside the GOPATH tree.
The most significant implication of the isolation rule is that commands like go build, go install, and go test should download versioned dependencies as needed (that is, if not already downloaded and cached).
- Quick start
- Go tool docs: Module maintenance, Download modules to local cache, Edit go.mod from tools or scripts, ...
- Go modules design documents
- GopherCon SG 2018: Go with Versions
- Blog: Go Modules in 2019, Using Go Modules
- justforfunc: #42, #43