Pull preloaded images based on digest, not tag

[inteon]

Fixes a long-standing issue where images were preloaded by tag instead of digest causing the digest check to fail when the upstream image changed.
Now, the images are pulled by digest, we modify the pulled tarball to have the desired image tag and we use the tarball that way.

Is a long-term fix for the issue we tried to fix here: #224

[tenstad]

LGTM!

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tenstad
Once this PR has been reviewed and has the lgtm label, please ask for approval from inteon. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SgtCoDFish]

Capturing discussion I remember from standup this morning:

• We might actually want to break CI if a digest changes, to alert us to potential supply-chain attacks. It's not certain we want to avoid that.
• This PR essentially introduces a "cert-manager tag" which is pinned to the digest we first observed that tag pointing to. That means that someone running with a locally-pulled image might get different behavior to what we have, since we'll have a different digest and it'll be a bit hidden. It's not certain that we want that behavior.

I don't think either of the above are blockers and I don't have time to actually review this PR - just trying to capture the discussion here.

[SgtCoDFish]

There's been a more recent kind release: https://github.com/kubernetes-sigs/kind/releases/tag/v0.26.0

I wonder if bumping to that is a short-term fix for the kind issues affecting CI?

[inteon]

There's been a more recent kind release: https://github.com/kubernetes-sigs/kind/releases/tag/v0.26.0

I wonder if bumping to that is a short-term fix for the kind issues affecting CI?

See #228

[inteon]

Capturing discussion I remember from standup this morning:

• We might actually want to break CI if a digest changes, to alert us to potential supply-chain attacks. It's not certain we want to avoid that.
• This PR essentially introduces a "cert-manager tag" which is pinned to the digest we first observed that tag pointing to. That means that someone running with a locally-pulled image might get different behavior to what we have, since we'll have a different digest and it'll be a bit hidden. It's not certain that we want that behavior.

I don't think either of the above are blockers and I don't have time to actually review this PR - just trying to capture the discussion here.

• I don't think it is very useful to break CI when the remote registry changes what digest a tag points to. We might want some notification/ alerts maybe? Also, if there was a supply-chain attack that got fixed, we would probably hear about it in another way. By pinning to the digest disregarding the upstream tag, we prevent upstream changes from breaking our flow and/or introducing any new supply-chain attacks.
• I don't think this will be a huge problem, the main offenders are kind's node image and busybox's image. For both, I think we are better off pinning the image digests and not breaking CI. This also ensures our tests are more reproducible (eg. when we want to patch an old version of one of our components and the upstream tag changed in the meantime). Additional to those two images, this mechanism is also used to preload the cert-manager images. These will typically not get their tag overwritten.

[hawksight]

Just adding here that I spent some time looking for other solutions by pulling various image in tarball and oci format. Basically what @inteon has done seems to be sensible.

I'd suggest that we trust kind images when they retag them, moving their version tags to new digests.

It seems kind does not support importing the OCI format images, only tarball. And when pulling a digest bases tarball... you don't get the actual tag in the manifest.json... even id you do @.