Not showing platforms as available after upgrading to EKS 1.30

[wesleyhaws]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

Background

We haven't changed anything with our pipelines and have been using:

- name: Set up QEMU
  uses: docker/setup-qemu-action@v3

Successfully. After upgrading our EKS cluster to version 1.30 I started noticing that the support platforms were saying they were getting installed but now the "Extracting available platforms" only shows the original two from the node.

We haven't changed anything with our self hosted github runner container either and it worked previously as well.

Expected behaviour

When running the step with:

with:
    platforms: arm64,arm

I expect it to install those platforms and make them available.

Actual behaviour

As a test I ran with:

- name: Set up QEMU
  uses: docker/setup-qemu-action@v3
  with:
    platforms: arm64,arm

And the output is:

Installing QEMU static binaries
  /usr/local/bin/docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install arm64,arm
  installing: arm64 OK
  installing: arm OK
  {
    "supported": [
      "linux/amd64",
      "linux/arm64",
      "linux/386",
      "linux/arm/v7",
      "linux/arm/v6"
    ],
    "emulators": [
      "qemu-aarch64",
      "qemu-arm"
    ]
  }
Extracting available platforms
  linux/amd64,linux/386

as you can see from the above output I would expect arm64 and arm to be listed as the available platforms.

Is there a gotcha I should be aware of? The image tonistiigi/binfmt:latest hasn't been updated in quite some time, so I doubt that the issue is from there 🤷 .

Repository URL

No response

Workflow run URL

No response

YAML workflow

name: Test workflow
on:
  push:
    paths:
      - "*"
jobs:
  dev-main:
    runs-on: [self-hosted, dev]
    steps:
      - name: Check out repository code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          platforms: arm64,arm

Workflow logs

Successfully checks out the code. I'm including the "failed" step from qemu install:

> Run docker/setup-qemu-action@v3
  with:
    platforms: arm64,arm
    image: docker.io/tonistiigi/binfmt:latest
> Docker info
  /usr/local/bin/docker version
  Client:
   Version:           26.1.2
   API version:       1.45
   Go version:        go1.21.10
   Git commit:        211e74b
   Built:             Wed May  8 13:59:47 2024
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.2
    API version:      1.45 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       ef1912d
    Built:            Wed May  8 13:26:45 2024
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          v1.7.15
    GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
   runc:
    Version:          1.1.12
    GitCommit:        v1.1.12-0-g51d5e94
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/local/bin/docker info
  Client:
   Version:    26.1.2
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.14.0
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.27.0
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 0
   Server Version: 26.1.2
   Storage Driver: overlay2
    Backing Filesystem: xfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: true
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 1
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 926c9586fe4a6236699318391cd44976a98e31f1
   runc version: v1.1.12-0-g51d5e94
   init version: de40ad0
   Security Options:
    seccomp
     Profile: builtin
   Kernel Version: 5.10.226-214.879.amzn2.x86_64
   Operating System: Ubuntu 22.04.4 LTS (containerized)
   OSType: linux
   Architecture: x86_64
   CPUs: 8
   Total Memory: 15.27GiB
   Name: sre-docker-ecr-9gxjv-xp2j4
   ID: 1b8c8ed6-7b70-4663-8224-898509e48595
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false
   Product License: Community Engine
  
Pulling binfmt Docker image
  /usr/local/bin/docker pull docker.io/tonistiigi/binfmt:latest
  latest: Pulling from tonistiigi/binfmt
  8d4d64c318a5: Pulling fs layer
  e9c608ddc3cb: Pulling fs layer
  e9c608ddc3cb: Verifying Checksum
  e9c608ddc3cb: Download complete
  8d4d64c318a5: Verifying Checksum
  8d4d64c318a5: Download complete
  8d4d64c318a5: Pull complete
  e9c608ddc3cb: Pull complete
  Digest: sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55
  Status: Downloaded newer image for tonistiigi/binfmt:latest
  docker.io/tonistiigi/binfmt:latest
> Image info
  /usr/local/bin/docker image inspect docker.io/tonistiigi/binfmt:latest
  [
      {
          "Id": "sha256:354472a378935adfe74a19600b89bd9ada7bb058306fff23b3d6613405852faf",
          "RepoTags": [
              "tonistiigi/binfmt:latest"
          ],
          "RepoDigests": [
              "tonistiigi/binfmt@sha256:66e11bea77a5ea9d6f0fe79b57cd2b189b5d15b93a2bdb925be22949232e4e55"
          ],
          "Parent": "",
          "Comment": "buildkit.dockerfile.v0",
          "Created": "2022-08-02T19:13:20.178433831Z",
          "DockerVersion": "",
          "Author": "",
          "Config": {
              "Hostname": "",
              "Domainname": "",
              "User": "",
              "AttachStdin": false,
              "AttachStdout": false,
              "AttachStderr": false,
              "Tty": false,
              "OpenStdin": false,
              "StdinOnce": false,
              "Env": [
                  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                  "QEMU_PRESERVE_ARGV0=1"
              ],
              "Cmd": null,
              "Image": "",
              "Volumes": {
                  "/tmp": {}
              },
              "WorkingDir": "/",
              "Entrypoint": [
                  "/usr/bin/binfmt"
              ],
              "OnBuild": null,
              "Labels": {
                  "org.opencontainers.image.created": "2022-08-02T18:32:39.936Z",
                  "org.opencontainers.image.description": "Cross-platform emulator collection distributed with Docker images",
                  "org.opencontainers.image.licenses": "MIT",
                  "org.opencontainers.image.revision": "a161c41c7aeaf3ef1c5b97f91aa02a12cca73432",
                  "org.opencontainers.image.source": "https://github.com/tonistiigi/binfmt",
                  "org.opencontainers.image.title": "Binfmt",
                  "org.opencontainers.image.url": "https://github.com/tonistiigi/binfmt",
                  "org.opencontainers.image.version": "qemu-v7.0.0-28"
              }
          },
          "Architecture": "amd64",
          "Os": "linux",
          "Size": 60182964,
          "GraphDriver": {
              "Data": {
                  "LowerDir": "/var/lib/docker/overlay2/d8f9a44ec85975c525bef7d2ccfbd8939723717cf4840cc7ae07419ba664962d/diff",
                  "MergedDir": "/var/lib/docker/overlay2/5e685716e0d815416eba78ea3c609c8a2f3afcc09e11b2843e6009d9975581b0/merged",
                  "UpperDir": "/var/lib/docker/overlay2/5e685716e0d815416eba78ea3c609c8a2f3afcc09e11b2843e6009d9975581b0/diff",
                  "WorkDir": "/var/lib/docker/overlay2/5e685716e0d815416eba78ea3c609c8a2f3afcc09e11b2843e6009d9975581b0/work"
              },
              "Name": "overlay2"
          },
          "RootFS": {
              "Type": "layers",
              "Layers": [
                  "sha256:4c67e4044f8c0fe3e3efaf76f2a3d5d3d866f8ef2e8a9da756949d90e576baa0",
                  "sha256:949acf1cb73a60306e050836deb85a26fe23e226f6bcc499872b057efbf22dd1"
              ]
          },
          "Metadata": {
              "LastTagTime": "0001-01-01T00:00:00Z"
          }
      }
  ]
> Installing QEMU static binaries
  /usr/local/bin/docker run --rm --privileged docker.io/tonistiigi/binfmt:latest --install arm64,arm
  installing: arm64 OK
  installing: arm OK
  {
    "supported": [
      "linux/amd64",
      "linux/arm64",
      "linux/386",
      "linux/arm/v7",
      "linux/arm/v6"
    ],
    "emulators": [
      "qemu-aarch64",
      "qemu-arm"
    ]
  }
> Extracting available platforms
  linux/amd64,linux/386

BuildKit logs

No response

Additional info

No response

[wesleyhaws]

Circling back around to this. Has there been any recent findings?

EDIT: I spent today debugging this and was able to get multi arch images to start building by doing this (NOTE: In the example here the --install all can probably be replaced with --install arm64,arm):

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --install all
docker buildx create --platform linux/amd64,linux/arm64 --driver=docker-container --name=multiarchbuilder --use --bootstrap
docker buildx use multiarchbuilder
echo "---------"
docker buildx ls
echo "---------"

Then making sure I build using that builder:

docker build --builder=multiarchbuilder ....

Also worked with straight buildx:

docker buildx --builder=multiarchbuilder  --platforms=linux/amd64,linux/arm64 ...

Not sure what needs to change here, maybe it was the use of multiarch/qemu-user-static vs the image used in the code here? idk. 🤷