Remove default dependency on Docker CLI for generating Dockerfiles

[lbussell]

This PR contains some changes needed to accomplish Consider using Docker custom build outputs (--output) for generated Dockerfiles and Readmes (dotnet/dotnet-docker#5654), by fixing ImageBuilder requires Docker to run all commands, even if they don't use it (#1428).

These changes shouldn't cause breaks for any features used by any commands. It does remove some filtering options from GenerateDockerfiles, but those options were unused.

Commits are separated into buildable chunks that each pass all tests, if that makes reviewing easier.

The problem

• Consider using Docker custom build outputs (--output) for generated Dockerfiles and Readmes dotnet-docker#5654 requires commands be used in a Dockerfile build and not a running container.
• In order to use ImageBuilder inside a Dockerfile build, we can't use any docker CLI commands because we aren't going to bind-mount the Docker socket.
• docker version and docker info are run for every command.
• Some commands call the docker CLI to figure out the default value for some options (arch and OS version).

Solution

• docker version and docker info are run for every command.
  • This is turned into a new option, --ci that is allowed for every command, that we can use in shared pipeline templates
• Some commands call the docker CLI to figure out the default value for some options (arch and OS version).
  • The arch and OS version filters are not useful for all commands. So, we can split ManifestFilterOptions into two separate sets of options: PlatformFilterOptions and DockerfileFilterOptions. Then, for GenerateDockerfiles, we use only DockerfileFilterOptions so that it won't call the docker CLI anymore.

Using the new ImageBuilder

Here's an example Dockerfile that we would use to generate Readmes and Dockerfiles. (additional work is needed to pass in branch via an ARG, also should really use COPY --link)

FROM mcr.microsoft.com/dotnet-buildtools/image-builder:linux-amd64 AS imagebuilder
WORKDIR /repo
COPY . .

# generate-dockerfiles generates Dockerfiles from templates
FROM imagebuilder AS generate-dockerfiles
RUN [ \
    "/image-builder/Microsoft.DotNet.ImageBuilder", \
    "generateDockerfiles", \
    "--var", "branch=nightly" ]

# generate-readmes generates Readmes from templates
FROM imagebuilder AS generate-readmes
RUN [ \
    "/image-builder/Microsoft.DotNet.ImageBuilder", \
    "generateReadmes", \
    "--manifest", "manifest.json", \
    "--var", "branch=nightly", \
    "'https://github.com/dotnet/dotnet-docker'" ]
RUN [ \
    "/image-builder/Microsoft.DotNet.ImageBuilder", \
    "generateReadmes", \
    "--manifest", "manifest.samples.json", \
    "--var", "branch=main", \
    "'https://github.com/dotnet/dotnet-docker'" ]

# generate-dockerfiles-output contains generated Dockerfiles
FROM scratch AS generate-dockerfiles-output
COPY --from=generate-dockerfiles /repo/src src

# generate-readmes-output contains generated Readmes
FROM scratch AS generate-readmes-output
COPY --from=generate-readmes /repo/*.md ./
COPY --from=generate-readmes /repo/.portal-docs ./.portal-docs

Results

--no-cache used for all Docker build commands, including those in PowerShell scripts. Measured with Measure-Command in PowerShell.

Command	Execution time
.\eng\dockerfile-templates\get-generateddockerfiles.ps1	7.78 seconds
.\eng\readme-templates\get-generatedreadmes.ps1	16.28 seconds
docker build --no-cache --target=generate-dockerfiles-output -f .\Dockerfile -o . .	5.23 seconds
docker build --no-cache --target=generate-readmes-output -f .\Dockerfile -o . .	5.13 seconds

Caveats

Doesn't work on Windows since BuildKit doesn't support WCOW yet.

Try it out

(cloning not required)

# Build ImageBuilder
docker build -t imagebuilder -f ./Dockerfile.linux https://github.com/lbussell/docker-tools.git#ci-mode-2:src/Microsoft.DotNet.ImageBuilder/

$dockerfile="https://raw.githubusercontent.com/lbussell/docker-tools/refs/heads/ci-demo/src/Microsoft.DotNet.ImageBuilder/Dockerfile.demo"

# See available options...
docker build -q --call=targets -f $dockerfile .

# Use the new Dockerfile to generate contents
docker build --no-cache --target=generate-dockerfiles-output -f $dockerfile -o . .
docker build --no-cache --target=generate-readmes-output -f $dockerfile -o . .

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.