windows/nanoserver:1909 throwing hcsshim::CreateScratchLayer - failed failed in Win32: The system cannot find the path specified. (0x3)

[mloskot]

Something must have changed in the mcr.microsoft.com/windows/nanoserver:1909 image in particular as it's been working for my custom images workflow for quite a while and now it's not.

Here is minimal reproducible example Dockerfile

ARG BASE_IMAGE
ARG BASE_IMAGE_TAG

FROM mcr.microsoft.com/windows/nanoserver:${BASE_IMAGE_TAG} AS build
LABEL stage=build
WORKDIR /Temp
RUN echo Test > test.txt

FROM mcr.microsoft.com/windows:${BASE_IMAGE_TAG} AS windows
LABEL stage=build

FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}
WORKDIR /
COPY --from=build /Temp/test.txt /
COPY --from=windows /Windows/System32/opengl32.dll /Windows/System32/

and runner that may be extended with more image tags:

@('20H2', '1909', '1809') | ForEach-Object {
    $baseTag = $_
    $imageName = ("test:{0}" -f $baseTag)
    Write-Output ("========== Building {0} ==============================" -f $imageName)
    docker image build --no-cache --force-rm --tag $imageName --build-arg BASE_IMAGE=mcr.microsoft.com/windows/servercore --build-arg BASE_IMAGE_TAG=$baseTag .
}

Build using base 1909 tag does NOT succeed:

> docker image build --no-cache --force-rm --tag test-1909 --build-arg BASE_IMAGE=mcr.microsoft.com/windows/servercore --build-arg BASE_IMAGE_TAG=1909 .

Sending build context to Docker daemon  3.072kB
Step 1/11 : ARG BASE_IMAGE
Step 2/11 : ARG BASE_IMAGE_TAG
Step 3/11 : FROM mcr.microsoft.com/windows/nanoserver:${BASE_IMAGE_TAG} AS build
 ---> 8ad8cde68ea6
Step 4/11 : LABEL stage=build
hcsshim::CreateScratchLayer - failed failed in Win32: The system cannot find the path specified. (0x3)

Possibly, it's related to microsoft/hcsshim#835

---

Build using 1809 tag does succeed:

> docker image build --no-cache --force-rm --tag test:1809 --build-arg BASE_IMAGE=mcr.microsoft.com/windows/servercore --build-arg BASE_IMAGE_TAG=1809

Sending build context to Docker daemon  3.072kB
Step 1/12 : ARG BASE_IMAGE
Step 2/12 : ARG BASE_IMAGE_TAG
Step 3/12 : FROM mcr.microsoft.com/windows/nanoserver:${BASE_IMAGE_TAG} AS build
 ---> f17a3fdd9bef
Step 4/12 : LABEL stage=build
 ---> Running in da0ea0410e34
Removing intermediate container da0ea0410e34
 ---> 75908064593a
Step 5/12 : WORKDIR /Temp
 ---> Running in d617b805d80f
Removing intermediate container d617b805d80f
 ---> 39028ea027c0
Step 6/12 : RUN echo Test > test.txt
 ---> Running in d25a07d23fb4
Removing intermediate container d25a07d23fb4
 ---> 6584f8267161
Step 7/12 : FROM mcr.microsoft.com/windows:${BASE_IMAGE_TAG} AS windows
 ---> f0ae9ae13a10
Step 8/12 : LABEL stage=build
 ---> Running in 85c4d53dfbae
Removing intermediate container 85c4d53dfbae
 ---> b9f9f2a470a6
Step 9/12 : FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}
 ---> 0ff2786a0d17
Step 10/12 : WORKDIR /
 ---> Running in 735e07ca3cb0
Removing intermediate container 735e07ca3cb0
 ---> 5c10380af2f2
Step 11/12 : COPY --from=build /Temp/test.txt /
 ---> 8c4a80048164
Step 12/12 : COPY --from=windows /Windows/System32/opengl32.dll /Windows/System32/
 ---> b10cba1f2333
Successfully built b10cba1f2333
Successfully tagged test:1809

Build using 20H2 tag does succeed:

> docker image build --no-cache --force-rm --tag test:20H2 --build-arg BASE_IMAGE=mcr.microsoft.com/windows/servercore --build-arg BASE_IMAGE_TAG=20H2

Sending build context to Docker daemon  3.584kB
Step 1/12 : ARG BASE_IMAGE
Step 2/12 : ARG BASE_IMAGE_TAG
Step 3/12 : FROM mcr.microsoft.com/windows/nanoserver:${BASE_IMAGE_TAG} AS build
 ---> 67ae078d1460
Step 4/12 : LABEL stage=build
 ---> Running in ba4bbf9b2da7
Removing intermediate container ba4bbf9b2da7
 ---> cd0e19dde913
Step 5/12 : WORKDIR /Temp
 ---> Running in 7dd192492a9d
Removing intermediate container 7dd192492a9d
 ---> 9575d6b04def
Step 6/12 : RUN echo Test > test.txt
 ---> Running in 1d53618b819b
Removing intermediate container 1d53618b819b
 ---> 5b6ec27ffd66
Step 7/12 : FROM mcr.microsoft.com/windows:${BASE_IMAGE_TAG} AS windows
 ---> 2fc6e1bac200
Step 8/12 : LABEL stage=build
 ---> Running in 356eb75dfd7b
Removing intermediate container 356eb75dfd7b
 ---> d943e66fdf9b
Step 9/12 : FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}
 ---> 0a5d9a80a5d0
Step 10/12 : WORKDIR /
 ---> Running in 11e13cabffe0
Removing intermediate container 11e13cabffe0
 ---> 67319d83f92d
Step 11/12 : COPY --from=build /Temp/test.txt /
 ---> 98b988f75e83
Step 12/12 : COPY --from=windows /Windows/System32/opengl32.dll /Windows/System32/
 ---> 3e12a5fcb12c
Successfully built 3e12a5fcb12c
Successfully tagged test:20H2

Environment

> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      19042  0

 docker version
Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.16.4
 Git commit:        f0df350
 Built:             Wed Jun  2 12:00:56 2021
 OS/Arch:           windows/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.24)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:56:41 2021
  OS/Arch:          windows/amd64
  Experimental:     false

docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.0-beta.3)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 21
 Server Version: 20.10.7
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: hyperv
 Kernel Version: 10.0 19042 (19041.1.amd64fre.vb_release.191206-1406)
 Operating System: Windows 10 Pro Version 2009 (OS Build 19042.1052)
 OSType: windows
 Architecture: x86_64
 CPUs: 24
 Total Memory: 63.77GiB
 Name: DEV690
 ID: X7MU:DIVE:AIL3:XYRQ:WSSJ:LXP6:B6ML:EZ2Z:J2Z4:3DPK:25TU:JQ5Q
 Docker Root Dir: C:\ProgramData\Docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[vrapolinario]

Thanks for raising this issue on 1909 images. However, that image is now out of support as you can check on: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/base-image-lifecycle. We recommend customers use the 2004 or 20H2 images at this point so they can get support on those images. From the information above, I see that the build process succeeded on 20H2.
Please let us know if you have any other questions on this.

[mloskot]

@vrapolinario Thank you for the explanation.

The 1809 is still supported though, isn't it?

Server Core	Long-Term, Semi-Annual	2019, 1809	17763	11/13/2018	01/09/2024	01/09/2029
Nano Server	Semi-Annual	1809	17763	11/13/2018	01/09/2024	N/A
Windows	Semi-Annual	1809	17763	11/13/2018	01/09/2024	N/A

[vrapolinario]

Yes, 1809 is supported. Do you see the error on 1809 as well?

[mloskot]

@vrapolinario No, I don't see the error on 1809. I was just double-checking I've got it right the 'skipping' of 1909 and that the situation is this indeed:

• 1809 - supported
• 1909 - NOT supported
• 2004 - supported
• 20H2 - supported

[vrapolinario]

That is correct as of today. Please keep an eye on that page to see the dates for end of support on each release.