ctr cann't export image pulled by crictl

[johnsonisjiang]

What happened:

when in containerd environment, if we use old version crictl and ctr, no matter the images pulled by ctr or crictl, the ctr can export the image successfully.
But when crictl >= 1.24.0, the issue happens, the image pulled by crictl, when use ctr export, it will failed; For comparison, images pulled by ctr have no problem.

ctr export image which pulled by crictl , will show error message

ctr -n k8s.io images export my_image.tar sha256:af282666faecdad41485bea589ff4c6f25af2abbf76c7b8dff24e7df2570798e
ctr: failed to get reader: content digest sha256:10d56570098259f40562e64b4d5ad9374c94af3e00688d62827e8c79c47d9617: not found

the crictl version: 1.25.0, ctr version 1.7.20

What you expected to happen:

I have another env (crictl:1.19.0; ctr:1.3.7), in this machine, images pulled by crictl can be exported by ctr successfully.
I think there maybe some changes between crictl versions, which changes the export perform.
Would you please tell me how to export images pulled by new version crictl?

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

the crictl version: 1.25.0, ctr version 1.7.20

[saschagrunert]

THank you for opening this report @johnsonisjiang. Both crictl v1.19.0 and v1.25.0 are end of life. crictl also just invokes the RPC to pull the image, so the main issue seems to be on the containerd side here.

[johnsonisjiang]

THank you for opening this report @johnsonisjiang. Both crictl v1.19.0 and v1.25.0 are end of life. crictl also just invokes the RPC to pull the image, so the main issue seems to be on the containerd side here.

Thanks, I got the reason, if the "/etc/containerd/config.toml"->"discard_unpacked_layers" was true, the image pulled by crictl may exported failed, but it have no effect on images pulled by ctr(ctr use contained api not grpc with containerd, default was false)

[saschagrunert]

@johnsonisjiang unfortunately I don't think there is much we can do from a cri-tools perspective.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[saschagrunert]

I'm going to close this one now, feel free to reopen if there is any action we can take.

[SergeyKanzhelev]

@johnsonisjiang please file a bug with containerd if this is an important scenario for you. In theory the discard_unpacked_layers should not be affecting image exporting. Or at least this must be documented clearly.

[samuelkarp]

In theory the discard_unpacked_layers should not be affecting image exporting.

discard_unpacked_layers causes the original compressed tars for layers to be discarded from the content store. Those layers have a specific digest, computed over the compressed tar. It is not possible to reliably reproduce this compressed tar; file ordering within the tar and differences in compression (either via different settings on the same implementation or entirely different gzip implementations) will affect the resulting file and thus the digest.

In other words: this is working-as-intended if discard_unpacked_layers is set to true. If you expect to export images from containerd, you should set discard_unpacked_layers to false.