Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: add content-md5 header for artifacts #14140

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

akloss-cibo
Copy link

This change will allow objects to be written to buckets with S3 Object Lock enabled.

I have manually fixed bits of the jsonschema and swagger, but I've been unable to build the Java client locally. It's not entirely clear to me that this PR needs to include those changes, but it seems like it does. If a kind soul can give me a tip on getting it to build, I'm happy to try again. (See this comment in the issue.)

Fixes #11182

Motivation

As described in 11182, being able to use an object-lock enabled S3 bucket is desirable for organizations wanted to protect assets from destruction.

Modifications

Added sendContentMD5 setting to S3 artifact configuration, and use that setting to enable sending the MD5 via argoproj/pkg.

Verification

I have built images using this code and enabled this setting and objects appear correctly in S3.

Here's the relevant part of the workflow-controller-configmap:

archiveLogs: true
s3:
  endpoint: s3.amazonaws.com
  bucket: <redacted>
  region: us-west-2
  keyFormat: "workflow\
    /{{workflow.name}}\
    /{{pod.name}}\
    /{{workflow.creationTimestamp.Y}}\
    /{{workflow.creationTimestamp.m}}\
    /{{workflow.creationTimestamp.d}}"
  accessKeySecret:
    name: argo-artifact-access
    key: access-key
  secretKeySecret:
    name: argo-artifact-access
    key: secret-key
  useSDKCreds: false
  sendContentMd5: true

Here's an example artifact:

% aws s3api head-object --bucket com-<redacted> --key workflow/<redacted>-c68c7dc83a/<redacted>-2013690313/2025/01/31/main.log
{
    "AcceptRanges": "bytes",
    "Expiration": "expiry-date=\"Mon, 03 Mar 2025 00:00:00 GMT\", rule-id=\"delete-workflow/\"",
    "LastModified": "2025-01-31T15:52:52+00:00",
    "ContentLength": 756,
    "ETag": "\"740a82039415c5a950e6458f0dc0ad9d\"",
    "VersionId": "HSVfA71cjnXTsaQNw_etCqGSr3yAx4su",
    "ContentType": "text/plain; charset=utf-8",
    "ServerSideEncryption": "aws:kms",
    "Metadata": {},
    "SSEKMSKeyId": "arn:aws:kms:us-west-2:<redacted>:key/<redacted>",
    "ReplicationStatus": "COMPLETED",
    "ObjectLockMode": "COMPLIANCE",
    "ObjectLockRetainUntilDate": "2025-03-10T15:52:51.505000+00:00"
}
%

This change will allow objects to be written to buckets with S3 Object
Lock enabled.

fixes argoproj#11182

Signed-off-by: Alec Kloss <[email protected]>
@MasonM
Copy link
Member

MasonM commented Feb 9, 2025

I checked out your branch locally and ran make pre-commit -B in the DevContainer, and it worked. Here's the commit, if you want to cherry-pick it: MasonM@3c36d35

I'm not sure why it doesn't work for you, but if you aren't using the DevContainer, I suggest trying that out.

Signed-off-by: Mason Malone <[email protected]>
Copy link
Member

@rohankmr414 rohankmr414 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's be verbose in the docs why we need to set sendContentMd5 to true

pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
@rohankmr414 rohankmr414 added the area/artifacts S3/GCP/OSS/Git/HDFS etc label Feb 12, 2025
@rohankmr414
Copy link
Member

rohankmr414 commented Feb 13, 2025

@akloss-cibo can you cherry-pick rohankmr414@c092458 on this branch to include the codegen changes, I've run make pre-commit -B on your branch. Also, can you resolve the conflicts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Artifacts cannot be stored in object-lock enabled S3 buckets
3 participants