Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: generate seccomp profile within pipeline #1325

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

ci: generate seccomp profile within pipeline #1325

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
62f2e5f
ci: generate seccomp profile within pipeline
alegrey91 Jan 19, 2025
d4faf39
ci(chart): move values file into different location
alegrey91 Jan 30, 2025
89e01ab
ci(makefile): change installation dir for harpoon
alegrey91 Jan 30, 2025
7ba3453
fix(chart): remove priorityClassName template condition
alegrey91 Jan 30, 2025
c4adc10
fix: helm test
alegrey91 Jan 30, 2025
a362cf7
fix: helm test
alegrey91 Jan 30, 2025
fb9be0d
fix: helm test
alegrey91 Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .github/workflows/releaser.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,40 @@ concurrency:
cancel-in-progress: true

jobs:
seccomp-generation:
name: Seccomp Generation
strategy:
fail-fast: false
matrix:
# differently from the e2e workflow
# we don't need all the versions of kubernetes
# to generate the seccomp profile.
k8s-version: [ 'v1.30.0' ]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 0
- uses: actions/setup-go@3041bf56c941b39c61721a86cd11f3bb1338122a # v5.2.0
with:
go-version-file: 'go.mod'
- uses: azure/setup-helm@fe7b79cd5ee1e45176fcad797de68ecaf3ca4814 # v4
with:
version: v3.14.2
- name: unit tracing
run: sudo make trace-unit
- name: e2e tracing
run: KIND_K8S_VERSION=${{ matrix.k8s-version }} make trace-e2e
- name: build seccomp profile
run: make seccomp
- name: upload artifact
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
with:
name: capsule-seccomp
path: capsule-seccomp.json

create-release:
needs: seccomp-generation
runs-on: ubuntu-latest
permissions:
contents: write
Expand All @@ -33,6 +66,11 @@ jobs:
- uses: anchore/sbom-action/download-syft@df80a981bc6edbc4e220a492d3cbe9f5547a6e75
- name: Install Cosign
uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3.7.0
- name: download artifact
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
name: capsule-seccomp
path: ./capsule-seccomp.json
- name: Run GoReleaser
uses: goreleaser/goreleaser-action@9ed2f89a662bf1735a48bc8557fd212fa902bebf # v6.1.0
with:
Expand Down
2 changes: 2 additions & 0 deletions .goreleaser.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ release:
- `ghcr.io/projectcapsule/charts/{{ .ProjectName }}:{{ .Version }}`

[Review the Major Changes section first before upgrading to a new version](https://artifacthub.io/packages/helm/projectcapsule/capsule/{{ .Version }}#major-changes)
extra_files:
- glob: ./capsule-seccomp.json
checksum:
name_template: 'checksums.txt'
changelog:
Expand Down
15 changes: 15 additions & 0 deletions Dockerfile.tracing
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# replacing capsule with harpoon Docker image,
# (importing only the capsule binary),
# so we can trace syscalls.
FROM alegrey91/harpoon:v0.9.4
WORKDIR /
COPY --from=localhost/capsule:latest /manager .

ENTRYPOINT ["/harpoon", \
"capture", \
"-f", "main.main", \
"-E", "NAMESPACE=capsule-system", \
"-i", "2", \
"-c", "-e", \
"-S", "-D", "/tmp/results/", \
"--", "/manager"]
61 changes: 61 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,20 @@ ko-build-capsule: ko
.PHONY: ko-build-all
ko-build-all: ko-build-capsule

.PHONY: docker-build-capsule-trace
docker-build-capsule-trace:
# we build the capsule docker image at first
# so that we can create the one for tracing
# getting the binary from the capsule one.
@docker build \
--no-cache \
-t localhost/capsule:latest \
-f Dockerfile .
@docker build \
--no-cache \
-t localhost/capsule-tracing:latest \
-f Dockerfile.tracing .

# Docker Image Publish
# ------------------

Expand Down Expand Up @@ -222,6 +236,12 @@ KO_VERSION = v0.14.1
ko:
$(call go-install-tool,$(KO),github.com/google/ko@$(KO_VERSION))

HARPOON := /usr/sbin/harpoon
HARPOON_VERSION := v0.9.4
harpoon: ## Download harpoon locally if necessary.
@curl -s https://raw.githubusercontent.com/alegrey91/harpoon/main/install | \
sudo bash -s -- --install-version $(HARPOON_VERSION)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big fan of this, we should try to install binaries into the project's bin/ folder, i am assuming this is going to install harpoon at system level. Since it,s go, can't we make a target like the other binary dependencies:

HARPOON = $(shell pwd)/bin/harpoon
HARPOON_VERSION = v0.9.4
harpoon:
	$(call go-install-tool,$(HARPOON),github.com/alegrey91/harpoon@$(HARPOON_VERSION))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @oliverbaehler, unfortunately I don't think we can currently use go install with harpoon.
The project has a complex toolchain that requires a separated compilation of the ebpf object.
About the installation directory, I think this is fine, I can set the location through the install script.


####################
# -- Helpers
####################
Expand Down Expand Up @@ -294,10 +314,51 @@ e2e-install: e2e-load-image
capsule \
./charts/capsule

.PHONY: trace-install
trace-install: trace-load-image
helm upgrade \
--dependency-update \
--debug \
--install \
--namespace capsule-system \
--create-namespace \
--set 'manager.image.pullPolicy=Never' \
--set 'manager.image.registry=localhost' \
--set 'manager.image.repository=capsule-tracing' \
--set 'manager.resources=null'\
--set "manager.image.tag=latest" \
--set "manager.image.securityContext.privileged=true" \
--set 'manager.livenessProbe.failureThreshold=10' \
--set 'manager.readinessProbe.failureThreshold=10' \
--values charts/capsule/values-tracing.yaml \
capsule \
./charts/capsule

.PHONY: trace-e2e
trace-e2e: kind
$(MAKE) docker-build-capsule-trace
$(KIND) create cluster --wait=60s --image kindest/node:$${KIND_K8S_VERSION:-v1.27.0} --config hack/kind-cluster.yml
$(MAKE) trace-install
$(MAKE) e2e-exec
$(KIND) delete cluster --name capsule-tracing

.PHONY: trace-unit
trace-unit: harpoon
$(HARPOON) analyze -e .git/ -e assets/ -e charts/ -e config/ -e docs/ -e e2e/ -e hack/ --directory /tmp/artifacts/ --save
$(HARPOON) hunt -D /tmp/results -F harpoon-report.yml --include-cmd-stdout --save

.PHONY: seccomp
seccomp:
$(HARPOON) build --add-syscall-sets=dynamic,docker -D /tmp/results --name capsule-seccomp.json --save

.PHONY: e2e-load-image
e2e-load-image: kind ko-build-all
$(KIND) load docker-image --nodes capsule-control-plane --name capsule $(CAPSULE_IMG):$(VERSION)

.PHONY: trace-load-image
trace-load-image:
$(KIND) load docker-image --nodes capsule-tracing-control-plane --name capsule-tracing localhost/capsule-tracing:latest

.PHONY: e2e-exec
e2e-exec: ginkgo
$(GINKGO) -v -tags e2e ./e2e
Expand Down
4 changes: 4 additions & 0 deletions charts/capsule/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ Here the values you can override:
| Key | Type | Default | Description |
|-----|------|---------|-------------|
| manager.hostNetwork | bool | `false` | Specifies if the container should be started in hostNetwork mode. Required for use in some managed kubernetes clusters (such as AWS EKS) with custom CNI (such as calico), because control-plane managed by AWS cannot communicate with pods' IP CIDR and admission webhooks are not working |
| manager.hostPID | bool | `false` | Specifies if the container should be started in hostPID mode. |
| manager.image.pullPolicy | string | `"IfNotPresent"` | Set the image pull policy. |
| manager.image.registry | string | `"ghcr.io"` | Set the image registry of capsule. |
| manager.image.repository | string | `"projectcapsule/capsule"` | Set the image repository of capsule. |
Expand All @@ -165,6 +166,9 @@ Here the values you can override:
| manager.rbac.existingRoles | list | `[]` | Specifies further cluster roles to be added to the Capsule manager service account. |
| manager.readinessProbe | object | `{"httpGet":{"path":"/readyz","port":10080}}` | Configure the readiness probe using Deployment probe spec |
| manager.resources | object | `{}` | Set the resource requests/limits for the Capsule manager container |
| manager.securityContext | object | `{}` | Set the securityContext for the Capsule container |
| manager.volumeMounts | list | `[]` | Set the additional volumeMounts needed for the Capsule manager container |
| manager.volumes | list | `[]` | Set the additional volumes needed for the Capsule manager container |
| manager.webhookPort | int | `9443` | Set an alternative to the default container port. Useful for use in some kubernetes clusters (such as GKE Private) with aggregator routing turned on, because pod ports have to be opened manually on the firewall side |

### ServiceMonitor Parameters
Expand Down
32 changes: 25 additions & 7 deletions charts/capsule/templates/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,15 @@ spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
{{- end }}
{{- if .Values.manager.hostPID }}
hostPID: {{ .Values.manager.hostPID }}
{{- else }}
hostPID: false
{{- end }}
{{- if eq .Values.priorityClassName "" }}
{{- else }}
alegrey91 marked this conversation as resolved.
Show resolved Hide resolved
priorityClassName: {{ .Values.priorityClassName }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand All @@ -59,13 +67,16 @@ spec:
secret:
defaultMode: 420
secretName: {{ include "capsule.secretTlsName" . }}
{{- if .Values.manager.volumes }}
{{- toYaml .Values.manager.volumes | nindent 8 }}
{{- end }}
containers:
- name: manager
args:
- --webhook-port={{ .Values.manager.webhookPort }}
- --enable-leader-election
- --zap-log-level={{ default 4 .Values.manager.options.logLevel }}
- --configuration-name={{ .Values.manager.options.capsuleConfiguration }}
- --webhook-port={{ .Values.manager.webhookPort }}
- --enable-leader-election
- --zap-log-level={{ default 4 .Values.manager.options.logLevel }}
- --configuration-name={{ .Values.manager.options.capsuleConfiguration }}
image: {{ include "capsule.managerFullyQualifiedDockerImage" . }}
imagePullPolicy: {{ .Values.manager.image.pullPolicy }}
env:
Expand All @@ -85,12 +96,19 @@ spec:
readinessProbe:
{{- toYaml .Values.manager.readinessProbe | nindent 12}}
volumeMounts:
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
- mountPath: /tmp/k8s-webhook-server/serving-certs
name: cert
readOnly: true
{{- if .Values.manager.volumeMounts }}
{{- toYaml .Values.manager.volumeMounts | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.manager.resources | nindent 12 }}
securityContext:
{{- if .Values.manager.securityContext }}
{{- toYaml .Values.manager.securityContext | nindent 12 }}
{{- else }}
{{- toYaml .Values.securityContext | nindent 12 }}
{{- end }}
Comment on lines +105 to +109
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just use securityContext ? Or what's the reason here

Copy link
Contributor Author

@alegrey91 alegrey91 Jan 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Values.securityContext is used by kubelet image.
.Values.manager.securityContext is used only by the capsule image when built for tracing.
When we don't set .Values.manager.securityContext, then .Values.securityContext come into play for both the images.
Does it make sense to you?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which kubelet image? We just have the manager that we are deploying..

ARe you talking about this here:

trace-install: trace-load-image
	helm upgrade \
	    --dependency-update \
		--debug \
		--install \
		--namespace capsule-system \
		--create-namespace \
		--set 'manager.image.pullPolicy=Never' \
		--set 'manager.image.registry=localhost' \
		--set 'manager.image.repository=capsule-tracing' \
		--set 'manager.resources=null'\
		--set "manager.image.tag=latest" \
		--set "manager.image.securityContext.privileged=true" \
		--set 'manager.livenessProbe.failureThreshold=10' \
		--set 'manager.readinessProbe.failureThreshold=10' \
		--values charts/capsule/values-tracing.yaml \
		capsule \
		./charts/capsule

This specifcally manager.image.securityContext.privileged=true (This path is wrong btw)? So the rest of the securitycontext is too restrictive? Otherwise make sure it points to the same values in the Makefile as in the chart values

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I meant the kubectl image which is mentioned in the values.yaml.
The point is that .Values.securityContext is shared between the capsule Deployment and the kubectl Job, so in order to be able to overwrite only the capsule Deployment I created a dedicated item for it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jobs take sec from .Vales.global.jobs.kubectl.securityContext, they are no longer shared

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, actually the reason why I created .Values.manager.securityContext was that when I overwrite the default values with tracing-values.yaml, this result in the following setting:

          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add:
              - SYS_ADMIN
              - NET_ADMIN
              - PERFOM
              drop:
              - ALL
            privileged: true
            readOnlyRootFilesystem: true

This is a combination of both (values.yaml, ci/tracing-values.yaml) that doesn't make sense.
(privilege: true and allowPrivilegeEscalation: false cannot be used at the same time).
For this reason I created .Values.manager.securityContext, so I could specify a different securityContext in particular cases where I need it.

{{- end }}
{{- end }}
Loading
Loading