Skip to content

Commit

Permalink
Merge pull request #71 from kubescape/docs-working-with-storage
Browse files Browse the repository at this point in the history 
docs: How to work with Storage
  • Loading branch information
@vladklokun
vladklokun authored Dec 12, 2023
2 parents cb8d509 + 19db893 commit 0ffe829
Show file tree
Hide file tree
Showing 2 changed files with 95 additions and 52 deletions.
145 changes: 93 additions & 52 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,62 +3,54 @@
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fkubescape%2Fstorage.svg?type=shield&issueType=license)](https://app.fossa.com/projects/git%2Bgithub.com%2Fkubescape%2Fstorage?ref=badge_shield&issueType=license)


Demonstration of how to use the k8s.io/apiserver library to build a functional API server.
An Aggregated APIServer for the Kubescape internal storage services.

**Note:** go-get or vendor this package as `k8s.io/sample-apiserver`.

## Purpose

You may use this code if you want to build an Extension API Server to use with API Aggregation, or to build a stand-alone Kubernetes-style API server.

However, consider two other options:
* **CRDs**: if you just want to add a resource to your kubernetes cluster, then consider using Custom Resource Definition a.k.a CRDs. They require less coding and rebasing. Read about the differences between Custom Resource Definitions vs Extension API Servers [here](https://kubernetes.io/docs/concepts/api-extension/custom-resources).
* **Apiserver-builder**: If you want to build an Extension API server, consider using [apiserver-builder](https://github.com/kubernetes-incubator/apiserver-builder) instead of this repo. The Apiserver-builder is a complete framework for generating the apiserver, client libraries, and the installation program.

If you do decide to use this repository, then the recommended pattern is to fork this repository, modify it to add your types, and then periodically rebase your changes on top of this repo, to pick up improvements and bug fixes to the apiserver.


## Compatibility

HEAD of this repo will match HEAD of k8s.io/apiserver, k8s.io/apimachinery, and k8s.io/client-go.

## Where does it come from?

`sample-apiserver` is synced from https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/sample-apiserver.
Code changes are made in that location, merged into `k8s.io/kubernetes` and later synced here.
The Kubescape Storage APIServer serves custom resources that Kubescape defines
for its operation. These custom reources might store internal Kubescape
configuration, scan artifacts, computed snapshots etc that help the entire
Kubescape in-cluster solution operate.

## Fetch sample-apiserver and its dependencies

Like the rest of Kubernetes, sample-apiserver has used
[godep](https://github.com/tools/godep) and `$GOPATH` for years and is
now adopting go 1.11 modules. There are thus two alternative ways to
go about fetching this demo and its dependencies.
[godep](https://github.com/tools/godep) and `$GOPATH` for years and is now
adopting go 1.11 modules. While upstream mentions two alternative ways to go
about fetching the sample repository and its dependencies, we recommend and
primarily use only one: using native Go 1.11 modules and vendoring.

### Fetch with godep
### When using native Go 1.11 modules

When NOT using go 1.11 modules, you can use the following commands.
When using go 1.11 modules (`GO111MODULE=on`), you first need to create the appropriate working directory:

```sh
go get -d k8s.io/sample-apiserver
cd $GOPATH/src/k8s.io/sample-apiserver # assuming your GOPATH has just one entry
godep restore
```
mkdir ~/github.com/kubescape
cd ~/github.com/kubescape
```

### When using go 1.11 modules
> [!WARNING]
> Due to the specifics of the code generation script `hack/update-codegen.sh`,
> your working directory should always match your module path. That is,
> `~/github.com/kubescape/storage` for this specific repo. If the directories
> don’t match and you store code in some other directory, you will write code
> in whatever directory you chose, but once you run codegen, it will generate
> the code only in `~/github.com/kubescape/storage`.
When using go 1.11 modules (`GO111MODULE=on`), issue the following
commands --- starting in whatever working directory you like.
Once you have a working directory set up, issue the following
commands in your working directory.

```sh
git clone https://github.com/kubernetes/sample-apiserver.git
cd sample-apiserver
git clone https://github.com/kubescape/storage.git
cd storage
```

Note, however, that if you intend to
[generate code](#changes-to-the-types) then you will also need the
code-generator repo to exist in an old-style location. One easy way
to do this is to use the command `go mod vendor` to create and
populate the `vendor` directory.
Note that when you need to [generate code](#changes-to-the-types) then you will
also need the code-generator repo to exist in an old-style location. One easy
way to do this is to use the command `go mod vendor` to create and populate the
`vendor` directory.

### A Note on kubernetes/kubernetes

Expand All @@ -74,11 +66,46 @@ then you already have a copy of this demo in
### Changes to the Types

If you change the API object type definitions in any of the
`pkg/apis/.../types.go` files then you will need to update the files
generated from the type definitions. To do this, first
[create the vendor directory if necessary](#when-using-go-111-modules)
and then invoke `hack/update-codegen.sh` with `sample-apiserver` as
your current working directory; the script takes no arguments.
`pkg/apis/.../types.go` files then you will need to update the files generated
from the type definitions. To do this, first [pull the dependencies and create
the vendor directory](#when-using-native-go-111-modules). Once you vendored the
dependencies, you will have the code generation scripts in your `vendor`
directory. To make code generation, you need to make them executable:

```
chmod +x vendor/k8s.io/code-generator/*.sh
```

Now you’re all set to generate the code for changed types. Do this with:

```
hack/update-codegen.sh
```

If you see any errors regarding `GOPATH`, just provide it manually:
```
GOPATH=$(go env GOPATH) hack/update-codegen.sh
```

The code generation script will give you warnings about API rule violations.
Don’t mind them. To address these warnings, add them to the exclusion list as
show in the updated upstream repo.

You will also see a warning about `generate-internal-groups.sh` being deprecated:
```
WARNING: generate-internal-groups.sh is deprecated.
WARNING: Please use k8s.io/code-generator/kube_codegen.sh instead.
```

This is valid, and upstream has also been updated to use the latest code
generation script — `kube_codegen.sh`. However, as of now it breaks code
generation for us, and we had no opportunity to reconcile the changes.

Once the code generation finishes successfully, you should be able to run tests and build the binary with no errors:
```
go build -v ./...
go test -v -failfast -count=1 ./...
```

### Authentication plugins

Expand All @@ -102,37 +129,51 @@ import _ "k8s.io/client-go/plugin/pkg/client/auth"

### Build the Binary

With `sample-apiserver` as your current working directory, issue the
With `storage` as your current working directory, issue the
following command:

```
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o artifacts/simple-image/kube-sample-apiserver
make binary
```

### Build the Container Image

With `sample-apiserver` as your current working directory, issue the
following commands with `MYPREFIX` and `MYTAG` replaced by something
suitable.
With `storage` as your current working directory, issue the
following commands:

```
docker build -t MYPREFIX/kube-sample-apiserver:MYTAG ./artifacts/simple-image
docker push MYPREFIX/kube-sample-apiserver:MYTAG
TAG=v1.2.3 make docker-build && TAG=v1.2.3 make docker-push
```

Take note that the Makefile targets use default values for the image tag and the Dockerfile path, so feel free to adjust them as environment variables as needed:
```
TAG=v1.2.3 IMAGE=quay.io/kubescape/storage make docker-build
TAG=v1.2.3 IMAGE=quay.io/kubescape/storage make docker-push
```

### Deploy into a Kubernetes Cluster

Edit `artifacts/example/deployment.yaml`, updating the pod template's image
reference to match what you pushed and setting the `imagePullPolicy`
to something suitable. Then call:
to something suitable.

If you’re running a Minikube cluster locally, build and tag an container image, use it in the `artifacts/example/deployment.yaml`, set `imagePullPolicy: Never` and this will let you use a local container image without having to push it to a container registry.

Then, make sure the appropriate namespace for the APIServer components exists:

```
kubectl apply -f artifacts/example/ns.yaml
```

Finally, create all the other Aggregated APIServer components:

```
kubectl apply -f artifacts/example
```

## Running it stand-alone

During development it is helpful to run sample-apiserver stand-alone, i.e. without
During development it is helpful to run the Storage APIServer stand-alone, i.e. without
a Kubernetes API server for authn/authz and without aggregation. This is possible, but needs
a couple of flags, keys and certs as described below. You will still need some kubeconfig,
e.g. `~/.kube/config`, but the Kubernetes cluster is not used for authn/z. A minikube or
Expand Down Expand Up @@ -208,4 +249,4 @@ only this superuser group is authorized.
```
## Changelog

Kubescape Storage changes are tracked on the [release](https://github.com/kubescape/storage/releases) page
Kubescape Storage changes are tracked on the [release](https://github.com/kubescape/storage/releases) page.
2 changes: 2 additions & 0 deletions hack/update-codegen.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-
# --output-base because this script should also be able to run inside the vendor dir of
# k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
# instead of the $GOPATH directly. For normal projects this can be dropped.

# intentionally omits applyconfigurations because it generates broken code for our types
"${CODEGEN_PKG}/generate-groups.sh" "client,deepcopy,informer,lister,defaulter,conversion,openapi" \
github.com/kubescape/storage/pkg/generated \
github.com/kubescape/storage/pkg/apis \
Expand Down

0 comments on commit 0ffe829

Please sign in to comment.