contrib: add nydus snapshotter

To serve as a containerd remote snapshotter, and power container rootfs
setup with nydus image service.

[Syncs with nydus-snapshotter up to commit
3b5a590f0d8e25abb9309b70060256cf630f1898("PullRequest: 9 refactor:
shared mount")]

Signed-off-by: Shuai Xue <[email protected]>
Signed-off-by: Liu, Bo <[email protected]>
Signed-off-by: baijia <[email protected]>
Signed-off-by: Peng Tao <[email protected]>
Signed-off-by: 慕陶 <[email protected]>

README.md

@@ -17,6 +17,7 @@ specification. Its key features include:
 Currently the repository includes following tools:
 
 * A `nydusify` tool to convert an OCI format container image into a nydus format container image
+* A `containerd-nydus-grpc` daemon to serve as containerd remote snapshotter and setup container rootfs with nydus
 * A `nydus-image` tool to convert an unpacked container image into a nydus format image
 * A `nydusd` daemon to parse a nydus format image and expose a FUSE mountpoint for containers to access
 
@@ -37,6 +38,10 @@ Build Nydus image from directory source: [Nydus Image Builder](./docs/nydus-imag
 
 Convert OCI image to Nydus image: [Nydusify](./docs/nydusify.md).
 
+## Build Nydus Snapshotter
+
+Build and run Nydus snapshotter: [Nydus Snapshotter](./contrib/nydus-snapshotter/README.md)
+
 ## Run Nydusd Daemon
 
 Run `nydusd` Daemon to serve Nydus image: [Nydusd](./docs/nydusd.md).

contrib/nydus-snapshotter/.gitignore

@@ -0,0 +1 @@
+bin/

contrib/nydus-snapshotter/BUILDING.md

@@ -0,0 +1,29 @@
+# Build nydus-snapshotter from source
+
+This doc includes:
+
+* [Build requirements] (#build-requirements)
+* [Build nydus snapshotter] (#build-nydus-snapshotter)
+* [Build nydus snapshotter image] (#build-nydus-snapshotter-image)
+
+## Build requirements
+
+To build the `nydus snapshotter` daemon, the following build system dependencies are required:
+
+* Go 1.13.x or above except 1.14.x
+
+## Build nydus snapshotter
+
+nydus snaphotter will be built under bin directory
+
+```bash
+make build
+```
+
+## Build nydus snapshotter image 
+
+build nydusd binary first and put nydusd binary under ./build/bin/
+
+```bash
+make build-image
+```

contrib/nydus-snapshotter/Makefile

@@ -0,0 +1,31 @@
+all:clear build
+
+VERSION=$(shell git rev-parse --verify HEAD --short=7)
+BRANCH=$(shell git rev-parse --abbrev-ref HEAD)
+PACKAGES ?= $(shell go list ./... | grep -v /vendor/)
+
+.PHONY: build
+build:
+	GOOS=linux go build -ldflags="-s -w -X 'main.Version=${VERSION}'" -v -o bin/containerd-nydus-grpc ./cmd/containerd-nydus-grpc
+
+.PHONY: clear
+clear:
+	rm -f bin/*
+	rm -rf _out
+
+.PHONY: vet
+vet:
+	go vet $(PACKAGES)
+
+.PHONY: test
+test: vet
+	go test -v -mod vendor -cover ${PACKAGES}
+
+.PHONY: cover
+cover: vet
+	@go test -coverprofile=_out/cover.out ${PACKAGES}
+	@go tool cover -func=_out/cover.out
+	@rm -f cover.out
+
+build-image:
+	docker build --build-arg VERSION=${VERSION} -t nydus-snapshotter:${VERSION} -f build/Dockerfile.release .

contrib/nydus-snapshotter/README.md

@@ -0,0 +1,159 @@
+# Nydus Snapshotter
+
+## Start Containerd
+
+For using nydus snapshotter with containerd, you need to install containerd beyond version 1.4.0, please refer to this [guide](https://github.com/containerd/containerd/blob/master/BUILDING.md) for more details. To add the nydus snapshotter plugin, add the plugin to containerd's config file (by default at /etc/containerd/config.toml).
+
+```toml
+# Plug nydus snapshotter into containerd
+# Containerd recognizes nydus snapshotter through specified socket address.
+# The specified address below is the default which nydus snapshotter listen to.
+[proxy_plugins]
+  [proxy_plugins.nydus]
+    type = "snapshot"
+    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
+
+# Use nydus as default snapshot through CRI
+[plugins."io.containerd.grpc.v1.cri".containerd]
+   snapshotter = "nydus"
+```
+
+Then you can start containerd in one terminal with following command.
+
+```bash
+$ /path/to/containerd --config /etc/containerd/config.toml
+```
+
+## Setting up the Nydus snapshotter
+
+### Generate Nydus config
+
+You can configure nydus snapshotter with custom configurations. The config file must be formatted with json and can be passed to nydus snapshotter with `--config-path` option. Your configuration file should look like below.
+
+```json
+{
+  "device": {
+    "backend": {
+      "type": "registry",
+      "config": {
+        "scheme": "https",
+        "timeout": 5,
+        "connect_timeout": 5,
+        "retry_limit": 0
+      }
+    },
+    "cache": {
+      "type": "blobcache",
+      "config": {
+        "work_dir": "/tmp/cache"
+      }
+    }
+  },
+  "mode": "direct",
+  "digest_validate": true,
+  "iostats_files": true,
+  "enable_xattr": true,
+  "fs_prefetch": {
+    "enable": true,
+    "threads_count": 10
+  }
+}
+```
+
+### Start Nydus snapshotter
+
+Nydus snapshotter is implemented as a [pproxy plugin](https://github.com/containerd/containerd/blob/04985039cede6aafbb7dfb3206c9c4d04e2f924d/PLUGINS.md#proxy-plugins) daemon (`containerd-nydus-grpc`) for containerd. You can start the daemon as following
+
+```bash
+# nydusd-path is the path of nydusd binary, you need to compile the binary first
+# address is the socket address that you configured in containerd config file
+# root is the path of nydus snapshotter
+# config-path is the path of your nydus configuration file you just generated
+
+$ ./containerd-nydus-grpc \
+  --nydusd-path /bin/nydusd \
+  --config-path /etc/nydus/config.json \
+  --root /var/lib/containerd/io.containerd.snapshotter.v1.nydus \
+  --address /var/run/containerd-nydus/containerd-nydus-grpc.sock 
+```
+
+### Check nydus snapshotter
+
+There is a default cli named `ctr` based on the GRPC api for containerd. This cli will allow you to create and manage containers run with containerd. And you can check if nydus snapshotter has started successfully by running the following commands:
+
+```bash
+$ ctr -a /run/containerd/containerd.sock plugin ls | grep nydus
+```
+
+## Using nydus snapshotter
+
+### Download crictl tools
+
+crictl is a tool to help developers debug their runtime without needing to set up Kubernetes components. `crictl` can be downloaded from cri-tools [release page](https://github.com/kubernetes-sigs/cri-tools/releases):
+
+```bash
+VERSION="v1.17.0"
+wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
+sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
+rm -f crictl-$VERSION-linux-amd64.tar.gz
+```
+
+### Create crictl config
+
+The runtime endpoint can be set in the config file. Please refer to [crictl](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md) document for more details.
+
+``` bash
+$ cat crictl.yaml
+runtime-endpoint: unix:///run/containerd/containerd.sock
+image-endpoint: unix:///run/containerd/containerd.sock
+timeout: 10
+debug: true
+```
+
+### Have fun with nydus
+
+You can create a container in the pod sandbox with config file.
+
+```bash
+$ cat pod-config.yaml
+metadata:
+  attempt: 1
+  name: nydus-sandbox
+  namespace: default
+  uid: hdishd83djaidwnduwk28bcsb
+log_directory: /tmp
+linux:
+  security_context:
+    namespace_options:
+      network: 2
+annotations:
+  "io.containerd.osfeature": "nydus.remoteimage.v1"
+
+$ cat container-config.yaml
+metadata:
+  name: nydus-container
+image:
+  image: <nydus-image>
+command:
+- /bin/sleep
+args:
+- 600
+log_path: container.1.log
+
+#auth is base64 of registry username:password
+$ crictl --config ./crictl.yaml run \
+ --auth <base64 of registry auth> \
+ ./container-config.yaml ./podsandbox-config.yaml
+```
+
+List and check running nydus container.
+
+```bash
+$ crictl --config ./crictl.yaml ps
+```
+
+Attach into nydus container.
+
+```bash
+$ crictl --config ./crictl.yaml exec -it <containerID> bash
+```

contrib/nydus-snapshotter/build/Dockerfile.release

@@ -0,0 +1,15 @@
+FROM golang:1.14 as builder
+ARG VERSION
+
+WORKDIR /src
+
+COPY . /src
+
+RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -mod vendor -a --ldflags="-s -w -X 'main.Version=${VERSION}'" -o /bin/nydus-snapshotter ./cmd/containerd-nydus-grpc
+
+FROM centos:7
+
+COPY --from=builder /src/build/bin/* /bin/
+COPY --from=builder /bin/nydus-snapshotter /bin/nydus-snapshotter
+
+ENTRYPOINT ["/bin/nydus-snapshotter"]

contrib/nydus-snapshotter/cmd/containerd-nydus-grpc/app/snapshotter/main.go

@@ -0,0 +1,57 @@
+/*
+ * Copyright (c) 2020. Ant Financial. All rights reserved.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package snapshotter
+
+import (
+	"context"
+
+	"github.com/pkg/errors"
+
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/pkg/filesystem/nydus"
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/pkg/filesystem/stargz"
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/pkg/signature"
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/pkg/utils/signals"
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/snapshot"
+)
+
+func Start(ctx context.Context, cfg Config) error {
+	verifier, err := signature.NewVerifier(cfg.PublicKeyFile, cfg.ValidateSignature)
+	if err != nil {
+		return errors.Wrap(err, "failed to initialize verifier")
+	}
+	fs, err := nydus.NewFileSystem(
+		nydus.WithNydusdBinaryPath(cfg.NydusdBinaryPath),
+		nydus.WithMeta(cfg.RootDir),
+		nydus.WithDaemonConfig(cfg.DaemonCfg),
+		nydus.WithVPCRegistry(cfg.ConvertVpcRegistry),
+		nydus.WithVerifier(verifier),
+	)
+	if err != nil {
+		return errors.Wrap(err, "failed to initialize nydus filesystem")
+	}
+
+	stargzFs, err := stargz.NewFileSystem(
+		stargz.WithMeta(cfg.RootDir),
+		stargz.WithNydusdBinaryPath(cfg.NydusdBinaryPath),
+		stargz.WithNydusImageBinaryPath(cfg.NydusImageBinaryPath),
+		stargz.WithDaemonConfig(cfg.DaemonCfg),
+	)
+	if err != nil {
+		return errors.Wrap(err, "failed to initialize stargz filesystem")
+	}
+
+	rs, err := snapshot.NewSnapshotter(ctx, cfg.RootDir, cfg.NydusdBinaryPath, fs, stargzFs, cfg.SharedDaemon, snapshot.AsynchronousRemove)
+	if err != nil {
+		return errors.Wrap(err, "failed to initialize snapshotter")
+	}
+
+	stopSignal := signals.SetupSignalHandler()
+	opt := snapshot.ServeOptions{
+		ListeningSocketPath: cfg.Address,
+	}
+	return snapshot.Serve(ctx, rs, opt, stopSignal)
+}

contrib/nydus-snapshotter/cmd/containerd-nydus-grpc/app/snapshotter/testdata/happypath/config.json

@@ -0,0 +1,29 @@
+{
+  "device": {
+    "backend": {
+      "type": "registry",
+      "config": {
+        "scheme": "https",
+        "auth": "",
+        "timeout": 5,
+        "connect_timeout": 5,
+        "retry_limit": 0
+      }
+    },
+    "cache": {
+      "type": "blobcache",
+      "config": {
+        "work_dir": "/tmp/nydus-rs/cache"
+      }
+    }
+  },
+  "mode": "direct",
+  "digest_validate": true,
+  "iostats_files": true,
+  "enable_xattr": true,
+  "fs_prefetch": {
+    "enable": true,
+    "threads_count": 10,
+    "merging_size": 131072
+  }
+}

contrib/nydus-snapshotter/cmd/containerd-nydus-grpc/app/snapshotter/validate.go

@@ -0,0 +1,51 @@
+/*
+ * Copyright (c) 2020. Ant Financial. All rights reserved.
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+
+package snapshotter
+
+import (
+	"os"
+
+	"github.com/pkg/errors"
+
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/cmd/containerd-nydus-grpc/pkg/command"
+	"gitlab.alipay-inc.com/antsys/nydus-snapshotter/pkg/filesystem/nydus"
+)
+
+type Config struct {
+	Address              string
+	ConvertVpcRegistry   bool
+	DaemonCfg            nydus.DaemonConfig
+	PublicKeyFile        string
+	RootDir              string
+	ValidateSignature    bool
+	NydusdBinaryPath     string
+	NydusImageBinaryPath string
+	SharedDaemon         bool
+}
+
+func Validate(args *command.Args, cfg *Config) error {
+	var daemonCfg nydus.DaemonConfig
+	if err := nydus.LoadConfig(args.ConfigPath, &daemonCfg); err != nil {
+		return errors.Wrapf(err, "failed to load config file %q", args.ConfigPath)
+	}
+
+	if args.ValidateSignature && args.PublicKeyFile != "" {
+		if _, err := os.Stat(args.PublicKeyFile); err != nil {
+			return errors.Wrapf(err, "failed to find publicKey file %q", args.PublicKeyFile)
+		}
+	}
+	cfg.DaemonCfg = daemonCfg
+	cfg.RootDir = args.RootDir
+	cfg.ValidateSignature = args.ValidateSignature
+	cfg.PublicKeyFile = args.PublicKeyFile
+	cfg.ConvertVpcRegistry = args.ConvertVpcRegistry
+	cfg.Address = args.Address
+	cfg.NydusdBinaryPath = args.NydusdBinaryPath
+	cfg.NydusImageBinaryPath = args.NydusImageBinaryPath
+	cfg.SharedDaemon = args.SharedDaemon
+	return nil
+}