Add dataplane performance test (#1140)

* Add dataplane performance test
nginx · Oct 17, 2023 · 6707e5a · 6707e5a
1 parent 6e1c9d8
commit 6707e5a
Show file tree

Hide file tree

Showing 7 changed files with 332 additions and 0 deletions.
diff --git a/tests/dataplane-performance/manifests/cafe-routes.yaml b/tests/dataplane-performance/manifests/cafe-routes.yaml
@@ -0,0 +1,60 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: coffee
+spec:
+  parentRefs:
+  - name: cafe
+  hostnames:
+  - cafe.example.com
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /latte
+    backendRefs:
+    - name: coffee-svc
+      port: 80
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /coffee
+      headers:
+      - name: version
+        value: v2
+    - path:
+        type: PathPrefix
+        value: /coffee
+      queryParams:
+      - name: TEST
+        value: v2
+    backendRefs:
+    - name: coffee-svc
+      port: 80
+---
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+  name: tea
+spec:
+  parentRefs:
+  - name: cafe
+  hostnames:
+  - cafe.example.com
+  rules:
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /tea
+      method: POST
+    backendRefs:
+    - name: coffee-svc
+      port: 80
+  - matches:
+    - path:
+        type: PathPrefix
+        value: /tea
+      method: GET
+    backendRefs:
+    - name: coffee-svc
+      port: 80
diff --git a/tests/dataplane-performance/manifests/coffee.yaml b/tests/dataplane-performance/manifests/coffee.yaml
@@ -0,0 +1,32 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: coffee
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: coffee
+  template:
+    metadata:
+      labels:
+        app: coffee
+    spec:
+      containers:
+      - name: coffee
+        image: nginxdemos/nginx-hello:plain-text
+        ports:
+        - containerPort: 8080
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: coffee-svc
+spec:
+  ports:
+  - port: 80
+    targetPort: 8080
+    protocol: TCP
+    name: http
+  selector:
+    app: coffee
diff --git a/tests/dataplane-performance/manifests/gateway.yaml b/tests/dataplane-performance/manifests/gateway.yaml
@@ -0,0 +1,10 @@
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: Gateway
+metadata:
+  name: cafe
+spec:
+  gatewayClassName: nginx
+  listeners:
+  - name: http
+    port: 80
+    protocol: HTTP
diff --git a/tests/dataplane-performance/results/1.0.0/1.0.0.md b/tests/dataplane-performance/results/1.0.0/1.0.0.md
@@ -0,0 +1,123 @@
+# Results
+
+## Test environment
+
+GKE cluster:
+
+- Node count: 3
+- Instance Type: e2-medium
+- k8s version: 1.27.4-gke.900
+- Zone: europe-west2-b
+- Total vCPUs: 6
+- Total RAM: 12GB
+- Max pods per node: 110
+
+Test VM:
+
+- Instance Type: e2-medium
+- Zone: europe-west2-b
+- vCPUS: 2
+- RAM: 4GB
+
+NGF deployment:
+
+- NGF version: edge - git commit a41e9e46d72788bceea8ba44e01d4062afec75fd
+- NGINX Version: 1.25.2
+
+## Test1: Running latte path based routing
+
+```console
+Running 30s test @ http://cafe.example.com/latte
+  2 threads and 10 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency     6.29ms    7.85ms 182.06ms   88.67%
+    Req/Sec     1.11k   588.11     2.31k    59.83%
+  Latency Distribution
+     50%    3.44ms
+     75%    7.09ms
+     90%   15.33ms
+     99%   31.86ms
+  66017 requests in 30.02s, 23.42MB read
+Requests/sec:   2199.43
+Transfer/sec:    799.00KB
+```
+
+## Test2: Running coffee header based routing
+
+```console
+Running 30s test @ http://cafe.example.com/coffee
+  2 threads and 10 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency    31.95ms   37.10ms 243.16ms   89.07%
+    Req/Sec   224.25    238.65     1.83k    90.32%
+  Latency Distribution
+     50%   14.31ms
+     75%   43.74ms
+     90%   76.70ms
+     99%  171.95ms
+  12913 requests in 30.03s, 4.59MB read
+Requests/sec:    430.05
+Transfer/sec:    156.65KB
+```
+
+## Test3: Running coffee query based routing
+
+```console
+Running 30s test @ http://cafe.example.com/coffee?TEST=v2
+  2 threads and 10 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency    29.26ms   43.85ms 457.05ms   82.26%
+    Req/Sec   632.30    464.79     2.07k    60.43%
+  Latency Distribution
+     50%    4.45ms
+     75%   45.84ms
+     90%  103.14ms
+     99%  145.53ms
+  37324 requests in 30.02s, 13.56MB read
+Requests/sec:   1243.13
+Transfer/sec:    462.53KB
+```
+
+## Test4: Running tea GET method based routing
+
+```console
+Running 30s test @ http://cafe.example.com/tea
+  2 threads and 10 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency    27.19ms   39.40ms 182.48ms   81.49%
+    Req/Sec   706.49    416.90     1.85k    65.42%
+  Latency Distribution
+     50%    4.06ms
+     75%   42.24ms
+     90%   99.21ms
+     99%  133.34ms
+  41265 requests in 30.04s, 14.56MB read
+Requests/sec:   1373.59
+Transfer/sec:    496.31KB
+```
+
+## Test5: Running tea POST method based routing
+
+```console
+Running 30s test @ http://cafe.example.com/tea
+  2 threads and 10 connections
+  Thread Stats   Avg      Stdev     Max   +/- Stdev
+    Latency    27.32ms   39.49ms 208.66ms   81.40%
+    Req/Sec   685.47    491.33     2.08k    58.43%
+  Latency Distribution
+     50%    4.19ms
+     75%   42.40ms
+     90%   99.38ms
+     99%  133.25ms
+  40723 requests in 30.03s, 14.37MB read
+Requests/sec:   1356.10
+Transfer/sec:    489.99KB
+```
+
+## Observations
+
+- Path based routing is the most performant of the routing methods
+- The njs based methods are less performant:
+  - Header based routing is the least performant of the routing methods, by a factor of roughly 5 compared to the path
+    based routing.
+  - The query and method based routing methods are approximately 2/3s as performant as the path based routing.
diff --git a/tests/dataplane-performance/scripts/post.lua b/tests/dataplane-performance/scripts/post.lua
@@ -0,0 +1 @@
+wrk.method = "POST"
diff --git a/tests/dataplane-performance/scripts/wrk-latency.sh b/tests/dataplane-performance/scripts/wrk-latency.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+echo -e "# Results\n"
+
+echo -e "## Test1: Running latte path based routing\n"
+echo -e '```console'
+wrk -t2 -c10 -d30 http://cafe.example.com/latte --latency
+echo -e '```'
+echo -e "\n## Test2: Running coffee header based routing\n"
+echo -e '```console'
+wrk -t2 -c10 -d30 http://cafe.example.com/coffee -H "version: v2" --latency
+echo -e '```'
+echo -e "\n## Test3: Running coffee query based routing\n"
+echo -e '```console'
+wrk -t2 -c10 -d30 http://cafe.example.com/coffee?TEST=v2 --latency
+echo -e '```'
+echo -e "\n## Test4: Running tea GET method based routing\n"
+echo -e '```console'
+wrk -t2 -c10 -d30 http://cafe.example.com/tea --latency
+echo -e '```'
+echo -e "\n## Test5: Running tea POST method based routing\n"
+echo -e '```console'
+wrk -t2 -c10 -d30 http://cafe.example.com/tea -s ${SCRIPT_DIR}/post.lua --latency
+echo -e '```'
diff --git a/tests/dataplane-performance/setup.md b/tests/dataplane-performance/setup.md
@@ -0,0 +1,80 @@
+# Dataplane Performance Testing
+
+## Goals
+
+- To capture the average latency of requests being proxied through NGINX using a variety of routing rules, so that we
+  can see if different routing rules provide different results, and so that we can know if any future work has an impact
+  on data plane performance.
+- A route is created and tested for each routing method below:
+  - path based routing
+  - header based routing
+  - query param based routing
+  - method based routing
+
+## Test Environment
+
+- A Kubernetes cluster with 3 nodes on GKE
+  - Node: e2-medium (2 vCPU, 4GB memory)
+- Tester VM on Google Cloud:
+  - Instance Type: e2-medium (2 vCPU, 4GB memory)
+  - Configuration:
+    - Debian
+    - Install packages: wrk
+  - Location - same zone as the Kubernetes cluster.
+
+## Setup
+
+1. Create cloud cluster
+2. Deploy CRDs:
+
+   ```bash
+   kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v0.8.1/standard-install.yaml
+   ```
+
+3. Deploy NGF from edge using Helm install, and expose using an internal LoadBalancer service:
+
+   ```console
+   helm install my-release oci://ghcr.io/nginxinc/charts/nginx-gateway-fabric  --version 0.0.0-edge \
+      --set service.annotations.'networking\.gke\.io\/load-balancer-type'="Internal" --create-namespace \
+      --wait -n nginx-gateway
+   ```
+
+## Tests
+
+1. First create the test resources. The manifests provided will deploy a single application and service, and two
+   HTTPRoutes with matches for path based, header based, query based, and method based routing.
+
+   ```console
+   kubectl apply -f manifests/gateway.yaml
+   kubectl apply -f manifests/coffee.yaml
+   kubectl apply -f manifests/cafe-routes.yaml
+   ```
+
+2. Get the external IP of the nginx-gateway service and add an entry in /etc/hosts for `<GW_IP> cafe.example.com`.
+
+   ```console
+   kubectl get service my-release-nginx-gateway-fabric \
+   --namespace nginx-gateway \
+   --output jsonpath='{.status.loadBalancer.ingress[0].ip}'
+   ```
+
+3. Copy the scripts in the `scripts/` directory to the test VM. Run tests using `wrk` (change the file output name to
+   version under test). The tests use `wrk` to send requests to coffee application using each of the configured routing
+   rules, capturing the number of requests and the average latency.
+
+   ```console
+   bash wrk-latency.sh > 1.0.0.md
+   ```
+
+4. Analyse the results and check for anomolies. Copy the results file from the test VM into the `results/` directory.
+   Append any findings or observations to the generated results document. Add test environment information to the
+   generated document.
+
+5. Cleanup the deployed resources
+
+   ```console
+   kubectl delete -f manifests/cafe-routes.yaml
+   kubectl delete -f manifests/coffee.yaml
+   kubectl delete -f manifests/gateway.yaml
+   helm uninstall my-release -n nginx-gateway
+   ```