Merge pull request spidernet-io#4196 from ty-dc/fix/ipam-image

optimize ipam concepts
ty-dc · Oct 25, 2024 · a810abd · a810abd
2 parents 709bfba + c1424a2
commit a810abd
Show file tree

Hide file tree

Showing 19 changed files with 380 additions and 258 deletions.
diff --git a/docs/concepts/ipam-des-zh_CN.md b/docs/concepts/ipam-des-zh_CN.md
diff --git a/docs/concepts/ipam-des.md b/docs/concepts/ipam-des.md
diff --git a/docs/concepts/ipam-performance-zh_CN.md b/docs/concepts/ipam-performance-zh_CN.md
@@ -8,16 +8,16 @@
 
 为什么要做 underlay IPAM CNI 插件的性能测试？
 
-1. IPAM 分配 IP 地址的速度，很大程度上的决定了应用发布的速度。
+1. IPAM 分配 IP 地址的速度，很大程度上决定了应用发布的速度。
 2. 大规模的 Kubernetes 集群在故障恢复时，underlay IPAM 往往会成为性能瓶颈。
-3. underlay 网络下，私有的 IPv4 地址有限。在有限的 IP 地址范围内，并发的创建 Pod 会涉及 IP 地址的抢占与冲突，能否快速的调节好有限的 IP 地址资源具有挑战。
+3. underlay 网络下，私有的 IPv4 地址有限。在有限的 IP 地址范围内，并发创建 Pod 会涉及 IP 地址的抢占与冲突，能否快速调节好有限的 IP 地址资源具有挑战。
 
 ## 环境
 
 - Kubernetes: `v1.26.7`
-- container runtime: `containerd v1.7.2`
+- Container runtime: `containerd v1.7.2`
 - OS: `Ubuntu 22.04 LTS`
-- kernel: `5.15.0-33-generic`
+- Kernel: `5.15.0-33-generic`
 
 | Node     | Role                  | CPU | Memory |
 | -------- | --------------------- | --- | ------ |
@@ -36,19 +36,19 @@
 
 本次测试基于 `0.3.1` 版本的 [CNI Specification](https://www.cni.dev/docs/spec/)，以 [macvlan](https://www.cni.dev/plugins/current/main/macvlan/) 搭配 Spiderpool 作为测试方案，并选择了开源社区中其它几种常见的网络方案作为对比：
 
-| 测试对象                       |      版本     |
-| ----------------------------- | ------------ |
-| Spiderpool based on macvlan   | `v0.8.0`     |
-| Whereabouts based on macvlan  | `v0.6.2`     |
-| Kube-OVN                      | `v1.12.2`    |
-| Cilium                        | `v1.14.3`    |
-| Calico                        | `v3.26.3`    |
+| 测试对象                       | 版本       |
+| ----------------------------- | ---------- |
+| Spiderpool based on macvlan   | `v0.8.0`   |
+| Whereabouts based on macvlan  | `v0.6.2`   |
+| Kube-OVN                      | `v1.12.2`  |
+| Cilium                        | `v1.14.3`  |
+| Calico                        | `v3.26.3`  |
 
 ## 方案
 
 测试思路主要是：
 
-1. Underlay IP 资源有限，IP 的泄露和分配重复，容易造成干扰 ，因此 IP 分配的准确性非常重要。
+1. Underlay IP 资源有限，IP 的泄露和分配重复容易造成干扰，因此 IP 分配的准确性非常重要。
 2. 在大量 Pod 启动时竞争分配 IP，IPAM 的分配算法要高效，才能保障 Pod 快速发布成功。
 
 因此，设计了 IP 资源和 Pod 资源数量相同的极限测试，计时 Pod 从创建到 Running 的时间，来变相测试 IPAM 的精确性和健壮性。测试的条件如下：
@@ -58,31 +58,31 @@
 
 ## 测试结果
 
-如下展示了 IPAM 性能测试结果，其中，包含了 `限制 IP 与 Pod 等量` 和 `不限制 IP 数量` 两种场景，来分别测试每个 CNI ，而 Calico 和 Cilium 等是基于 IP block 预分配机制分配 IP，因此没法相对 "公平" 的进行 `限制 IP 与 Pod 等量` 测试，只进行 `不限制 IP 数量` 场景测试。
+如下展示了 IPAM 性能测试结果，其中，包含了 `限制 IP 与 Pod 等量` 和 `不限制 IP 数量` 两种场景，来分别测试每个 CNI。Calico 和 Cilium 等是基于 IP block 预分配机制分配 IP，因此没法相对 "公平" 地进行 `限制 IP 与 Pod 等量` 测试，只进行 `不限制 IP 数量` 场景测试。
 
-  |       测试对象                |   限制 IP 与 Pod 等量    |   不限制 IP 数量  |
-  | ---------------------------  | ---------------------- | --------------- |
-  | Spiderpool based on macvlan  |          207s          |       182       |
-  | Whereabouts based on macvlan |          失败           |       2529s    |
-  | Kube-OVN                     |          405s          |       343s      |
-  | Cilium                       |           NA           |       215s      |
-  | Calico                       |           NA           |       322s      |
+| 测试对象                       | 限制 IP 与 Pod 等量 | 不限制 IP 数量 |
+| ----------------------------- | ------------------ | ------------- |
+| Spiderpool based on macvlan   | 207s               | 182s          |
+| Whereabouts based on macvlan  | 失败               | 2529s         |
+| Kube-OVN                      | 405s               | 343s          |
+| Cilium                        | NA                 | 215s          |
+| Calico                        | NA                 | 322s          |
 
 ## 分析
 
 ![performance](../images/ipam-performance.png)
 
-Spiderpool 的 IPAM 分配原理，是整个集群节点的所有 Pod 都从同一个 CIDR 中分配 IP，所以 IP 分配和释放需要面临激烈的竞争，IP 分配性能的挑战会更大；Whereabouts 和 Calico 、Cilium的 IPAM 分配原理，是每个节点都有一个小的 IP 集合，所以 IP 分配的竞争比较小，IP 分配性能的挑战会小。但从上述实验数据上看，虽然 Spdierpool 的 IPAM 原理是 "吃亏" 的，但是分配 IP 的性能却是很好的。
+Spiderpool 的 IPAM 分配原理，是整个集群节点的所有 Pod 都从同一个 CIDR 中分配 IP，所以 IP 分配和释放需要面临激烈的竞争，IP 分配性能的挑战会更大；Whereabouts 和 Calico、Cilium 的 IPAM 分配原理，是每个节点都有一个小的 IP 集合，所以 IP 分配的竞争比较小，IP 分配性能的挑战会小。但从上述实验数据上看，虽然 Spiderpool 的 IPAM 原理是 "吃亏" 的，但是分配 IP 的性能却是很好的。
 
 - 在测试过程中，遇到如下现象：
 
-  Whereabouts based on macvlan：在`限制 IP 与 Pod 等量`场景下，在 300s 内 261 个 Pod 以较为匀速的状态达到了 `Running` 状态，在 1080s 时，分配 768 个 IP 地址。自此之后的 Pod 增长速率大幅降低，在 2280s 时达到 845 个，后续 Whereabouts 就基本不工作了，耗时类比于正无穷。由于 IP 地址数量与 Pod 数等量，如果 IPAM 组件未能正确的回收 IP，新 Pod 将因为缺少 IP 资源，且无法获取到可用的 IP，从而无法启动。并且观察到在启动失败的 Pod 中，出现了如下的一些错误：
+    Whereabouts based on macvlan：在 `限制 IP 与 Pod 等量` 场景下，在 300s 内 261 个 Pod 以较为匀速的状态达到了 `Running` 状态，在 1080s 时，分配 768 个 IP 地址。自此之后的 Pod 增长速率大幅降低，在 2280s 时达到 845 个，后续 Whereabouts 就基本不工作了，耗时类比于正无穷。由于 IP 地址数量与 Pod 数等量，如果 IPAM 组件未能正确回收 IP，新 Pod 将因为缺少 IP 资源，且无法获取到可用的 IP，从而无法启动。并且观察到在启动失败的 Pod 中，出现了如下的一些错误：
 
-  ```bash
-  [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader
+    ```bash
+    [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader
 
-  name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175"
-  ```
+    name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175"
+    ```
 
 ## 总结
 

diff --git a/docs/concepts/ipam-performance.md b/docs/concepts/ipam-performance.md
@@ -2,7 +2,7 @@
 
 **English** | [**简体中文**](./ipam-performance-zh_CN.md)
 
-*[Spiderpool](https://github.com/spidernet-io/spiderpool) is an underlay networking solution that provides rich IPAM and CNI integration capabilities, this article will compare it with the mainstream IPAM CNI plug-ins (e.g. [Whereabouts](https://github.com/spidernet-io/spiderpool), [Kube-OVN](https://github.com/spidernet-io/spiderpool)) and the widely used IPAM CNI plug-ins that are running in underlay scenarios. This article will compare it with the mainstream IPAM CNI plug-ins running in underlay scenarios (e.g., [Whereabouts](https://github.com/k8snetworkplumbingwg/whereabouts), [Kube-OVN](https://github.com/kubeovn/kube-ovn)) and the widely-used overlay IPAM CNI plugins [calico-ipam](https://github.com/projectcalico/calico)、[cilium](https://github.com/cilium/cilium) in `1000 Pod` scenarios.*
+*[Spiderpool](https://github.com/spidernet-io/spiderpool) is an underlay networking solution that provides rich IPAM and CNI integration capabilities. This article will compare it with the mainstream IPAM CNI plug-ins (e.g., [Whereabouts](https://github.com/k8snetworkplumbingwg/whereabouts), [Kube-OVN](https://github.com/kubeovn/kube-ovn)) and the widely-used overlay IPAM CNI plugins ([calico-ipam](https://github.com/projectcalico/calico), [cilium](https://github.com/cilium/cilium)) in `1000 Pod` scenarios.*
 
 ## Background
 
@@ -15,9 +15,9 @@ Why do we need to do performance testing on the underlay IPAM CNI plugin?
 ## ENV
 
 - Kubernetes: `v1.26.7`
-- container runtime: `containerd v1.7.2`
+- Container runtime: `containerd v1.7.2`
 - OS: `Ubuntu 22.04 LTS`
-- kernel: `5.15.0-33-generic`
+- Kernel: `5.15.0-33-generic`
 
 | Node     | Role                  | CPU | Memory |
 | -------- | --------------------- | --- | ------ |
@@ -36,13 +36,13 @@ Why do we need to do performance testing on the underlay IPAM CNI plugin?
 
 This test is based on the `0.3.1` version of [CNI Specification](https://www.cni.dev/docs/spec/), with [macvlan](https://www.cni.dev/plugins/current/main/macvlan/) and Spiderpool as the test object, and selected several other common network solutions in the open source community as a comparison:
 
-| test object                   |    version   |
-| ----------------------------- | ------------ |
-| Spiderpool based on macvlan   | `v0.8.0`     |
-| Whereabouts based on macvlan  | `v0.6.2`     |
-| Kube-OVN                      | `v1.12.2`    |
-| Cilium                        | `v1.14.3`    |
-| Calico                        | `v3.26.3`    |                                                          |
+| Test Object                   | Version   |
+| ----------------------------- | --------- |
+| Spiderpool based on macvlan   | `v0.8.0`  |
+| Whereabouts based on macvlan  | `v0.6.2`  |
+| Kube-OVN                      | `v1.12.2` |
+| Cilium                        | `v1.14.3` |
+| Calico                        | `v3.26.3` |
 
 ## Plan
 
@@ -58,31 +58,31 @@ Therefore, we designed a limit test with the same number of IP resources and Pod
 
 ## Result
 
-The following shows the results of the IPAM performance test, which includes two scenarios, `The number of IPs is equal to the number of Pods` and `IP sufficient`, to test each CNI, whereas Calico and Cilium, for example, are based on the IP block pre-allocation mechanism to allocate IPs, and therefore can't perform the `The number of IPs is equal to the number of Pods` test in a relatively `fair` way, and only perform the `IP sufficient` scenario. We can only test `unlimited IPs` scenarios.
+The following shows the results of the IPAM performance test, which includes two scenarios, `The number of IPs is equal to the number of Pods` and `IP sufficient`, to test each CNI. Calico and Cilium, for example, are based on the IP block pre-allocation mechanism to allocate IPs, and therefore can't perform the `The number of IPs is equal to the number of Pods` test in a relatively `fair` way, and only perform the `IP sufficient` scenario. We can only test `unlimited IPs` scenarios.
 
-  |       test object            | Limit IP to Pod Equivalents |  IP sufficient  |
-  | ---------------------------  | --------------------------- | --------------- |
-  | Spiderpool based on macvlan  |             207s            |       182       |
-  | Whereabouts based on macvlan |           failure           |       2529s     |
-  | Kube-OVN                     |             405s            |       343s      |
-  | Cilium                       |              NA             |       215s      |
-  | Calico                       |              NA             |       322s      |
+| Test Object                   | Limit IP to Pod Equivalents | IP Sufficient |
+| ----------------------------- | --------------------------- | ------------- |
+| Spiderpool based on macvlan   | 207s                        | 182s          |
+| Whereabouts based on macvlan  | Failure                     | 2529s         |
+| Kube-OVN                      | 405s                        | 343s          |
+| Cilium                        | NA                          | 215s          |
+| Calico                        | NA                          | 322s          |
 
-## analyze
+## Analysis
 
 ![performance](../images/ipam-performance.png)
 
-Spiderpool allocate IP addresses from the same CIDR range to all Pods in the whole cluster. Consequently, IP allocation and release face intense competition, presenting larger challenges in terms of IP allocation performance. By comparison, Whereabouts, Calico, and Cilium adopt an IPAM allocation principle where each node has a small IP address pool. This reduces the competition for IP allocation and mitigates the associated performance challenges. However, experimental data shows that despite Spiderpool's "lossy" IPAM principle, its IP allocation performance is actually quite good.
+Spiderpool allocates IP addresses from the same CIDR range to all Pods in the whole cluster. Consequently, IP allocation and release face intense competition, presenting larger challenges in terms of IP allocation performance. By comparison, Whereabouts, Calico, and Cilium adopt an IPAM allocation principle where each node has a small IP address pool. This reduces the competition for IP allocation and mitigates the associated performance challenges. However, experimental data shows that despite Spiderpool's "lossy" IPAM principle, its IP allocation performance is actually quite good.
 
 - During testing, the following phenomenon was encountered:
 
-  Whereabouts based on macvlan：We tested the combination of macvlan and Whereabouts in a scenario where the available number of IP addresses matches the number of Pods in a 1:1 ratio. Within 300 seconds, 261 Pods reached the "Running" state at a relatively steady pace. By the 1080-second mark, 768 IP addresses were allocated. Afterward, the growth rate of Pods significantly slowed down, reaching 845 Pods by 2280 seconds. Subsequently, Whereabouts essentially stopped working, resulting in a positively near-infinite amount of time needed for further allocation. In our testing scenario, where the number of IP addresses matches the number of Pods in a 1:1 ratio, if the IPAM component fails to properly reclaim IP addresses, new Pods will fail to start due to a lack of available IP resources. And observed some of the following errors in the Pod that failed to start:
+    Whereabouts based on macvlan: We tested the combination of macvlan and Whereabouts in a scenario where the available number of IP addresses matches the number of Pods in a 1:1 ratio. Within 300 seconds, 261 Pods reached the "Running" state at a relatively steady pace. By the 1080-second mark, 768 IP addresses were allocated. Afterward, the growth rate of Pods significantly slowed down, reaching 845 Pods by 2280 seconds. Subsequently, Whereabouts essentially stopped working, resulting in a positively near-infinite amount of time needed for further allocation. In our testing scenario, where the number of IP addresses matches the number of Pods in a 1:1 ratio, if the IPAM component fails to properly reclaim IP addresses, new Pods will fail to start due to a lack of available IP resources. We observed some of the following errors in the Pod that failed to start:
 
-  ```bash
-  [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader
+    ```bash
+    [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader
 
-  name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175"
-  ```
+    name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175"
+    ```
 
 ## Summary
 

diff --git a/docs/images/ipam-concepts.jpg b/docs/images/ipam-concepts.jpg