From ce7aab4400ca79088dcf7715fc52cfd736e0ed40 Mon Sep 17 00:00:00 2001 From: Cyclinder Date: Fri, 25 Oct 2024 16:31:58 +0800 Subject: [PATCH] optimize ipam concepts Signed-off-by: ty-dc --- docs/concepts/ipam-des-zh_CN.md | 113 +++++++------ docs/concepts/ipam-des.md | 192 ++++++++++------------- docs/concepts/ipam-performance-zh_CN.md | 52 +++--- docs/concepts/ipam-performance.md | 50 +++--- docs/images/ipam-concepts.jpg | Bin 0 -> 71444 bytes docs/usage/faq-zh_CN.md | 2 +- docs/usage/ipvlan_bandwidth-zh_CN.md | 1 - docs/usage/ipvlan_bandwidth.md | 2 +- docs/usage/kubevirt-zh_CN.md | 2 +- docs/usage/multi_cni_coexist-zh_CN.md | 4 +- docs/usage/multi_cni_coexist.md | 4 +- docs/usage/operator-zh_CN.md | 147 +++++++++++++++++ docs/usage/readme-zh_CN.md | 2 +- docs/usage/spider-multus-config-zh_CN.md | 3 +- docs/usage/spider-multus-config.md | 2 +- docs/usage/underlay_cni_service-zh_CN.md | 6 +- docs/usage/underlay_cni_service.md | 2 +- 17 files changed, 354 insertions(+), 230 deletions(-) create mode 100644 docs/images/ipam-concepts.jpg diff --git a/docs/concepts/ipam-des-zh_CN.md b/docs/concepts/ipam-des-zh_CN.md index 4ff4b4f135..18856b4f8e 100644 --- a/docs/concepts/ipam-des-zh_CN.md +++ b/docs/concepts/ipam-des-zh_CN.md @@ -4,39 +4,31 @@ ## Underlay 网络和 Overlay 网络的 IPAM -云原生网络中出现了两种技术类别："Overlay 网络方案" 和 "Underlay 网络方案"。 -云原生网络对于它们没有严格的定义，我们可以从很多 CNI 项目的实现原理中，简单抽象出这两种技术流派的特点，它们可以满足不同场景下的需求。 +云原生网络中出现了两种技术类别："Overlay 网络方案" 和 "Underlay 网络方案"。云原生网络对于它们没有严格的定义，我们可以从很多 CNI 项目的实现原理中，简单抽象出这两种技术流派的特点，它们可以满足不同场景下的需求。 Spiderpool 是为 Underlay 网络特点而设计，以下对两种方案进行比较，能够更好说明 Spiderpool 的特点和使用场景。 ### Overlay 网络方案 IPAM -本方案实现了 Pod 网络同宿主机网络的解耦，例如 [Calico](https://github.com/projectcalico/calico)、[Cilium](https://github.com/cilium/cilium) 等 CNI 插件， -这些插件多数使用了 vxlan 等隧道技术，搭建起一个 Overlay 网络平面，再借用 NAT 技术实现南北向的通信。 +本方案实现了 Pod 网络同宿主机网络的解耦，例如 [Calico](https://github.com/projectcalico/calico)、[Cilium](https://github.com/cilium/cilium) 等 CNI 插件，这些插件多数使用了 vxlan 等隧道技术，搭建起一个 Overlay 网络平面，再借用 NAT 技术实现南北向的通信。 这类技术流派的 IPAM 分配特点是： -1. Pod 子网中的 IP 地址按照节点进行了分割 +1. **Pod 子网中的 IP 地址按照节点进行了分割** - 以一个更小子网掩码长度为单位，把 Pod subnet 分割出更小的 IP block 集合，依据 IP 使用的用量情况，每个 node 都会获取到一个或者多个 IP block。 + 以一个更小子网掩码长度为单位，把 Pod subnet 分割出更小的 IP block 集合，依据 IP 使用的用量情况，每个 node 都会获取到一个或者多个 IP block。 - 这意味着两个特点：第一，每个 node 上的 IPAM 插件只需要在本地的 IP block 中分配和释放 IP 地址时，与其它 node 上的 IPAM 无 IP 分配冲突，IPAM 分配效率更高。 - 第二，某个具体的 IP 地址跟随 IP block 集合，会相对固定的一直在某个 node 上被分配，没法随同 Pod 一起被调度漂移。 + 这意味着两个特点：第一，每个 node 上的 IPAM 插件只需要在本地的 IP block 中分配和释放 IP 地址时，与其它 node 上的 IPAM 无 IP 分配冲突，IPAM 分配效率更高。第二，某个具体的 IP 地址跟随 IP block 集合，会相对固定的一直在某个 node 上被分配，没法随同 Pod 一起被调度漂移。 -2. IP 地址资源充沛 +2. **IP 地址资源充沛** - 只要 Pod 子网不与相关网络重叠，再能够合理利用 NAT 技术，Kubernetes 单个集群可以拥有充沛的 IP 地址资源。 - 因此，应用不会因为 IP 不够而启动失败，IPAM 组件面临的异常 IP 回收压力较小。 + 只要 Pod 子网不与相关网络重叠，再能够合理利用 NAT 技术，Kubernetes 单个集群可以拥有充沛的 IP 地址资源。因此，应用不会因为 IP 不够而启动失败，IPAM 组件面临的异常 IP 回收压力较小。 -3. 没有应用 "IP 地址固定"需求 +3. **没有应用 "IP 地址固定"需求** - 对于应用 IP 地址固定需求，有无状态应用和有状态应用的区别：对于 Deployment 这类无状态应用，因为 Pod 名称会随着 Pod 重启而变化， - 应用本身的业务逻辑也是无状态的，因此对于 "IP 地址固定" 的需求，只能让所有 Pod 副本固定在一个 IP 地址的集合内；对于 StatefulSet - 这类有状态应用，因为 Pod name 等信息都是固定的，应用本身的业务逻辑也是有状态的，因此对于 "IP 地址固定"需求，要实现单个 Pod 和具体 IP 地址的强绑定。 + 对于应用 IP 地址固定需求，有无状态应用和有状态应用的区别：对于 Deployment 这类无状态应用，因为 Pod 名称会随着 Pod 重启而变化，应用本身的业务逻辑也是无状态的，因此对于 "IP 地址固定" 的需求，只能让所有 Pod 副本固定在一个 IP 地址的集合内；对于 StatefulSet 这类有状态应用，因为 Pod name 等信息都是固定的，应用本身的业务逻辑也是有状态的，因此对于 "IP 地址固定"需求，要实现单个 Pod 和具体 IP 地址的强绑定。 - 在 "Overlay 网络方案"方案下，多是借助了 NAT 技术向集群外部暴露服务的入口和源地址，借助 DNS、clusterIP 等技术来实现集群东西向通信。 - 其次，IPAM 的 IP block 方式把 IP 相对固定到某个节点上，而不能保证应用副本的跟随调度。 - 因此，应用的 "IP 地址固定"能力无用武之地，当前社区的主流 CNI 多数不支持 "IP 地址固定"，或者支持方法较为简陋。 + 在 "Overlay 网络方案"方案下，多是借助了 NAT 技术向集群外部暴露服务的入口和源地址，借助 DNS、clusterIP 等技术来实现集群东西向通信。其次，IPAM 的 IP block 方式把 IP 相对固定到某个节点上，而不能保证应用副本的跟随调度。因此，应用的 "IP 地址固定"能力无用武之地，当前社区的主流 CNI 多数不支持 "IP 地址固定"，或者支持方法较为简陋。 这个方案的优点是，无论集群部署在什么样的底层网络环境上，CNI 插件的兼容性都非常好，且都能够为 Pod 提供子网独立、IP 地址资源充沛的网络。 @@ -46,41 +38,35 @@ Spiderpool 是为 Underlay 网络特点而设计，以下对两种方案进行 Underlay 网络方案的实施，有两种典型的场景：一种是集群部署实施在"传统网络"上；一种是集群部署在 IAAS 环境上，例如公有云。以下总结了"传统网络场景"的 IPAM 特点： -1. 单个 IP 地址应该能够在任一节点上被分配 +1. **单个 IP 地址应该能够在任一节点上被分配** - 这个需求有多方面的原因：随着数据中心的网络设备增加、多集群技术的发展，IPv4 地址资源稀缺，要求 IPAM 提高 IP 资源的使用效率； - 对于有 "IP 地址固定"需求的应用，其 Pod 副本可能会调度到集群的任意一个节点上，并且，在故障场景下还会发生节点间的漂移，要求 IP 地址一起漂移。 + 这个需求有多方面的原因：随着数据中心的网络设备增加、多集群技术的发展，IPv4 地址资源稀缺，要求 IPAM 提高 IP 资源的使用效率；对于有 "IP 地址固定"需求的应用，其 Pod 副本可能会调度到集群的任意一个节点上，并且，在故障场景下还会发生节点间的漂移，要求 IP 地址一起漂移。 - 因此，在集群中的任意一个节点上，一个 IP 地址应该具备能够被分配给 Pod 使用的可能。 + 因此，在集群中的任意一个节点上，一个 IP 地址应该具备能够被分配给 Pod 使用的可能。 -2. 同一应用的不同副本，能实现跨子网获取 IP 地址 +2. **同一应用的不同副本，能实现跨子网获取 IP 地址** - 例如，一个集群中，宿主机1的区域只能使用子网 172.20.1.0/24，而宿主机2的区域只能使用子网 172.20.2.0/24，在此背景下， - 当一个应用跨子网部署副本时，要求 IPAM 能够在不同的节点上，为同一个应用下的不同 Pod 分配出子网匹配的 IP 地址。 + 例如，一个集群中，宿主机1的区域只能使用子网 172.20.1.0/24，而宿主机2的区域只能使用子网 172.20.2.0/24，在此背景下，当一个应用跨子网部署副本时，要求 IPAM 能够在不同的节点上，为同一个应用下的不同 Pod 分配出子网匹配的 IP 地址。 -3. 应用 IP 地址固定 +3. **应用 IP 地址固定** - 很多传统应用在云化改造前，是部署在裸金属环境上的，服务之间的网络未引入 NAT 地址转换，微服务架构中需要感知对方的源 IP 或目的 IP， - 并且，网络管理员也习惯了使用防火墙等手段来精细管控网络安全。 + 很多传统应用在云化改造前，是部署在裸金属环境上的，服务之间的网络未引入 NAT 地址转换，微服务架构中需要感知对方的源 IP 或目的 IP，并且，网络管理员也习惯了使用防火墙等手段来精细管控网络安全。 - 因此，应用上云后，无状态应用希望能够实现 IP 范围的固定，有状态应用希望能够实现 IP 地址的唯一对应，这样，能够减少对微服务架构的改造工作。 + 因此，应用上云后，无状态应用希望能够实现 IP 范围的固定，有状态应用希望能够实现 IP 地址的唯一对应，这样，能够减少对微服务架构的改造工作。 -4. 一个 Pod 的多网卡获取不同子网的 IP 地址 +4. **一个 Pod 的多网卡获取不同子网的 IP 地址** - 既然是对接 Underlay 网络，Pod 就会有多网卡需求，以使其通达不同的 Underlay 子网，这要求 IPAM 能够给应用的不同网卡分配不同子网下的 IP 地址。 + 既然是对接 Underlay 网络，Pod 就会有多网卡需求，以使其通达不同的 Underlay 子网，这要求 IPAM 能够给应用的不同网卡分配不同子网下的 IP 地址。 -5. IP 地址冲突 +5. **IP 地址冲突** - 在 Underlay 网络中，更加容易出现 IP 冲突，例如，Pod 与集群外部的主机 IP 发生了冲突，与其它对接了相同子网的集群冲突， - 而 IPAM 组件很难感知外部这些冲突的 IP 地址，多需要借助 CNI 插件进行实时的 IP 冲突检测。 + 在 Underlay 网络中，更加容易出现 IP 冲突，例如，Pod 与集群外部的主机 IP 发生了冲突，与其它对接了相同子网的集群冲突，而 IPAM 组件很难感知外部这些冲突的 IP 地址，多需要借助 CNI 插件进行实时的 IP 冲突检测。 -6. 已用 IP 地址的释放回收 +6. **已用 IP 地址的释放回收** - 因为 Underlay 网络 IP 地址资源的稀缺性，且应用有 IP 地址固定需求，所以，"应当"被释放的 IP 地址若未被 IPAM 组件回收，新启动的 Pod 可能会因为缺少 IP 地址而失败。 - 这就要求 IPAM 组件拥有更加精准、高效、及时的 IP 回收机制。 + 因为 Underlay 网络 IP 地址资源的稀缺性，且应用有 IP 地址固定需求，所以，"应当"被释放的 IP 地址若未被 IPAM 组件回收，新启动的 Pod 可能会因为缺少 IP 地址而失败。这就要求 IPAM 组件拥有更加精准、高效、及时的 IP 回收机制。 -这个方案的优势有：无需网络 NAT 映射的引入，对应用的云化网络改造，提出了最大的便利；底层网络的火墙等设备，可对 Pod 通信实现相对较为精细的管控；无需隧道技术， -网络通信的吞吐量和延时性能也相对的提高了。 +这个方案的优势有：无需网络 NAT 映射的引入，对应用的云化网络改造，提出了最大的便利；底层网络的防火墙等设备，可对 Pod 通信实现相对较为精细的管控；无需隧道技术，网络通信的吞吐量和延时性能也相对的提高了。 ## Spiderpool IPAM @@ -90,41 +76,49 @@ Underlay 网络方案的实施，有两种典型的场景：一种是集群部 [ipvlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/ipvlan), [sriov CNI](https://github.com/k8snetworkplumbingwg/sriov-cni), [ovs CNI](https://github.com/k8snetworkplumbingwg/ovs-cni), -[Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) +[Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni), [calico CNI](https://github.com/projectcalico/calico), [weave CNI](https://github.com/weaveworks/weave) +Spiderpool IPAM 主要分配两个大模块：分配与回收；整体流程如下： + +1. Pod 启动，Spiderpool 会检查对应 Pod 的 `SpiderEndpoint` 对象是否存在，如果存在，对于 StatefulSet 这类有状态应用，Pod 名称都是固定的，且应用有固定 IP 的需求，Spiderpool 会仍使用 SpiderEndpoint 中的记录的 IP 地址为其分配 IP；对于 Deployment 这类无状态应用，因为 Pod 名称会随着 Pod 重启而变化，只能将 IP 固定在一定范围内，Spiderpool 会通过分配算法重新分配 IP 地址给 Pod，同步变更后的 Pod 信息到 IP 池和 SpiderEndpoint 中。如果不存在，Spiderpool 通过分配算法（获取候选池 -> 过滤候选池 -> 候选池排序），给 Pod 分配 IP 地址， 会同时创建一个对应 Pod 的 `SpiderEndpoint` 对象，该对象中记录着 Pod 所使用的 IP 地址、UID 等信息，它会带一个 finalizer，并随 Pod 一起设置为 `OwnerReference`。 + +2. Spiderpool 基于两种方式确保 IP 地址的健壮性，第一，Spiderpool 的 Informer 机制追踪 Pod 的生命周期，第二，周期性的在全局默认间隔时间内扫描 IPPool 的 IP 状态。对于 StatefulSet 这类有状态应用，由于其固定 IP 地址的需求，且只要它的 Pod 副本是有效的，Spiderpool 不会回收其 IP 地址和 SpiderEndpoint 对象，以便于它能持续的获得相同的 IP 地址。对于无效的 StatefulSet 应用和 Deployment 这类无状态应用，将回收其 IP 地址和 SpiderEndpoint 对象。 + +![Spiderpool IPAM 分配与回收流程](../images/ipam-concepts.jpg) + ### Spiderpool IP 分配算法 当 Pod 创建时，它将按照以下步骤获取 IP 分配；IP 分配生命周期将经历 `获取候选池`、`过滤候选池`、`候选池排序` 三个大阶段。 -- 获取候选池：Spiderpool 有多种池选择规则，会严格遵守 **高优先级到低优先级** 的池选择规则，获取**高优先级规则**命中的所有池，将它们标记为候选者身份，以有资格被进一步考虑。 +- **获取候选池**：Spiderpool 有多种池选择规则，会严格遵守 **高优先级到低优先级** 的池选择规则，获取 **高优先级规则** 命中的所有池，将它们标记为候选者身份，以有资格被进一步考虑。 -- 过滤候选池：Spiderpool 通过亲和性等过滤机制，更精确地从所有候选池中选择合适的候选池，以满足特定的需求或复杂的使用场景。 +- **过滤候选池**：Spiderpool 通过亲和性等过滤机制，更精确地从所有候选池中选择合适的候选池，以满足特定的需求或复杂的使用场景。 -- 候选池排序：对于多候选池，Spiderpool 根据 SpiderIPPool 对象中的优先级规则对这些候选者进行排序，然后按顺序从有空闲 IP 的 IP 池中开始选择 IP 地址进行分配。 +- **候选池排序**：对于多候选池，Spiderpool 根据 SpiderIPPool 对象中的优先级规则对这些候选者进行排序，然后按顺序从有空闲 IP 的 IP 池中开始选择 IP 地址进行分配。 #### 获取候选池 Spiderpool 提供多种池选择规则，在为 Pod 分配 IP 地址时，会严格遵守 **高优先级到低优先级** 的池选择规则。以下规则按照从 **高优先级到低优先级** 的顺序列出，如果同时存在下面的多个规则，前一个规则将 **覆盖** 后一个规则。 -- 优先级 1 ：SpiderSubnet 注解。 +- **优先级 1**：SpiderSubnet 注解。 SpiderSubnet 资源代表 IP 地址的集合，当需要为应用分配固定的 IP 地址时，应用管理员需要平台管理员告知可用的 IP 地址和路由属性等，但双方分属两个不同的运营部门，这使得每一个应用创建的工作流程繁琐，借助于 Spiderpool 的 SpiderSubnet 功能，它能自动从中子网分配 IP 给 IPPool，并且还能为应用固定 IP 地址，极大的减少了运维的成本。创建应用时可以使用 `ipam.spidernet.io/subnets` 或 `ipam.spidernet.io/subnet` 注解指定 Subnet，从而实现从子网中随机选取 IP 地址自动创建 IP 池，并从池中分配固定 IP 地址给应用。有关详情，请参阅 [SpiderSubnet](../usage/spider-subnet-zh_CN.md)。 -- 优先级 2 ：SpiderIPPool 注解。 +- **优先级 2**：SpiderIPPool 注解。 一个 Subnet 中的不同 IP 地址，可分别存储到不同的 IPPool 实例中（Spiderpool 会校验 IPPool 之间的地址集合不重叠）。依据需求，SpiderIPPool 中的 IP 集合可大可小。能很好的应对 Underlay 网络的 IP 地址资源有限情况，且这种设计特点，创建应用时，结合 SpiderIPPool 注解 `ipam.spidernet.io/ippools` 或 `ipam.spidernet.io/ippool` 能绑定不同的 IPPool，也能分享相同的 IPPool，既能够让所有应用共享使用同一个 Subnet，又能够实现 "微隔离"。有关详情，请参阅 [SpiderIPPool 注解](../reference/annotation.md)。 -- 优先级 3 ：命名空间默认 IP 池。 +- **优先级 3**：命名空间默认 IP 池。 通过在命名空间中设置注解 `ipam.spidernet.io/default-ipv4-ippool` 或 `ipam.spidernet.io/default-ipv6-ippool` 指定默认的 IP 池。在该租户中创建应用时，如果没有其他高优先级的池规则，那么将从该租户可用的候选池中尝试分配 IP 地址。有关详情，请参阅 [命名空间注解](../reference/annotation.md)。 -- 优先级 4 ：CNI 配置文件。 +- **优先级 4**：CNI 配置文件。 通过在 CNI 配置文件中的 `default_ipv4_ippool` 和 `default_ipv6_ippool` 字段设置全局的 CNI 默认池，其可以设置多个 IP 池用作备选池，当应用使用该 CNI 配置网络时并调用 Spiderpool ，对于每个应用副本，Spiderpool 都会按照 "IP 池数组" 中元素的顺序依次尝试分配 IP 地址，在每个节点分属不同的地区或数据中心的场景，如果应用副本被调度到的节点，符合第一个 IP 池的节点亲和规则，Pod 会从该池中获得 IP 分配，如果不满足，Spiderpool 会尝试从备选池中选择 IP 池继续为 Pod 分配 IP ，直到所有备选池全部筛选失败。详细信息请参考[CNI 配置](../reference/plugin-ipam.md)。 -- 优先级 5 ：集群默认 IPPool。 +- **优先级 5**：集群默认 IPPool。 在 SpiderIPPool CR 对象中，可以通过将 **spec.default** 字段设置为 `true`，将池设置为集群默认 IPPool，默认为 `false`。详细信息请参考[集群默认 IPPool](../reference/crd-spiderippool.md) @@ -166,23 +160,28 @@ Spiderpool 提供多种池选择规则，在为 Pod 分配 IP 地址时，会严 > > 1. 具有属性 `IPPool.Spec.PodAffinity` 和 `IPPool.Spec.NodeName` 的 _IPPoolA_ 的优先级高于具有单一关联属性 `IPPool.Spec.PodAffinity` 的 _IPPoolB_。 > 2. 具有单个属性 `IPPool.Spec.PodAffinity` 的 _IPPoolA_ 的优先级高于具有属性 `IPPool.Spec.NodeName` 和 `IPPool.Spec.NamespaceName` 的 _IPPoolB_。 -> 3. 具有属性 `IPPool.Spec.PodAffinity` 和 `IPPool.Spec.NodeName` 的 _IPPoolA_ 的优先级高于具有属性 `IPPool.Spec.PodAffinity`、`IPPool.Spec.NamespaceName` 和 `IPPool.Spec.MultusName` 的 _IPPoolB_ 。 +> 3. 具有属性 `IPPool.Spec.PodAffinity` 和 `IPPool.Spec.NodeName` 的 _IPPoolA_ 的优先级高于具有属性 `IPPool.Spec.PodAffinity`、`IPPool.Spec.NamespaceName` 和 `IPPool.Spec.MultusName` 的 _IPPoolB_。 NOTE： > 如果 Pod 属于 StatefulSet，则会优先分配符合上面规则的 IP 地址。 一旦 Pod 重新启动，它将尝试重用最后分配的 IP 地址。 -## IP 回收机制 +### IP 回收算法 -在 Kubernetes 中，垃圾回收（Garbage Collection，简称GC）对于 IP 地址的回收非常重要。IP 地址的可用性关系到 Pod 是否能够启动成功。GC 机制可以自动回收这些不再使用的 IP 地址，避免资源浪费和 IP 地址的耗尽。 +在 Kubernetes 中，垃圾回收（Garbage Collection，简称GC）对于 IP 地址的回收至关重要。IP 地址的可用性直接影响 Pod 是否能够成功启动。同时 GC 机制也可以自动回收不再使用的 IP 地址，避免资源浪费和 IP 地址耗尽。 +在 IPAM 中会记录分配给 Pod 使用的 IP 信息，但当这些 Pod 在 Kubernetes 集群中已经不复存在，而这些仍被记录在 IPAM 中的 IP 可称为 `僵尸 IP`。Spiderpool 针对 `僵尸 IP` 具有如下两种回收方式： -在 IPAM 中记录了分配给 Pod 使用的 IP 地址，但是这些 Pod 在 Kubernetes 集群中已经不复存在，这些 IP 可称为 `僵尸 IP` ，Spiderpool 可针对 `僵尸 IP` 进行回收，它的实现原理如下： +1. 实时追踪 Pod 事件，判断是否需要回收 IP 地址和其对应的 SpiderEndpoint 对象。 +2. 基于环境变量 `SPIDERPOOL_GC_DEFAULT_INTERVAL_DURATION` 定义的间隔时间（默认为 10 分钟）周期性扫描 IP 池的健壮性。 -在集群中 `delete Pod` 时，但由于`网络异常`或 `cni 二进制 crash` 等问题，导致调用 `cni delete` 失败，从而导致 IP 地址无法被 cni 回收。 +上述完备的 IP 回收算法，能够确保所有场景下 IP 地址的正确回收，包括如下的一些特殊场景： -- 在 `cni delete 失败` 等故障场景，如果一个曾经分配了 IP 的 Pod 被销毁后，但在 IPAM 中还记录分配着IP 地址，形成了僵尸 IP 的现象。Spiderpool 针对这种问题，会基于周期和事件扫描机制，自动回收这些僵尸 IP 地址。 -- 因其他意外导致 **无状态** Pod 一直处于 `Terminating` 阶段，Spiderpool 将在 Pod 的 `spec.terminationGracePeriodSecond` + [spiderpool-controller ENV](./../reference/spiderpool-controller.md#env) `SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY` 时间后，自动释放其 IP 地址。该功能可通过环境变量 `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED` 来控制。该能力能够用以解决 `节点正常但 Pod 删除失败` 的故障场景。 +- 在集群中 `delete Pod` 时，由于`网络异常`或 `cni 二进制 crash` 等问题，导致调用 `cni delete` 失败，从而导致 IP 地址无法被 cni 回收。 + + - 在 `cni delete 失败` 等故障场景下，如果一个曾经分配了 IP 的 Pod 被销毁，但在 IPAM 中仍记录着 IP 地址，形成僵尸 IP。Spiderpool 基于周期和事件扫描机制，自动回收这些僵尸 IP 地址。 -节点意外宕机后，集群中的 Pod 永久处于 `Terminating` 阶段，Pod 占用的 IP 地址无法被释放。 + - 因其他意外导致 **无状态** Pod 一直处于 `Terminating` 阶段，Spiderpool 将在 Pod 的 `spec.terminationGracePeriodSecond` + [spiderpool-controller ENV](./../reference/spiderpool-controller.md#env) `SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY` 时间后，自动释放其 IP 地址。该功能可通过环境变量 `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED` 控制，解决 `节点正常但 Pod 删除失败` 的故障场景。 -- 对处于 `Terminating` 阶段的 **无状态** Pod，Spiderpool 将在 Pod 的 `spec.terminationGracePeriodSecond` + [spiderpool-controller ENV](./../reference/spiderpool-controller.md#env) `SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY` 时间后，自动释放其 IP 地址。该功能可通过环境变量 `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED` 来控制。该能力能够用以解决 `节点意外宕机` 的故障场景。 +- 节点意外宕机后，集群中的 Pod 永久处于 `Terminating` 阶段，Pod 占用的 IP 地址无法被释放。 + + - 对于处于 `Terminating` 阶段的 **无状态** Pod，Spiderpool 将在 Pod 的 `spec.terminationGracePeriodSecond` + [spiderpool-controller ENV](./../reference/spiderpool-controller.md#env) `SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY` 时间后，自动释放其 IP 地址。该功能可通过环境变量 `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED` 控制，解决 `节点意外宕机` 的故障场景。 diff --git a/docs/concepts/ipam-des.md b/docs/concepts/ipam-des.md index 1d9c68fb86..18fa98c19d 100644 --- a/docs/concepts/ipam-des.md +++ b/docs/concepts/ipam-des.md @@ -2,74 +2,70 @@ **English** | [**简体中文**](./ipam-des-zh_CN.md) -## IPAM for Underlay and overlay network solutions +## IPAM for Underlay and Overlay Network Solutions -There are two technologies in cloud-native networking: "overlay network" and "underlay network". -Despite no strict definition for underlay and overlay networks in cloud-native networking, we can simply abstract their characteristics from many CNI projects. The two technologies meet the needs of different scenarios. +There are two technologies in cloud-native networking: "overlay network" and "underlay network". Despite no strict definition for underlay and overlay networks in cloud-native networking, we can simply abstract their characteristics from many CNI projects. The two technologies meet the needs of different scenarios. Spiderpool is designed for underlay networks, and the following comparison of the two solutions can better illustrate the features and usage scenarios of Spiderpool. -### IPAM for overlay networks +### IPAM for Overlay Networks -These solutions implement the decoupling of the Pod network and host network, such as [Calico](https://github.com/projectcalico/calico), [Cilium](https://github.com/cilium/cilium) and other CNI plugins. Typically, they use tunnel technology such as vxlan to build an overlay network plane, and use NAT technology for north-south traffic. +These solutions implement the decoupling of the Pod network and host network, such as [Calico](https://github.com/projectcalico/calico), [Cilium](https://github.com/cilium/cilium), and other CNI plugins. Typically, they use tunnel technology such as vxlan to build an overlay network plane and use NAT technology for north-south traffic. These IPAM solutions have the following characteristics: -1. Divide Pod subnets into node-based IP blocks +1. **Divide Pod subnets into node-based IP blocks** - In terms of a smaller subnet mask, the Pod subnet is divided into smaller IP blocks, and each node is assigned one or more IP blocks depending on the actual IP allocation account. + In terms of a smaller subnet mask, the Pod subnet is divided into smaller IP blocks, and each node is assigned one or more IP blocks depending on the actual IP allocation account. - First, since the IPAM plugin on each node only needs to allocate and release IP addresses in the local IP block, there is no IP allocation conflict with IPAM on other nodes, achieving more efficient allocation. - Second, a specific IP address follows an IP block and is allocated within one node all the time, so it cannot be assigned on other nodes together with a bound Pod. + - Since the IPAM plugin on each node only needs to allocate and release IP addresses in the local IP block, there is no IP allocation conflict with IPAM on other nodes, achieving more efficient allocation. + - A specific IP address follows an IP block and is allocated within one node all the time, so it cannot be assigned on other nodes together with a bound Pod. -2. Sufficient IP address resources +2. **Sufficient IP address resources** - Subnets not overlapping with any CIDR, could be used by the cluster, so the cluster has enough IP address resources as long as NAT technology is used in an appropriate manner. As a result, IPAM components face less pressure to reclaim abnormal IP addresses. + Subnets not overlapping with any CIDR could be used by the cluster, so the cluster has enough IP address resources as long as NAT technology is used in an appropriate manner. As a result, IPAM components face less pressure to reclaim abnormal IP addresses. -3. No requirement for static IP addresses +3. **No requirement for static IP addresses** - For the static IP address requirement, there is a difference between a stateless application and a stateful application. Regarding stateless application like deployment, the Pod's name will change when the Pod restarts, and the business logic of the application itself is stateless. Thus static IP addresses means that all the Pod replicas are fixed in a set of IP addresses; for stateful applications such as statefulset, considering both the fixed information including Pod's names and stateful business logic, the strong binding of one Pod and one specific IP address needs to be implemented for static IP addresses. + For the static IP address requirement, there is a difference between a stateless application and a stateful application. Regarding stateless applications like deployment, the Pod's name will change when the Pod restarts, and the business logic of the application itself is stateless. Thus, static IP addresses mean that all the Pod replicas are fixed in a set of IP addresses. For stateful applications such as statefulset, considering both the fixed information including Pod's names and stateful business logic, the strong binding of one Pod and one specific IP address needs to be implemented for static IP addresses. - The "overlay network solution" mostly exposes the ingress and source addresses of services to the outside of the cluster with the help of NAT technology, and realizes the east-west communication through DNS, clusterIP and other technologies. - In addition, although the IP block of IPAM fixes the IP to one node, it does not guarantee the application replicas follow the scheduling. Therefore, there is no scope for the static IP address capability. Most of the mainstream CNIs in the community have not yet supported "static IP addressed", or support it in a rough way. + The "overlay network solution" mostly exposes the ingress and source addresses of services to the outside of the cluster with the help of NAT technology and realizes the east-west communication through DNS, clusterIP, and other technologies. In addition, although the IP block of IPAM fixes the IP to one node, it does not guarantee the application replicas follow the scheduling. Therefore, there is no scope for the static IP address capability. Most of the mainstream CNIs in the community have not yet supported "static IP addresses" or support it in a rough way. -The advantage of the "overlay network solution" is that the CNI plugins are highly compatible with any underlying network environment, and can provide independent subnets with sufficient IP addresses for Pods. +The advantage of the "overlay network solution" is that the CNI plugins are highly compatible with any underlying network environment and can provide independent subnets with sufficient IP addresses for Pods. -### IPAM for underlay networks +### IPAM for Underlay Networks These solutions share the node's network for Pods, which means Pods can directly obtain IP addresses in the node network. Thus, applications can directly use their own IP addresses for east-west and north-south communications. There are two typical scenarios for underlay network solutions: clusters deployed on a "legacy network" and clusters deployed on an IAAS environment, such as a public cloud. The following summarizes the IPAM characteristics of the "legacy network scenario": -1. An IP address able to be assigned to any node +1. **An IP address able to be assigned to any node** - As the number of network devices in the data center increases and multi-cluster technology evolves, IPv4 address resources become scarce, thus requiring IPAM to improve the efficiency of IP usage. - As the Pod replicas of the applications requiring "static IP addresses" could be scheduled to any node in the cluster and drift between nodes, IP addresses might drift together. + As the number of network devices in the data center increases and multi-cluster technology evolves, IPv4 address resources become scarce, thus requiring IPAM to improve the efficiency of IP usage. As the Pod replicas of the applications requiring "static IP addresses" could be scheduled to any node in the cluster and drift between nodes, IP addresses might drift together. - Therefore, an IP address should be able to be allocated to a Pod on any node. + Therefore, an IP address should be able to be allocated to a Pod on any node. -2. Different replicas within one application could obtain IP addresses across subnets +2. **Different replicas within one application could obtain IP addresses across subnets** - Take as an example one node could access subnet 172.20.1.0/24 while another node just only access subnet 172.20.2.0/24. In this case, when the replicas within one application need to be deployed across subnets, IPAM is required to be able to assign subnet-matched IP addresses to the application on different nodes. + Take as an example one node could access subnet 172.20.1.0/24 while another node just only access subnet 172.20.2.0/24. In this case, when the replicas within one application need to be deployed across subnets, IPAM is required to be able to assign subnet-matched IP addresses to the application on different nodes. -3. Static IP addresses +3. **Static IP addresses** - For some traditional applications, the source IPs or destination IPs need to be sensed in the microservice. And network admins are used to enabling fine-grained network security control via firewalls and other means. + For some traditional applications, the source IPs or destination IPs need to be sensed in the microservice. And network admins are used to enabling fine-grained network security control via firewalls and other means. - Therefore, in order to reduce the transformation chores after the applications move to the Kubernetes, applications need static IP addresses. + Therefore, in order to reduce the transformation chores after the applications move to Kubernetes, applications need static IP addresses. -4. Pods with Multiple NICs need IP addresses of different underlay subnets +4. **Pods with Multiple NICs need IP addresses of different underlay subnets** - Since the Pod is connected to an underlay network, it has the need for multiple NICs to reach different underlay subnets. + Since the Pod is connected to an underlay network, it has the need for multiple NICs to reach different underlay subnets. -5. IP conflict +5. **IP conflict** - Underlay networks are more prone to IP conflicts. For instance, Pods conflict with host IPs outside the cluster, or conflict with other clusters under the same subnet. But it is difficult for IPAM to discover these conflicting IP addresses externally unless CNI plugins are involved for real-time IP conflict detection. + Underlay networks are more prone to IP conflicts. For instance, Pods conflict with host IPs outside the cluster, or conflict with other clusters under the same subnet. But it is difficult for IPAM to discover these conflicting IP addresses externally unless CNI plugins are involved for real-time IP conflict detection. -6. Release and recover IP addresses +6. **Release and recover IP addresses** - Because of the scarcity of IP addresses in underlay networks and the static IP address requirements of applications, a newly launched Pod may fail due to the lack of IP addresses owing to some IP addresses not being released by abnormal Pods. - This requires IPAMs to have a more accurate, efficient and timely IP recovery mechanism. + Because of the scarcity of IP addresses in underlay networks and the static IP address requirements of applications, a newly launched Pod may fail due to the lack of IP addresses owing to some IP addresses not being released by abnormal Pods. This requires IPAMs to have a more accurate, efficient, and timely IP recovery mechanism. The advantages of the underlay network solution include: no need for network NAT mapping, which makes cloud-based network transformation for applications way more convenient; the underlying network firewall and other devices can achieve relatively fine control of Pod communication; no tunneling technology contributes to improved throughput and latency performance of network communications. @@ -77,69 +73,69 @@ The advantages of the underlay network solution include: no need for network NAT Any CNI project compatible with third-party IPAM plugins can work well with Spiderpool IPAM, such as: -[Macvlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/macvlan), -[vlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/vlan), -[ipvlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/ipvlan), -[SR-IOV CNI](https://github.com/k8snetworkplumbingwg/sriov-cni), -[ovs CNI](https://github.com/k8snetworkplumbingwg/ovs-cni), -[Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni), -[Calico CNI](https://github.com/projectcalico/calico), -[Weave CNI](https://github.com/weaveworks/weave) +- [Macvlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/macvlan) +- [Vlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/vlan) +- [Ipvlan CNI](https://github.com/containernetworking/plugins/tree/main/plugins/main/ipvlan) +- [SR-IOV CNI](https://github.com/k8snetworkplumbingwg/sriov-cni) +- [OVS CNI](https://github.com/k8snetworkplumbingwg/ovs-cni) +- [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) +- [Calico CNI](https://github.com/projectcalico/calico) +- [Weave CNI](https://github.com/weaveworks/weave) + +Spiderpool IPAM is primarily divided into two major modules: allocation and recovery; the overall process is as follows: + +1. When a Pod starts, Spiderpool checks if the corresponding Pod's `SpiderEndpoint` object exists. If it exists, for StatefulSet applications that have a fixed Pod name and require a fixed IP, Spiderpool will continue to use the IP address recorded in SpiderEndpoint for allocation. For Deployment applications that are stateless, because the Pod name changes with each Pod restart, Spiderpool can only fix the IP within a certain range and will reassign an IP address to the Pod through an allocation algorithm, synchronizing the updated Pod information to the IP pool and SpiderEndpoint. If it does not exist, Spiderpool will allocate an IP address to the Pod through an allocation algorithm (getting candidate pools -> filtering candidate pools -> sorting candidate pools), and simultaneously create a `SpiderEndpoint` object corresponding to the Pod, which records the IP address, UID, and other information used by the Pod. It will also carry a finalizer and be set as an `OwnerReference` along with the Pod. + +2. Spiderpool ensures the robustness of IP addresses through two methods. First, Spiderpool's Informer mechanism tracks the lifecycle of Pods. Second, it periodically scans the IP status of IPPool within the global default interval. For StatefulSet applications that require fixed IP addresses, as long as their Pod replicas are valid, Spiderpool will not reclaim their IP addresses and SpiderEndpoint objects, allowing them to consistently obtain the same IP address. For invalid StatefulSet applications and Deployment applications that are stateless, their IP addresses and SpiderEndpoint objects will be reclaimed. + +![Spiderpool IPAM Allocation and Reclaim](../images/ipam-concepts.jpg) ### IP Allocation Algorithm -When creating a Pod, it will follow the steps below to get IP allocations.The lifecycle of IP allocation involves three major stages: `candidate pool acquisition`, `candidate pool filtering`, and `candidate pool sorting`. +When creating a Pod, it will follow the steps below to get IP allocations. The lifecycle of IP allocation involves three major stages: `candidate pool acquisition`, `candidate pool filtering`, and `candidate pool sorting`. -- Candidate pool acquisition: Spiderpool follows a strict rule of selecting pools from **high to low priority**. It identifies all pools that match **the high priority rules** and marks them as candidates for further consideration. +- **Candidate pool acquisition**: Spiderpool follows a strict rule of selecting pools from **high to low priority**. It identifies all pools that match **the high priority rules** and marks them as candidates for further consideration. -- Candidate pool filtering: Spiderpool applies filtering mechanisms such as affinity to carefully select the appropriate candidate pools from the available options. This ensures that specific requirements or complex usage scenarios are satisfied. +- **Candidate pool filtering**: Spiderpool applies filtering mechanisms such as affinity to carefully select the appropriate candidate pools from the available options. This ensures that specific requirements or complex usage scenarios are satisfied. -- Candidate pool sorting: in cases where multiple candidate pools exist, Spiderpool sorts them based on the priority rules defined in the SpiderIPPool object. IP addresses are then allocated sequentially, starting from the pool with available addresses. +- **Candidate pool sorting**: In cases where multiple candidate pools exist, Spiderpool sorts them based on the priority rules defined in the SpiderIPPool object. IP addresses are then allocated sequentially, starting from the pool with available addresses. #### Candidate Pool Acquisition Spiderpool offers a variety of pool selection rules when assigning IP addresses to Pods. The selection process strictly adheres to a **high to low priority** order. The following rules are listed in **descending order of priority**, and if multiple rules apply at the same time, the preceding rule will **overwrite** the subsequent one. -- The 1st priority: SpiderSubnet annotation +1. **The 1st priority: SpiderSubnet annotation** - The SpiderSubnet resource represents a collection of IP addresses. When an application requires a fixed IP address, the application administrator needs to inform their platform counterparts about the available IP addresses and routing attributes. However, as they belong to different operational departments, this process becomes cumbersome, resulting in complex workflows for creating each application. To simplify this, Spiderpool's SpiderSubnet feature automatically allocates IP addresses from subnets to IPPool and assigns fixed IP addresses to applications. This greatly reduces operational costs. When creating an application, you can use the `ipam.spidernet.io/subnets` or `ipam.spidernet.io/subnet` annotation to specify the Subnet. This allows for the automatic creation of an IP pool by randomly selecting IP addresses from the subnet, which can then be allocated as fixed IPs for the application. For more details, please refer to [SpiderSubnet](../usage/spider-subnet.md). + The SpiderSubnet resource represents a collection of IP addresses. When an application requires a fixed IP address, the application administrator needs to inform their platform counterparts about the available IP addresses and routing attributes. However, as they belong to different operational departments, this process becomes cumbersome, resulting in complex workflows for creating each application. To simplify this, Spiderpool's SpiderSubnet feature automatically allocates IP addresses from subnets to IPPool and assigns fixed IP addresses to applications. This greatly reduces operational costs. When creating an application, you can use the `ipam.spidernet.io/subnets` or `ipam.spidernet.io/subnet` annotation to specify the Subnet. This allows for the automatic creation of an IP pool by randomly selecting IP addresses from the subnet, which can then be allocated as fixed IPs for the application. For more details, please refer to [SpiderSubnet](../usage/spider-subnet.md). -- The 2nd priority: SpiderIPPool annotation +2. **The 2nd priority: SpiderIPPool annotation** - Different IP addresses within a Subnet can be stored in separate instances of IPPool (Spiderpool ensures that there is no overlap between the address sets of IPPools). The size of the IP collection in SpiderIPPool can vary based on requirements. This design feature is particularly beneficial when dealing with limited IP address resources in the Underlay network. When creating an application, the SpiderIPPool annotation `ipam.spidernet.io/ippools` or `ipam.spidernet.io/ippool` can be used to bind different IPPools or share the same IPPool. This allows all applications to share the same Subnet while maintaining "micro-isolation". For more details, please refer to [SpiderIPPool annotation](../reference/annotation.md). + Different IP addresses within a Subnet can be stored in separate instances of IPPool (Spiderpool ensures that there is no overlap between the address sets of IPPools). The size of the IP collection in SpiderIPPool can vary based on requirements. This design feature is particularly beneficial when dealing with limited IP address resources in the Underlay network. When creating an application, the SpiderIPPool annotation `ipam.spidernet.io/ippools` or `ipam.spidernet.io/ippool` can be used to bind different IPPools or share the same IPPool. This allows all applications to share the same Subnet while maintaining "micro-isolation". For more details, please refer to [SpiderIPPool annotation](../reference/annotation.md). -- The 3th priority: namespace default IP pool +3. **The 3rd priority: Namespace default IP pool** - By setting the annotation `ipam.spidernet.io/default-ipv4-ippool` or `ipam.spidernet.io/default-ipv6-ippool` in the namespace, you can specify the default IP pool. When creating an application within that tenant, if there are no other higher-priority pool rules, it will attempt to allocate an IP address from the available candidate pools for that tenant. For more details, please refer to [Namespace Annotation](../reference/annotation.md). + By setting the annotation `ipam.spidernet.io/default-ipv4-ippool` or `ipam.spidernet.io/default-ipv6-ippool` in the namespace, you can specify the default IP pool. When creating an application within that tenant, if there are no other higher-priority pool rules, it will attempt to allocate an IP address from the available candidate pools for that tenant. For more details, please refer to [Namespace Annotation](../reference/annotation.md). -- The fourth priority: CNI configuration file +4. **The 4th priority: CNI configuration file** - The global CNI default pool can be set by configuring the `default_ipv4_ippool` and `default_ipv6_ippool` fields in the CNI configuration file. Multiple IP pools can be defined as alternative pools. When an application uses this CNI configuration network and invokes Spiderpool, each application replica is sequentially assigned an IP address according to the order of elements in the "IP pool array". In scenarios where nodes belong to different regions or data centers, if the node where an application replica scheduled matches the node affinity rule of the first IP pool, the Pod obtains an IP from that pool. If it doesn't meet the criteria, Spiderpool attempts to assign an IP from the alternative pools until all options have been exhausted. For more information, please refer to [CNI Configuration](../reference/plugin-ipam.md). + The global CNI default pool can be set by configuring the `default_ipv4_ippool` and `default_ipv6_ippool` fields in the CNI configuration file. Multiple IP pools can be defined as alternative pools. When an application uses this CNI configuration network and invokes Spiderpool, each application replica is sequentially assigned an IP address according to the order of elements in the "IP pool array". In scenarios where nodes belong to different regions or data centers, if the node where an application replica is scheduled matches the node affinity rule of the first IP pool, the Pod obtains an IP from that pool. If it doesn't meet the criteria, Spiderpool attempts to assign an IP from the alternative pools until all options have been exhausted. For more information, please refer to [CNI Configuration](../reference/plugin-ipam.md). -- The fifth priority: cluster's default IP pool +5. **The 5th priority: Cluster's default IP pool** - Within the SpiderIPPool CR object, setting the **spec.default** field to `true` designates the pool as the cluster's default IP pool (default value is `false`). For more information, please refer to [Cluster's Default IP Pool](../reference/crd-spiderippool.md). + Within the SpiderIPPool CR object, setting the **spec.default** field to `true` designates the pool as the cluster's default IP pool (default value is `false`). For more information, please refer to [Cluster's Default IP Pool](../reference/crd-spiderippool.md). #### Candidate Pool Filtering To determine the availability of candidate IP pools for IPv4 and IPv6, Spiderpool filters them using the following rules: - IP pools in the `terminating` state are filtered out. - - The `spec.disable` field of an IP pool indicates its availability. A value of `true` means the IP pool is not usable. - - Check if the `IPPool.Spec.NodeName` and `IPPool.Spec.NodeAffinity` match the Pod's scheduling node. Mismatching values result in filtering out the IP pool. - - Check if the `IPPool.Spec.NamespaceName` and `IPPool.Spec.NamespaceAffinity` match the Pod's namespace. Mismatching values lead to filtering out the IP pool. - - Check if the `IPPool.Spec.NamespaceName` matches the Pod's `matchLabels`. Mismatching values lead to filtering out the IP pool. - - Check if the `IPPool.Spec.MultusName` matches the current NIC Multus configuration of the Pod. If there is no match, the IP pool is filtered out. - - Check if all IPs within the IP pool are included in the IPPool instance's `exclude_ips` field. If it is, the IP pool is filtered out. - - Check if all IPs in the pool are reserved in the ReservedIP instance. If it is, the IP pool is filtered out. - - An IP pool will be filtered out if its available IP resources are exhausted. #### Candidate Pool Sorting @@ -147,18 +143,15 @@ To determine the availability of candidate IP pools for IPv4 and IPv6, Spiderpoo After filtering the candidate pools, Spiderpool may have multiple pools remaining. To determine the order of IP address allocation, Spiderpool applies custom priority rules to sort these candidates. IP addresses are then selected from the pools with available IPs in the following manner: - IP pool resources with the `IPPool.Spec.PodAffinity` property are given the highest priority. - - IPPool resources with either the `IPPool.Spec.NodeName` or `IPPool.Spec.NodeAffinity` property are given the secondary priority. The `NodeName` takes precedence over `NodeAffinity`. - - Following that, IP pool resources with either the `IPPool.Spec.NamespaceName` or `IPPool.Spec.NamespaceAffinity` property maintain the third-highest priority. The `NamespaceName` takes precedence over `NamespaceAffinity`. - - IP pool resources with the `IPPool.Spec.MultusName` property receive the lowest priority. > Here are some simple instances to describe this rule. > -> 1. *IPPoolA* with properties `IPPool.Spec.PodAffinity` and `IPPool.Spec.NodeName` has higher priority than *IPPoolB* with single affinity property `IPPool.Spec.PodAffinity`. -> 2. *IPPoolA* with single property `IPPool.Spec.PodAffinity` has higher priority than *IPPoolB* with properties `IPPool.Spec.NodeName` and `IPPool.Spec.NamespaceName`. -> 3. *IPPoolA* with properties `IPPool.Spec.PodAffinity` and `IPPool.Spec.NodeName` has higher priority than *IPPoolB* with properties `IPPool.Spec.PodAffinity`,`IPPool.Spec.NamespaceName` and `IPPool.Spec.MultusName`. +> 1. *IPPoolA* with properties `IPPool.Spec.PodAffinity` and `IPPool.Spec.NodeName` has higher priority than *IPPoolB* with a single affinity property `IPPool.Spec.PodAffinity`. +> 2. *IPPoolA* with a single property `IPPool.Spec.PodAffinity` has higher priority than *IPPoolB* with properties `IPPool.Spec.NodeName` and `IPPool.Spec.NamespaceName`. +> 3. *IPPoolA* with properties `IPPool.Spec.PodAffinity` and `IPPool.Spec.NodeName` has higher priority than *IPPoolB* with properties `IPPool.Spec.PodAffinity`, `IPPool.Spec.NamespaceName`, and `IPPool.Spec.MultusName`. > > If a Pod belongs to StatefulSet, IP addresses that meet the aforementioned rules will be allocated with priority. When a Pod is restarted, it will attempt to reuse the previously assigned IP address. @@ -166,66 +159,53 @@ After filtering the candidate pools, Spiderpool may have multiple pools remainin ### Context -When a pod is normally deleted, the CNI plugin will be called to clean IP on a pod interface and make IP free on IPAM database. -This can make sure all IPs are managed correctly and no IP leakage issue occurs. +When a pod is normally deleted, the CNI plugin will be called to clean IP on a pod interface and make IP free on the IPAM database. This can make sure all IPs are managed correctly and no IP leakage issue occurs. -But on cases, it may go wrong and IP of IPAM database is still marked as used by a nonexistent pod. +But in some cases, it may go wrong and the IP of the IPAM database is still marked as used by a nonexistent pod. -when some errors happened, the CNI plugin is not called correctly when pod deletion. This could happen like cases: +When some errors happen, the CNI plugin is not called correctly when pod deletion. This could happen in cases like: - When a CNI plugin is called, its network communication goes wrong and fails to release IP. - -- The container runtime goes wrong and fails to call CNI plugin. - -- A node breaks down and then always can not recover, the api-server makes pods of the breakdown node to be `deleting` status, but the CNI plugin fails to be called. +- The container runtime goes wrong and fails to call the CNI plugin. +- A node breaks down and then always cannot recover, the api-server makes pods of the breakdown node to be in `deleting` status, but the CNI plugin fails to be called. BTW, this fault could be simply simulated by removing the CNI binary on a host when pod deletion. This issue will make a bad result: -- the new pod may fail to run because the expected IP is still occupied. - -- the IP resource is exhausted gradually although the actual number of pods does not grow. +- The new pod may fail to run because the expected IP is still occupied. +- The IP resource is exhausted gradually although the actual number of pods does not grow. -Some CNI or IPAM plugins could not handle this issue. For some CNIs, the administrator self needs to find the IP with this issue and use a CLI tool to reclaim them. -For some CNIs, it runs an interval job to find the IP with this issue and not reclaim them in time. For some CNIs, there is not any mechanism at all to fix the IP issue. +Some CNI or IPAM plugins could not handle this issue. For some CNIs, the administrator self needs to find the IP with this issue and use a CLI tool to reclaim them. For some CNIs, it runs an interval job to find the IP with this issue and not reclaim them in time. For some CNIs, there is not any mechanism at all to fix the IP issue. ### Solution -For some CNIs, its IP CIDR is big enough, so the leaked IP issue is not urgent. -For Spiderpool, all IP resources are managed by administrator, and an application will be bound to a fixed IP, so the IP reclaim can be finished in time. +For some CNIs, its IP CIDR is big enough, so the leaked IP issue is not urgent. For Spiderpool, all IP resources are managed by the administrator, and an application will be bound to a fixed IP, so the IP reclaim can be finished in time. -### SpiderIPPool garbage collection +### SpiderIPPool Garbage Collection To prevent IP from leaking when the ippool resource is deleted, Spiderpool has some rules: -- For an ippool, if IP still taken by pods, Spiderpool uses webhook to reject deleting request of the ippool resource. - -- For a deleting ippool, the IPAM plugin will stop assigning IP from it, but could release IP from it. - -- The ippool sets a finalizer by the spiderpool controller once it is created. After the ippool goes to be `deleting` status, - the spiderpool controller will remove the finalizer when all IPs in the ippool are free, then the ippool object will be deleted. - -### SpiderEndpoint garbage collection +- For an ippool, if IP is still taken by pods, Spiderpool uses webhook to reject the deleting request of the ippool resource. +- For a deleting ippool, the IPAM plugin will stop assigning IP from it but could release IP from it. +- The ippool sets a finalizer by the spiderpool controller once it is created. After the ippool goes to `deleting` status, the spiderpool controller will remove the finalizer when all IPs in the ippool are free, then the ippool object will be deleted. +- When a pod is deleted, Spiderpool will release its IPs with the recorded data by a corresponding `SpiderEndpoint` object, then the spiderpool controller will remove the `Current` data of the SpiderEndpoint object and remove its finalizer. (For the StatefulSet `SpiderEndpoint`, Spiderpool will delete it directly if its `Current` data was cleaned up) -Once a pod is created and gets IPs from `SpiderIPPool`, Spiderpool will create a corresponding `SpiderEndpoint` object at the same time. -It will take a finalizer (except the StatefulSet pod) and will be set to `OwnerReference` with the pod. +### SpiderIPPool Garbage Collection Algorithm -When a pod is deleted, Spiderpool will release its IPs with the recorded data by a corresponding `SpiderEndpoint` object, -then spiderpool controller will remove the `Current` data of SpiderEndpoint object and remove its finalizer. -(For the StatefulSet `SpiderEndpoint`, Spiderpool will delete it directly if its `Current` data was cleaned up) +In Kubernetes, garbage collection (GC for short) is crucial for recycling IP addresses. The availability of IP addresses is critical to whether a Pod can start successfully. The GC mechanism can automatically reclaim these unused IP addresses, avoiding waste of resources and exhaustion of IP addresses. The IP information assigned to the Pod will be recorded in IPAM, but when these Pods no longer exist in the Kubernetes cluster, these IPs that are still recorded in IPAM can be called `zombie IPs`. Spiderpool has the following two recycling methods for `zombie IPs`: -In Kubernetes, garbage collection (Garbage Collection, GC for short) is very important for the recycling of IP addresses. The availability of IP addresses is critical to whether a Pod can start successfully. The GC mechanism can automatically reclaim these unused IP addresses, avoiding waste of resources and exhaustion of IP addresses. This article will introduce Spiderpool's excellent GC capabilities. +1. Real-time tracking of Pod events to determine whether the IP address and its corresponding SpiderEndpoint object need to be recycled. +2. Periodically scan the robustness of the IP pool based on the interval defined by the environment variable `SPIDERPOOL_GC_DEFAULT_INTERVAL_DURATION` (the default is 10 minutes). -### Project Functions +The above complete IP recovery algorithm can ensure the correct recovery of IP addresses in all scenarios, including the following special scenarios: -The IP addresses assigned to Pods are recorded in IPAM, but these Pods no longer exist in the Kubernetes cluster. These IPs can be called `zombie IPs`. Spiderpool can recycle `zombie IPs`. Its implementation principle is as follows : +- When `deleting Pod` in the cluster, but due to problems such as `network exception` or `cni binary crash`, the call to `cni delete` fails, resulting in the IP address not being reclaimed by cni. -When `deleting Pod` in the cluster, but due to problems such as `network exception` or `cni binary crash`, the call to `cni delete` fails, resulting in the IP address not being reclaimed by cni. + - In failure scenarios such as `cni delete failure`, if a Pod that has been assigned an IP is destroyed, but the IP address is still recorded in the IPAM, a phenomenon of zombie IP is formed. For this kind of problem, Spiderpool will automatically recycle these zombie IP addresses based on the cycle and event scanning mechanism. -- In failure scenarios such as `cni delete failure`, if a Pod that has been assigned an IP is destroyed, but the IP address is still recorded in the IPAM, a phenomenon of zombie IP is formed. For this kind of problem, Spiderpool will automatically recycle these zombie IP addresses based on the cycle and event scanning mechanism. -- In some accidents, the **stateless** Pod is in a constant `Terminating` phase, Spiderpool will automatically release its IP address after the Pod's `spec.terminationGracePeriodSecond` + [spiderpool-controller ENV](./../reference/spiderpool-controller.md#env) `SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY` periods. This feature can be controlled by the environment variable `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED`. This capability can be used to solve the failure scenario of `unexpected Pod downtime with Node ready`. + - In some accidents, the **stateless** Pod is in a constant `Terminating` phase, Spiderpool will automatically release its IP address after the Pod's `spec.terminationGracePeriodSecond` + [spiderpool-controller ENV](./../reference/spiderpool-controller.md#env) `SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY` periods. This feature can be controlled by the environment variable `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED`. This capability can be used to solve the failure scenario of `unexpected Pod downtime with Node ready`. -After a node goes down unexpectedly, the Pod in the cluster is permanently in the `Terminating` phase, and the IP address occupied by the Pod cannot be released. +- After a node goes down unexpectedly, the Pod in the cluster is permanently in the `Terminating` phase, and the IP address occupied by the Pod cannot be released. -- For the **stateless** Pod in `Terminating` phase, Spiderpool will automatically release its IP address after the Pod's `spec.terminationGracePeriodSecond`. This feature can be controlled by the environment variable `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED`. This capability can be used to solve the failure scenario of `unexpected node downtime`. + - For the **stateless** Pod in the `Terminating` phase, Spiderpool will automatically release its IP address after the Pod's `spec.terminationGracePeriodSecond`. This feature can be controlled by the environment variable `SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED`. This capability can be used to solve the failure scenario of `unexpected node downtime`. diff --git a/docs/concepts/ipam-performance-zh_CN.md b/docs/concepts/ipam-performance-zh_CN.md index f74106300a..36b03db5cd 100644 --- a/docs/concepts/ipam-performance-zh_CN.md +++ b/docs/concepts/ipam-performance-zh_CN.md @@ -8,16 +8,16 @@ 为什么要做 underlay IPAM CNI 插件的性能测试？ -1. IPAM 分配 IP 地址的速度，很大程度上的决定了应用发布的速度。 +1. IPAM 分配 IP 地址的速度，很大程度上决定了应用发布的速度。 2. 大规模的 Kubernetes 集群在故障恢复时，underlay IPAM 往往会成为性能瓶颈。 -3. underlay 网络下，私有的 IPv4 地址有限。在有限的 IP 地址范围内，并发的创建 Pod 会涉及 IP 地址的抢占与冲突，能否快速的调节好有限的 IP 地址资源具有挑战。 +3. underlay 网络下，私有的 IPv4 地址有限。在有限的 IP 地址范围内，并发创建 Pod 会涉及 IP 地址的抢占与冲突，能否快速调节好有限的 IP 地址资源具有挑战。 ## 环境 - Kubernetes: `v1.26.7` -- container runtime: `containerd v1.7.2` +- Container runtime: `containerd v1.7.2` - OS: `Ubuntu 22.04 LTS` -- kernel: `5.15.0-33-generic` +- Kernel: `5.15.0-33-generic` | Node | Role | CPU | Memory | | -------- | --------------------- | --- | ------ | @@ -36,19 +36,19 @@ 本次测试基于 `0.3.1` 版本的 [CNI Specification](https://www.cni.dev/docs/spec/)，以 [macvlan](https://www.cni.dev/plugins/current/main/macvlan/) 搭配 Spiderpool 作为测试方案，并选择了开源社区中其它几种常见的网络方案作为对比： -| 测试对象 | 版本 | -| ----------------------------- | ------------ | -| Spiderpool based on macvlan | `v0.8.0` | -| Whereabouts based on macvlan | `v0.6.2` | -| Kube-OVN | `v1.12.2` | -| Cilium | `v1.14.3` | -| Calico | `v3.26.3` | +| 测试对象 | 版本 | +| ----------------------------- | ---------- | +| Spiderpool based on macvlan | `v0.8.0` | +| Whereabouts based on macvlan | `v0.6.2` | +| Kube-OVN | `v1.12.2` | +| Cilium | `v1.14.3` | +| Calico | `v3.26.3` | ## 方案 测试思路主要是： -1. Underlay IP 资源有限，IP 的泄露和分配重复，容易造成干扰 ，因此 IP 分配的准确性非常重要。 +1. Underlay IP 资源有限，IP 的泄露和分配重复容易造成干扰，因此 IP 分配的准确性非常重要。 2. 在大量 Pod 启动时竞争分配 IP，IPAM 的分配算法要高效，才能保障 Pod 快速发布成功。 因此，设计了 IP 资源和 Pod 资源数量相同的极限测试，计时 Pod 从创建到 Running 的时间，来变相测试 IPAM 的精确性和健壮性。测试的条件如下： @@ -58,31 +58,31 @@ ## 测试结果 -如下展示了 IPAM 性能测试结果，其中，包含了 `限制 IP 与 Pod 等量` 和 `不限制 IP 数量` 两种场景，来分别测试每个 CNI ，而 Calico 和 Cilium 等是基于 IP block 预分配机制分配 IP，因此没法相对 "公平" 的进行 `限制 IP 与 Pod 等量` 测试，只进行 `不限制 IP 数量` 场景测试。 +如下展示了 IPAM 性能测试结果，其中，包含了 `限制 IP 与 Pod 等量` 和 `不限制 IP 数量` 两种场景，来分别测试每个 CNI。Calico 和 Cilium 等是基于 IP block 预分配机制分配 IP，因此没法相对 "公平" 地进行 `限制 IP 与 Pod 等量` 测试，只进行 `不限制 IP 数量` 场景测试。 - | 测试对象 | 限制 IP 与 Pod 等量 | 不限制 IP 数量 | - | --------------------------- | ---------------------- | --------------- | - | Spiderpool based on macvlan | 207s | 182 | - | Whereabouts based on macvlan | 失败 | 2529s | - | Kube-OVN | 405s | 343s | - | Cilium | NA | 215s | - | Calico | NA | 322s | +| 测试对象 | 限制 IP 与 Pod 等量 | 不限制 IP 数量 | +| ----------------------------- | ------------------ | ------------- | +| Spiderpool based on macvlan | 207s | 182s | +| Whereabouts based on macvlan | 失败 | 2529s | +| Kube-OVN | 405s | 343s | +| Cilium | NA | 215s | +| Calico | NA | 322s | ## 分析 ![performance](../images/ipam-performance.png) -Spiderpool 的 IPAM 分配原理，是整个集群节点的所有 Pod 都从同一个 CIDR 中分配 IP，所以 IP 分配和释放需要面临激烈的竞争，IP 分配性能的挑战会更大；Whereabouts 和 Calico 、Cilium的 IPAM 分配原理，是每个节点都有一个小的 IP 集合，所以 IP 分配的竞争比较小，IP 分配性能的挑战会小。但从上述实验数据上看，虽然 Spdierpool 的 IPAM 原理是 "吃亏" 的，但是分配 IP 的性能却是很好的。 +Spiderpool 的 IPAM 分配原理，是整个集群节点的所有 Pod 都从同一个 CIDR 中分配 IP，所以 IP 分配和释放需要面临激烈的竞争，IP 分配性能的挑战会更大；Whereabouts 和 Calico、Cilium 的 IPAM 分配原理，是每个节点都有一个小的 IP 集合，所以 IP 分配的竞争比较小，IP 分配性能的挑战会小。但从上述实验数据上看，虽然 Spiderpool 的 IPAM 原理是 "吃亏" 的，但是分配 IP 的性能却是很好的。 - 在测试过程中，遇到如下现象： - Whereabouts based on macvlan：在`限制 IP 与 Pod 等量`场景下，在 300s 内 261 个 Pod 以较为匀速的状态达到了 `Running` 状态，在 1080s 时，分配 768 个 IP 地址。自此之后的 Pod 增长速率大幅降低，在 2280s 时达到 845 个，后续 Whereabouts 就基本不工作了，耗时类比于正无穷。由于 IP 地址数量与 Pod 数等量，如果 IPAM 组件未能正确的回收 IP，新 Pod 将因为缺少 IP 资源，且无法获取到可用的 IP，从而无法启动。并且观察到在启动失败的 Pod 中，出现了如下的一些错误： + Whereabouts based on macvlan：在 `限制 IP 与 Pod 等量` 场景下，在 300s 内 261 个 Pod 以较为匀速的状态达到了 `Running` 状态，在 1080s 时，分配 768 个 IP 地址。自此之后的 Pod 增长速率大幅降低，在 2280s 时达到 845 个，后续 Whereabouts 就基本不工作了，耗时类比于正无穷。由于 IP 地址数量与 Pod 数等量，如果 IPAM 组件未能正确回收 IP，新 Pod 将因为缺少 IP 资源，且无法获取到可用的 IP，从而无法启动。并且观察到在启动失败的 Pod 中，出现了如下的一些错误： - ```bash - [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader + ```bash + [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader - name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175" - ``` + name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175" + ``` ## 总结 diff --git a/docs/concepts/ipam-performance.md b/docs/concepts/ipam-performance.md index 667b439409..67128512db 100644 --- a/docs/concepts/ipam-performance.md +++ b/docs/concepts/ipam-performance.md @@ -2,7 +2,7 @@ **English** | [**简体中文**](./ipam-performance-zh_CN.md) -*[Spiderpool](https://github.com/spidernet-io/spiderpool) is an underlay networking solution that provides rich IPAM and CNI integration capabilities, this article will compare it with the mainstream IPAM CNI plug-ins (e.g. [Whereabouts](https://github.com/spidernet-io/spiderpool), [Kube-OVN](https://github.com/spidernet-io/spiderpool)) and the widely used IPAM CNI plug-ins that are running in underlay scenarios. This article will compare it with the mainstream IPAM CNI plug-ins running in underlay scenarios (e.g., [Whereabouts](https://github.com/k8snetworkplumbingwg/whereabouts), [Kube-OVN](https://github.com/kubeovn/kube-ovn)) and the widely-used overlay IPAM CNI plugins [calico-ipam](https://github.com/projectcalico/calico)、[cilium](https://github.com/cilium/cilium) in `1000 Pod` scenarios.* +*[Spiderpool](https://github.com/spidernet-io/spiderpool) is an underlay networking solution that provides rich IPAM and CNI integration capabilities. This article will compare it with the mainstream IPAM CNI plug-ins (e.g., [Whereabouts](https://github.com/k8snetworkplumbingwg/whereabouts), [Kube-OVN](https://github.com/kubeovn/kube-ovn)) and the widely-used overlay IPAM CNI plugins ([calico-ipam](https://github.com/projectcalico/calico), [cilium](https://github.com/cilium/cilium)) in `1000 Pod` scenarios.* ## Background @@ -15,9 +15,9 @@ Why do we need to do performance testing on the underlay IPAM CNI plugin? ## ENV - Kubernetes: `v1.26.7` -- container runtime: `containerd v1.7.2` +- Container runtime: `containerd v1.7.2` - OS: `Ubuntu 22.04 LTS` -- kernel: `5.15.0-33-generic` +- Kernel: `5.15.0-33-generic` | Node | Role | CPU | Memory | | -------- | --------------------- | --- | ------ | @@ -36,13 +36,13 @@ Why do we need to do performance testing on the underlay IPAM CNI plugin? This test is based on the `0.3.1` version of [CNI Specification](https://www.cni.dev/docs/spec/), with [macvlan](https://www.cni.dev/plugins/current/main/macvlan/) and Spiderpool as the test object, and selected several other common network solutions in the open source community as a comparison: -| test object | version | -| ----------------------------- | ------------ | -| Spiderpool based on macvlan | `v0.8.0` | -| Whereabouts based on macvlan | `v0.6.2` | -| Kube-OVN | `v1.12.2` | -| Cilium | `v1.14.3` | -| Calico | `v3.26.3` | | +| Test Object | Version | +| ----------------------------- | --------- | +| Spiderpool based on macvlan | `v0.8.0` | +| Whereabouts based on macvlan | `v0.6.2` | +| Kube-OVN | `v1.12.2` | +| Cilium | `v1.14.3` | +| Calico | `v3.26.3` | ## Plan @@ -58,31 +58,31 @@ Therefore, we designed a limit test with the same number of IP resources and Pod ## Result -The following shows the results of the IPAM performance test, which includes two scenarios, `The number of IPs is equal to the number of Pods` and `IP sufficient`, to test each CNI, whereas Calico and Cilium, for example, are based on the IP block pre-allocation mechanism to allocate IPs, and therefore can't perform the `The number of IPs is equal to the number of Pods` test in a relatively `fair` way, and only perform the `IP sufficient` scenario. We can only test `unlimited IPs` scenarios. +The following shows the results of the IPAM performance test, which includes two scenarios, `The number of IPs is equal to the number of Pods` and `IP sufficient`, to test each CNI. Calico and Cilium, for example, are based on the IP block pre-allocation mechanism to allocate IPs, and therefore can't perform the `The number of IPs is equal to the number of Pods` test in a relatively `fair` way, and only perform the `IP sufficient` scenario. We can only test `unlimited IPs` scenarios. - | test object | Limit IP to Pod Equivalents | IP sufficient | - | --------------------------- | --------------------------- | --------------- | - | Spiderpool based on macvlan | 207s | 182 | - | Whereabouts based on macvlan | failure | 2529s | - | Kube-OVN | 405s | 343s | - | Cilium | NA | 215s | - | Calico | NA | 322s | +| Test Object | Limit IP to Pod Equivalents | IP Sufficient | +| ----------------------------- | --------------------------- | ------------- | +| Spiderpool based on macvlan | 207s | 182s | +| Whereabouts based on macvlan | Failure | 2529s | +| Kube-OVN | 405s | 343s | +| Cilium | NA | 215s | +| Calico | NA | 322s | -## analyze +## Analysis ![performance](../images/ipam-performance.png) -Spiderpool allocate IP addresses from the same CIDR range to all Pods in the whole cluster. Consequently, IP allocation and release face intense competition, presenting larger challenges in terms of IP allocation performance. By comparison, Whereabouts, Calico, and Cilium adopt an IPAM allocation principle where each node has a small IP address pool. This reduces the competition for IP allocation and mitigates the associated performance challenges. However, experimental data shows that despite Spiderpool's "lossy" IPAM principle, its IP allocation performance is actually quite good. +Spiderpool allocates IP addresses from the same CIDR range to all Pods in the whole cluster. Consequently, IP allocation and release face intense competition, presenting larger challenges in terms of IP allocation performance. By comparison, Whereabouts, Calico, and Cilium adopt an IPAM allocation principle where each node has a small IP address pool. This reduces the competition for IP allocation and mitigates the associated performance challenges. However, experimental data shows that despite Spiderpool's "lossy" IPAM principle, its IP allocation performance is actually quite good. - During testing, the following phenomenon was encountered: - Whereabouts based on macvlan：We tested the combination of macvlan and Whereabouts in a scenario where the available number of IP addresses matches the number of Pods in a 1:1 ratio. Within 300 seconds, 261 Pods reached the "Running" state at a relatively steady pace. By the 1080-second mark, 768 IP addresses were allocated. Afterward, the growth rate of Pods significantly slowed down, reaching 845 Pods by 2280 seconds. Subsequently, Whereabouts essentially stopped working, resulting in a positively near-infinite amount of time needed for further allocation. In our testing scenario, where the number of IP addresses matches the number of Pods in a 1:1 ratio, if the IPAM component fails to properly reclaim IP addresses, new Pods will fail to start due to a lack of available IP resources. And observed some of the following errors in the Pod that failed to start: + Whereabouts based on macvlan: We tested the combination of macvlan and Whereabouts in a scenario where the available number of IP addresses matches the number of Pods in a 1:1 ratio. Within 300 seconds, 261 Pods reached the "Running" state at a relatively steady pace. By the 1080-second mark, 768 IP addresses were allocated. Afterward, the growth rate of Pods significantly slowed down, reaching 845 Pods by 2280 seconds. Subsequently, Whereabouts essentially stopped working, resulting in a positively near-infinite amount of time needed for further allocation. In our testing scenario, where the number of IP addresses matches the number of Pods in a 1:1 ratio, if the IPAM component fails to properly reclaim IP addresses, new Pods will fail to start due to a lack of available IP resources. We observed some of the following errors in the Pod that failed to start: - ```bash - [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader + ```bash + [default/whereabout-9-5c658db57b-xtjx7:k8s-pod-network]: error adding container to network "k8s-pod-network": error at storage engine: time limit exceeded while waiting to become leader - name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175" - ``` + name "whereabout-9-5c658db57b-tdlms_default_e1525b95-f433-4dbe-81d9-6c85fd02fa70_1" is reserved for "38e7139658f37e40fa7479c461f84ec2777e29c9c685f6add6235fd0dba6e175" + ``` ## Summary diff --git a/docs/images/ipam-concepts.jpg b/docs/images/ipam-concepts.jpg new file mode 100644 index 0000000000000000000000000000000000000000..2a2cc6459bd7e49badeb398eb63661d1266188b2 GIT binary patch literal 71444 zcmeEucUV)~_FycaQl)pil+Zz?DMh@}6PgJ-2Q=kn?N-vxFZD@bNLCk{|R@v>-~d1mO`V7bocvF*AMu?F`bi#sWIh#p7P}d z_yC{)9f0}|`69Ht#Y^GKlk#x^xC5L3zW_i0 z4}b$ehJwigt^lqA6i=oB8h|sWf54CH848@GKKldCQ&Uq>UpP-gbK(4j3p8|$bTqUK zv==VWGto0JUcAV3k%o?$h4~^21;6;CkW)Y2JahIu<;9D%7icM||C?~~62Nr+bmy7A zGp7UrrNp`}CPp6!=qG0QI@EROio}K6QbT{o6$Vfa)}*Vw&?8sOip9 zQBmBL@<>H}j)|FtRp8F~t4VAeoPxI?X67Sg_Ui%ID$?+%knA;63%^86vbtAxPHy!D zIjD)fqm%bzJfT9^*Z~EU*U+j|)bx4MrcJ4Vhf?z?O0hoy)sHHu&rvc}m;k3woj!G* z=G56!XMb=>@s87Hn9fo$vj|+hgBoIGy9I%IJwEW;Xhx)ooYUJ(CnO- zYV%?GflM3Y8~-nkH@5^nHC()@G(G$51ke~~IHNdEc(@g_B0s$4JpHeWqLf6z7}qGf8lGKe8Xh*flcJ4^v#c5A#)l(70I1Zi);md zf6LJaC(QNNy#Da(;i{*<#_{&0zltPpwt8^?;M&1JW*^ziVYl2_ueeo* zr;nK~>lGg}(a1RBpOxt*U7pDtfEF=bSD9&g1kh)G{%>sf{~+t^eWno6@tf=LmSd{u z8V_kHt-~zq696&q1aQDEzF8jFCIZcLGF?8bxXin$u;fBf*cHY0ReDN>f6DNH-2QwG zn+MPyIf}3`D@k*n+$~(zUh#PTS%P3?qmc4G8jC3=M-HUMS=mzID|*W!hX%hHSwvS4 znKz9UQvF)EMrmupTp5b^%vo7F{$-3o3+Sx zVDypriyAs4S2B&{kR(g=6TffnoEvwC6vBX3b%h$p5<0@Ns5*NyjoJkum*cgG_L#Uk zrZSP=ep64BStzQ}3N4J&B!04d=w81jqnW6_@7kRRatsqA<53*E-GL2=DQRg~H8yxg zaRHMseCnHKuYE+Mnbf*8Y_*o-*GS(l9MC8G5C(N81p#_dD~We+V?)J~iu;v$U@<kQA+?{&=rT|9@ciW{7k*lKt-5=S>2tE zhD8l0(zt!mpZ2FWgloqz;KU2lFl%BlK;t|a9x)`?y|RD7Jt`!!!bcv5W{0D!QBQc+gIVw~jkDSl$t~}NCOeeFrzN?ET)gO+jN+iWx^5F5} zt@h}lU%nOQGp!)9IwC>8>CojK8|8#Uq0o_WBHG&8iGs2e5E9}sC=YCpFEZ?_QSJzy zaq|eMtS)R{eR2Xg0mOFBO*Agrc-8B8n$oJ(8l3JO;$RY7@Ia?4JG^p)F1Z}Vj>!c^L;TSdjCF84;N+RN@ zt7CiR)Z=KN`n?6a1--KGoUL`NeRW>^KWGpCrpojCg2zUsA>@&eLt0k<>8#zD&cE7D ze8@ECO2y2ML}%|;f$g-aMM_htI|1k-e_UQpb0TBa;gNmYR8a>vb=p-{dW&X#mDvz) zYf_1rE;j88N4XUjyXEnUcy2~eb#Y%`Y^78s=QTw>YqOF`q%EhU+Z5b@Jw|QLhTAc0 zL{NcWqkK-IIF0VQuDsHv4(%snNqZ=KWyO96j{}#P47j(Pqp;tSx6|VNNVKoDO6h!D zgv>}5!)7IsOcVO8v1YN9x3=VZSZ%Mti+Sl}MI1pnu>wNWDk|9Gwal@tp^`~YQ@hb( zijc3gF?(zaV&!hrlCMgW46t8Mg__~7Co$8-4iU6anb4;4@QHS(I83U0QJa!V#beT4 znlIJD&DwaBg?=)EYev5YUY${7A&wZYKXVmHLxQE1@^7f~XU19# z+|DBljTN$TteQw6nsP86chHcl>o8%JfS7j*} z&cyyO9`yQ}mStIWx$XitS^IAAQ1OI6SnLbydo-eA-BhA&F&r_Q7VJJwkgoP);t%EN zK+6fr^{U3GhK?f#zG+iaSazT*XE5kanblwQ|HCuJXY@po-_pPeOfo#(Hzee2W10kMiT)TKo)3LOG=M8!#xn8bVNloIeQ+`|LwxgIx z?j_zG;;#tf?fS@VB}{O;#txP>o8;D$sddS9=D$A7YjdBS8;S-ij>lZPMWl8j2b$aT!Bk z)^6;{1Sm46gl@zsW&4oB;4tPuu{cqzwzf6Z9{#)H=#H0Uj#{VsM z6|f+MR?#UuzwPl|eX}i3Lf?-4fx6PY#Jg|qbO_JL9}IJQtKv#!*V%jDJRhzStFVlZ zW-|wQc8>}zpKd$egTAD+oc}YBV9#w}M%9Nw;WGH)OdQFdXnE(1P*|tVtE1>#oiwiZ z-cx$TYE!9xBz8v%BeH7R+DhRcDmJiEN$mwMb_Q>kTwNtpD$eR``-#N}m&sIRTI=>z z$_OPNX=&&7m9Y7eZnP2Larzg_)&0zf?blZ4w$4p%!( z2d>O3v~FW^M#RrEabWsIu0)ps`l94gc4A0 z!JuVtqy;V)Bw@94-f6m^R-X-o=bN)ZjPpt-v9Y2s#6=zLM}Xbdvi~)Moi1E+_-xT( zl=wYjeYL$%q7-|RE!Oet&D!A72NWOsH$YQ_uUT40Z9ldT2_o8${$FyKpBN}C8HiI8 zILw-Cv6!LMXIl3;zwKt?_iGzGvHzaapUMExD>&~|Zcc*2?%Mbw9K)&&3An^5C&Vor zN(@b6!Z{~OeXA7Oq0Qwon37oL#h6ako#NOcgxi)QmXHCP2zJ$^}}ZAWo}G%}eLd(+1T^+*p!Ww=h~rPEJt3>AzVo=-2Zt52a-HV%+^3 zL#p}ylEd|feDW_o%LqR|YxkGj3_r;s&-UQ*Q*MiU$;`LUm}}M}W0H}dycY?h0Z0{O zkAvrjJ(qm5r>d31#g!Xfhpnzxdt_vW&79VVlFtFB!+!n4EQ92&Y4#YU9w+DB=Py&< zSm|yfq^j2C1R>5zVzHG%LQ=g2UG`u8u2ToWTu4ztGgUmEJ#R?tU)YB2cdh1O`GIX1 zK+In+-+8M?*JMA9EzdpYGRR%WpQBluQ3*?RK|ehQsm= zGd>AE?wxF-XS1K3daWRZQuS3Y+Pe(`R}I7=IDzUkul6FMm^vnqiR-UIZGZhg;1d74 zrr}9d2oCeoGMNr0ozCZD-~k8mVpKzmBV>{>`+MqguZ$yA$ICzH5$w?aCO(W+ckJ z1Xx)A+&uof_$gDXEba#yUvE|hm)BClNRiK?QDlhdzUSCAPSW2j{U6I_d)W7}mjz+{ zUVZ>?$o{BcE5P3(O(lamZ8z$*hN#qfiN`U780(*xxg!^}!ZUs~Z|-Y~=Yu{2*T|V2 zpC%vrjg%3}x0)cJM+AiqrW(CO=bMbTwRd-p_-h`c9&wV|^&IX|Y7ibAio5B!R&9~Il5HGCkt54Y{nfLL3(4Ohq) z0fiHQOsM6y*a(C3AtbJl(FcO;&=Y`H$<|WE?~?M4t%B;0lu`yNBRt36zd-m(dX(8{ z8RQw%r*B8$^3$F=Vz>7t+U}U|J5!Xvw5B=9FN(makc}tKo(kj*Hzx2c_K5CK^_syy}X&%KJ>@=^ti^W?}bIVcyiYtSBiu zJ!Q}}YX8_SSpu^(maUlYpkg?wN~A?13^9${7sQ2t;!EpNF;ItVH}$_!HT+stI%@R^ zqFd+c`|UI4`x8~rcX1t5C9?oQ54)kYcwN)=784c>`6N+uzM3!r5m&LQTIbtH@wDz# zZ5g)BXV^s11oJ9!lf(Q92b1hRdTPF{NCX#ejM|oLRryZ4@QeFG}R} z%S`di>+)fe`XH5Imt!oA@Ed?xj#|ckVC{n&UgHy!FG5raD9@-4gTQF`mtj(Y!jbr5hN8=)XAtku?6 zbYG9xvk}yBUm{U2uQUSrh5cQk-6NuY$?@8+sgVpJ@V>~{xXU=)W^D8p?aaU8^S`9CHVdm%woOt3;t2q6!0~HNxzj9Gr0GrA znKhQPdAgPpF&@tGxBYV;uCV_Z-RQSc)^;^T{tR!%3K9e0u5HSs?)uFuM{{4JDUq+w zBEzz7$=+F7=0-zIz-h6o&t=k!%szXkF{kF{1KV9LJ#@w9+QkQ11gI&YR#C^K;e4io zT?uKm7(>Zs1BTvgv-Rzi8bqZ{S$8t;43yJbNQ5jTF9cEgB@&Y_JBS8jY%b8u))4m) zvmQWIl;WbIVcvW_$&*JU3V$yzP*j+SBU&rVa^-=hpplL{AEsS~yQB`;MljBWGI15Q zk>w=leB|s^t61vvkkl@O%Wi@~vg>jJL@A)j*175vfO1??WZoz~&hdjyyJS4yxOHyp z^IKKKox=U?+lysaEszuCv(lR~+})LO&tlQtza-~&0&xd9mvz^!1`Ih{R!ObxMbfC) z#zAEt-%h%WgdikL4x^T=iUXMI#38ct#7^3de&xup3`y!MJ;)*1t)yQccJE*}27XzG z*4Q+l-l!zxT~#XA=y3wO&w?)mavjVV-Ec3BKe#0Wb}W1faeVec{Ji}Q|NByc`kGad z+aSEE?sPVtEc(9TVE0+dC}l^F-VPCRkJXgupVMrlOJ|lpotRz;_dS=IGgKOr>XuX| z2UK48#SAD5nTFINguLhv@^XjgG&M7XF>J(8&WsT2FVaDyZ4wV@RJ&MHZ}`g+Rcy{D z+zb@F(|UKnDy~o>a>fu-fYAmS-QIV(aCb^oiM(IZFJ(d;r7^*KCS(pvO)}_W=7M&l zJ?a-o>|n!Uw4eF~Jsa90WB0t| zx+z9_|ixhXe^KK(euX@$|yE4XZs-OL3{#}KB z!ho;jkvi6H%7EkIvTRPZj9%|N%LGRd)85zL0A~ZMe{Tm=^j=MsZM02FCx9=Ut0w@X zNTKm0=Cq(Gzaoh;f&Du(@@w*Sw0GcoMEE{ts=pFKnKYJ=lj`b~UC>rEPx~&V-C%s& z!g&H%5{$pm5kv>>UyS0&h9V_D^0hFl`027)G8#)NP6~3{C>T4Xxv48g!fb);u`>$n zl_My>&qpZ;kMVlD6F_2CNPSh(cf0PZbuSpUl7edVs?BfbPnFI1N*uQ5Rlivke&-Qh z!yoOnKm9t9NgAcNCL1giEeQ2y_j+d&zbW!mBjw1@gqzB+(gdEL;ohUZj_mi)^H_S) z-=uJ^TR20+Qb^8Br!a17B)HI^HE^Vy%3yGF13OK4RgE9XE!2OXbe_B{&3N63G|koP zI|!HUMwXf3KN;e#KfXV@Nz+8qZdmho=eA+qC~cmH=BAkfb1H8{=#lpgFr}5EeGn7~f zJv@uA7`E2%$Yx-11Pk@U>Pq`M>8#=P*03|Ou$-wkQ~Ss7=b^C2Kq8ZX_^57>CPq(h zLqEwKgzcw>-*TK!jk+07G~OX%g*5l{`k-qk0I`h|0KW}ZI-WB2iy`~E1Wwz$r+F%yZ-Pzs2>&>aU7zfqEErl4i>ff} zhSAESm_Dn0o?@k%m0~??<;{HpM%Qkm%&}3s{$GScVSxl}5WqBtyZ$BcHSMZi| zkkx$_FQ-(Eaeb9xDHwgFO{Ht(bCOQHfo|OG7?8NQalKzIIGjU*{2FE)m3(M%PIMgRlm|35EvTyI&Q!jnN=+i2`WxjvSIiG0L)B zzJZJFcO`7FfQ7qVQ)e3ga*-7&CfGi21X0!WxVZ;Aoa#lLJncoQu9E5oK^dcjdxZqA zK6RR>FR>>Vj0Wki=WZ8}3&W(+(HjUQs}I$L?q~ZL_I&JjSnLQvP5IEaxDKtRvgqK2 zV0}`XcTM?$imjibA0o+6ed_IoRGXh`99uh1+sbeyRL$|TV<7GlGdB0)=Q|uewJJi) z!+m*M9O6!XD&T0NblIHp+nU~PpsO{|Ip_tLtl~6eZM4nQ5R>We*0R&-r9VkmqU;FX+s0+>Gmv>d(d80=hJ zdndlCXPw#p_9&obrE5Svw5X@o#qGxO7{S_zTYR7-TGUnLH8EIC>+_KjAIsa#+Im@T09^t; z*9BZI-Lkw)gGlWU#|I5>Y!khQ3`)DEwI6CmR+?CHn>@IX{{WY@ZGM>hE|4EauxJ17 z-s9x8RF%x_C6bh_g%e9Cj*TEN)bDYv6pTiYmvmSST2tm%;BxCuBR4OjB-NU8KUXLi zEH##=j?!t6_pg0A0Ytp-S2bkbP)pshj(P-Oz{ma1y1U7W$H#$X1qJ*14nqOrAP_gk zQBwRnQ11R?EA1S@Z-zQB=tHmbi~#VH6OE;y5iR(6y{u zo{tTGdKBYk(`?oJpt|SICtQr0n_cIZ&#RUHMw1U^S|pOCS*Je4Rk@ILV-&Pd1f7?u z0o}M005GIS?Hp>ksMliWVYQQN?KVs=?s_74FsBcrGp4P6nK1^-jPUjI_;5dKR=Xbz zF6_t2pJL1&x6M@@<$m!kJ#rzc)`wTMU*w^Fz02}8&}Ez)tf)e2vnTE)Kvr$PN}zWB2aWU1FfN zaqw*Nxx{$A4{Oe^q3f3qb%n8`p(cq^v9Sq{QOVEzWpgvwW$134DrpeHkU<+6%KM-F@VzdL8+4V+(%#_D*(fwDnQ`hqT2rUR$9f zlI;BuT!cYv1h+{I5WNVL#y!8F|6{Ie_1DMC>ogkCc2Dd~6VngBJi(Fe|WIV55ab5Lh?r^|oYj zW4OG-Lo{HUkqq_+o3rNkJEFf9X8tnrqFU%yFmxM^=VpHi)~Y`^&G zY+mNmVMjv6)2^2DC4nzKBBZwU%hREqnp=vf@ysx#srD9j1;a}AXCFGO(reIPW#rGz zy4?ceOOhnPNvV&h9KmQ6p<$u#gL_+4ZBmNAhbH=2h7jAo0)SD_i@AWU5BgtEsR z*7asW_DPB@GOGp?ICZIHV_-|aZOb~w|vE~kE6KyOX-+fC7P&PbP1Nb%DidODxa6hzd$EE)T0(bb*WLp zE(L>$vT|L0k$oNxcdEHsYx1UUfBH6K9|iGolSvy^*^1k9dyeiy5$MQYx-hwl$Icx>7ZDpS&{ld9_@=Nji~2e_!Y^)>Jg z>&g?WN|mIQP&jl|@tEbRPr3d4A8vS>7;DP5(mgnSv z#-_Ic2jjIh_*byr#m1DaAM-cz-+!Mz++-@swC1~%K4pCk+CnCV9R)twV71f!qL|cB zVz*xP`na(?h~biyq=R!~GtI39 zKvxG7<%fNj7s#e6t-2gCeJRx2B064S)n)2fcUkm&vBbkISesJyku1Ts&}0N^|Db@b zRP&6!?&t1m8~N1X^jvi9B^X)J)LpSv;`?mq)L4l{6>Hvsp}q<=NKx$`kA|gli64T8;0zh!@~%j zw8d*14h8av-;mUMpj#`JxXg9s&Trx3)8hFfaO{n@p4zEP_!q5L)-R@ENz!_E(P-#} zQSov8(Sr^*%dAEI2VY}p3T{T3-}g;klvC(w=E;+*u)K!TXd zlMCKGFDX5Q9dq0%SJ9?1gnhp*>=CTrSC-)+5$kjc5>{mc*~tzvw=FjivII zVEo>K*SZ0Fw~Vo8Oe-jHR)6LB;`6XR*(Rx--6*4I*kRJ}EmL zWkeS>Ly2N<^f(wlePr{__0cFt?n~_XN*9JM8C&XYKWpjKixFP8loTXu2Pmk1SgJo!JPitPYW%G59NTl-kSR4z9LP+%*UJC4tx*r> zZ4_ko3(2U2r-mExiN!n#f})m_2F>;rtVQK0cI(FCl}P;y^Qq)DCIJ}6{7}X2QrQw1 z^Y(Z)E103(W4wwxa+kUUp)8~uteR}8&mL97A8)|IAg=7Kpr|ja3;Ar=-B>Z4-b5tp z`9CvdCfp_tMwbD7B&hghdBi=7qa^)ig<)ekASR+EZpkuUA%6krG%cHJ;Ks`Q#JL1K zhd?wgf$#=;eY)2;5yjZ*-94!(^*BkXG_!6hl(MqqS(yM2RhjJ3OceB?p57)B!3+Ve zUZ|OIPpVOeCO>mT%d4Q=w39BeaZY4IGXkC4{q5#DxL+h)pRLf+d;LH-N1QbDse}9^ z536^pGgrrbILEI>w^#VCtQXeI)=B=UjJQfeaz%lu?jDzPe+s$h{$&x( z@M%3)zt>PB*zgqd6Ud4q2JO>T!S%=bMP?i=tgDrfwB;iaV~}HoS$V&|CrB@IMgle+ zL!->;L<8%0lT|;D)+P|asxeUoeFrHGK{q6As29vL8$;Bi#o{(n8iTHt*_7Xv9I!DT zT!<{LF_NR+oQpfeNqx}Cm3|K3y)*QCwE7XT{uP`$AOM9C5EK);v&)jew_p6ewbt-P zz>B$Zp1tm`(NBmS<$zHZM0$kYg)CoUY(I4UK)?_kdz8Vc(-RGEWxgCva(y}ov=>dg zWV~NyX-ck)Vbe~P_*;KfAY?7%?eW&)Lo^yKjB?a^85zNODA`tcTBhEN>kYPbl_wnduj+xjNs+yz1c~rq1zWUkYcuU$7ErP& zLM$*2cV}Kn_pm9xWlW=g zR|h@3(523+5hY(jLm86ZR0@pP85{3>a6fxKtW1kkv!)2s(!;pdBKWUT zc5g#i=oMl0qhrp_&JHEg(H~M|Q=gm6TJ<%g-T_;#k%Py8mQ!xlCy3PbNuX7d22#!$cg#rZf67CmjO210#!d96O6a78pM@pBf$L1BCGoA*e-*Wtw zp%#ipD~NSP#>mNSUwksUAyJR@vM(6X0xc0Z&ZIjnT|T<}{RA+3=y53okNoU)XSCJs z#Z^J42B-Ns?&)!$+GFbke&6+x7=&TL@N#k7IM3aM?2H^b45=18AT(S@b&Y9jJx|E& z)A4wt_`LuwKdZd@BZD4!7Y5KP0|l|z3QJejXxL0($Ra&G9-Xt0vn)HNNVq@$9wc@G z_^`Jzdf$E0rF6WxKh5!zuLG?~>iFcB#t*_oVNDQ)=WvZLmW->WaS0aVo_7ptKrbvQ z(}J&Cd7Mi@!b2z;pKXh~ed~;5pzH=D!4ZotASy6dha*(cPtz9l54GCd3_qAon&+6h zLcE;cR>KwBwP|%jk$L$>kgYO;gjFP@zM8C=2)c&DNezUad1c7P3RG~L4smfQ+5eW- zQ{h#wzq#qi-Q{hM+(^EFoIrirCck#SUOb!Z#2mRhEyL|Z@Zz%~S~f;lxW>VOV$&N^ zeLM3we!k1BpL0qAE}wU@KU5ZuKJ!ImTc*xeH@v>ccllREPAf|mvEit7!4t{(%JFHM+al@wyyHdS1u9^=-bm zU&i^yc|o<hyz<^RR`s=;{bm?A!BfK|aG<^|pD%R1U#Y4mE!q{NK+sxP${BWSCcU4U z?J^I(&KI;tH0N|-;*XpNMV|-ulC&yxvIKn(IqOFyM_q{*D zt!}fX?i@5N??Z1Jfcn+XE$633^e1{wb~h%#C91^ddD}P5>=AWhJ1=8mJE*2nSYYWuyvZfqVDfA{xg zXKOVY(q)UG?@YFz`2UV6e{Af2n>e|xbX?qiaARCH^Iwzl1tzwLRxT-5a=iOu-E;r< z9_!r$338Kz&IhggFYo;w^II`I%p7>O-e>o8)<=Bv$7*oLG(HSeWY3HF3lmBB<(Z_Q@HR4)qg5mxuw&goIPDclOVXcqw;2UjSxDdo4pel<%{N z8D`LRYmXEFna#9>cZ(~B@p}YK#|OH~JEFlxE|f{N%dbAxC4Ky{o?<`$$&?t>kuVYc z<P=u;gObqsX#*jDN=%{!kvDWz47hr7kU6lyQm`$!>jj__(S(IC|s471ZICpwhz_ z6q~83F&9-1Q>RE&uJ2*#WwZOXmQKrk7P)afq_qVvpCZtl%-i$tKgs9oFmjrhAx;2N z06>D`%irB$DgDlUmlzIk$e3W(A;#+dCih1RE)c~D>3+kMF7;q9 zGFVA9eP?o&Y!gV&_@qQ{Og^~Hypdt!9QKO*i`{vp4Uj1DOCWEvkmy1_%sbXk@w<7* zzmt&XZ&ttavo+gW-xWdk;x3i5ZJqiH8^@_^-&fNpx*E*r*Oz}`i3#`K&^ah}dqcHi z-~{lSUI{@YG+Q`vKvwWw;9g9^-?dGpFpP>Z&PhUnn>>rvHL>e^UB2Ohow;He^4I5i zle?mJ5x$-A2ltRE*^?nfyrwE!XzeU zj8-QBad{o zLQzrr!)T`KFw96z^ktBz`agHA#{Z^kb*lS1m^G-^m)!^q(To<#{CmrkIuviszx(E? zV(cDLL8~v5n73{J;et}tbdc|cZOB69z|G6%r`WLGnJjE!Lqkck;||A=mc$np(QG^G z+C8)HQy&4AqCWi@PyWgMw0I21c+jC#W4T=2!;u4^C)uq5`_`e!qdfh-Q`K+@cXf~- z+{|(nwfrn8x`>d_?!cvH^zl+r7C?<@H-GP1m|a_a!pyBt351)(Mu}q}Ie3Ax&-E9S zL`s%Oe_2->t>M)3euP{@p_us@%J8QNw+_YZisglIpoEi9Uk{&Bp^JC8DCmxvUrcmg@Yj??pcZE3Lc~ZGqWc`!EB6}n-BF$u!9sCRq zrHdQoIAeLxVNKT;UUAg)f?*`DCgOUwzXhu++j()PdbXekV*R=i2T@>$#vGu+!kVCP z>I3V>CthVKt5zRTk$!J5+xGK&?e4?;sw&Yq4|6fY(TjO+a)l$))P-MwAp9<=aheY; z86;`V4J1tx!4ltXHwWIO5P940mAF@nacVrd{h?d2gcR;?FAZDg2{Pftq;%2*-4mzv zx?mXbAUsj^FkEB7gX^75gO%i5?r#>mJ&jYv=F!D_vO==_MCtx?Kb8(Oq&c5&Fg5`~ z?2C~z=Qn8)ta4$3kS8IaNW$Bh{Kc0`#39XB6Mkg?$~;XJ zG)gEa_hR@MTpsc|e}O@4RKV`aXODsKc1*v{3E<33PO^~~qlk>HGV20%t$RaMo>X+1 zb?Huc5!5_RdtvXbEvd+ky7W7Lo?AUCx-y=$+_)BGlQCqtTU3dFNXJT-kUvDs`4zhB zJ`=v7?PuiX+l^;RqxR${+`W7$>YzzusNazl@rHe*lJxzv7!t#uG0fx+XG)lv7{Jx)dB`-BVcXaSWjL3ww3$j5m z0%Wa-zsh$T-uJ@2`a6E-{jF>uK}H!NSnzP~S71M(Fsvw;D$O(Yr_>C>wT?ffubx?auh?XZ+9v*g{Mt@%+ScvipEBDIFJlH zFy90#NF~esebYc97mp*7Iq8UtbJ;LI?|EVQt$1T>k3%6yK7&{VR$y^%tQ-bIV`Ee+ z*t#GkYqjb5pY{r>N4t(7(bf?Lb@XNz7l-WA3@e%H>x%R!LqImpy)0-@ zErI~Uk~gC1XBES!wGb)2y5p!&Xf~Y3;N*%96El z%s$>Sq3-aix}EvF=oEA!*jsJiH81@ML>)$-PydB~EJE*AiEidQzV2Z@_EAIWCPxKt z9p~qo(P5iD<6=wS7*r%i*@(M!ERogWpRY$?@dBEf!(?q&w}O|aOdklX8LfNLv_$z` zs(t#i^wPHhrglGTfY@-QLq=lSXCp)4?j<_>m@8^lSuVEcKohCxsvypvDYsRn%}hv}4y=0|7B| z+83T8C6YHMvRH_3@+FjE{*}jeaKvEoalAzq%~RiL_BVX9w@YGkZm+Yxf00*dILXDO zz}L|Ya(x!aB_IMb66#72XvVqSF9qp3L}aQyHr6}%$|BY4(R>8?iVH7KqnpPyGsHZ~xa?b_=r?1lLqjr;zel>rAFcL zSIf+i10AZ{IwcJlhJ-1KysQgoBuJo56@>QB47sjnKc~|xG zUOCg!FHUwr@oT-*`j3G*EB?zambpnnuFb68n3VX@e0<}ftP{B*Z0mj78lTpjm+DYIn%*lJ#*DgN zuo%Z-C?3s>hE{)E$$2OBCi;k9bl0vDVPS04ygXCw!$$R{x+kZ{SUGj113}pWrEO3j zm*+u%tJ9Ic(~ zJ-F8pzg#uzA5{6-Z@t?Renser??v_ED^+p)yjJkefhqsQ=;;JwZ>4ep-?k;M$%ovS z40rs#FkBTAo@NG;lTX6_JAVHmudx1SIsdWm`QxYxqxjX_{`e)3d-jSgD7$0Kk^oYx zkmY|aHjJ*kFb$KGm}NI85)gxp?MW{)J9I%-@=P8YTn`ycx?~6W+%+0kmQv-|k)Wu8 z0#-!e0!1QtES-~tzjbw)nSNtUGgH?ct08)dp=fRW!0`%eQgd!f&PQo`zw&4=tkfK|9|bWlCxoQu=XKM1xFMIS?2ATjk{|a8G#k zU`Aesa)s3Pj>`(a^4IWYhtRdNV|P$xDLkQCxJ*qO2hJJFzKv}`9yj|*n?5`lAtpaG z99TVx)wyy=S<52)vAaMZ@yfl}l1PgO$_<+jqXL>b=(J$>ER>TN8$4!g-rxrcei_jL ztxUVTT**p50qnMHod5zp7n=~o1;M(JkrjfM1x`aTD4_6Wy9WsBIWfv^8!#wV^h<#DP|w18 zE!pEOb8$C}?LH?5D9pt*^2^=>&AQnX6p?l=$z2yfmoaStv-3svD+#IbwC+@^XJCw7x-@Mrc6bCUwH8p>5b3XVHqrTGMKWhHpKK=KP zpcN|L9xNoevz3lXw?fQI&e(s%!+l&zZ4}78dfKHqdiWlk*#r6d;!gfK_M{`9VJUrR zgP5`7RBd=s^yk7~lbuav{1;3_N-uj&p-!g(lhGT#W9VK_Zp^eHp|4xQOVVif^j2lb z{cm$g@Ht=I(IC}LpadOe-D#JgnFAgn<8dhG#CJ|O${76F;6nc*q_s#r!~A($Xbpc| zNh(wIcZA{)e>Nf z?)fZG&gcDfz6|iUGdcgt*k`8i>=-+Kh=$fye?90)IDVQxo{S6m zq^4?Q=2?y1FZ2?Qt)wiBJFWIv_p0-%r+Hm~v1=y*wQyXQEV<4r$U3E6W-Y2_VGq^L zA9Srs{kG!HGgvi%x(&Yoa>7YVV;`;a5X#OsZ_J^#Uz%DwoYyc(@1{UOeRs1BWv+=5B``lb>>yvfPS=Ec zOvK=A*mtX`Im*^Cf2Qk=Hu{b`tM3~g0WLV0{G_NqQC0jujj0?|HHs4t$df99m{RHH zVAWvj3cj_b#`u1q295ql{qCg z^sy`E(F6{SH9+*!HVI9I=^!DgJua+e;118mzZa}>Em3On9>L~CN|m63+EXq3__*9} zb76T8qMz!5^`yGtrb_KxU;V@S3_Gu)T5gTJG9OuVF0`^IC8ssN71ZOvF2lGmX2|_! zorQmhoq%20AOE!UpFxN;eLPu%c-TR_al`mRC`fT(co{rAOs*n;6|Eqma!5x_m#k>} z_v)s6&r83s_duRBW<*^)AM)_NO26IL8tDG1x3+I~%#V@erboR)ZU>jBE%ONr){rZw z7`Mt?0x3T{itDy7RW>H)D2D8>PkzA=WB2ml>i;qg_dlFp6+3=b^HBEdODVmI5?8jb zq@pJS#lveu4Ud(Y8uRCsrwul&#MS+z;ZkHr^njeKjNr?km{67uIrFX~Gdapj^X1u_ z%}p6AWrx$m`+)zeB-HVeXx4^9#YgWYj!ilp z5^^M-=`1NQWu4}+dh*VjB@BcoFq+I6<^a7keJ|1H;sTYojbO2jq1sOPkE-X!8%;Sq zvZb{ayUZvn=*o=>etS_Up4u-7$v#aBRG3$9vhyUA16iD)u>mS)}$K<^1PQrZ!T+lAPV!A$O0g4@g?3 z!<`G|v00*D7)J*lQO3i615>lrj;xEvMjzf5dt60C7tS1vEL`e7O52O;_%kB@N&55s zVllGLjoq2fZal0Bcy+x>(!L1#WIGMqT}RjE)j@_sxJ}V$9TW=n^;b@e6|CWZ@b=zu zO=Vl(I5Uo2R8Xn{QbGr%N)tv(Afbl<(gz5EgkBX8b(G!-3B3p;K!AXh08&H+qzi-q zQkC98IwI)r$lN>QeP-@F_j7;m`|?i?`<%1S-s|kM_g-s#*LPjwa1|O*WxH%j%5;WJ zv<*MqDa+O-J?E_d$*+%a$(9|d8b6GP2XNv=KfI}86tiiL@F}@)UBN|LF-md;)qiEu zLOk`#hfN0lew^Ge#Mx!R*#o7YjB?Ft1n(gT!fm_($9rdYiYM7G8@SXyc9c4a&?1U4 z<~FMKdEXJ%2}p{k1nv<9=tfYJJbMj+6E9njsNcTt(Z4$`EDixeE@5-ZxdB{m6>2K? zYs;F3M(5#ccx`$XDa%5$=d4LB=)2=~zhRzgzhR!j>RDJaT&j-3EiJzJN*ciuT?0o! zA(}@)nGSD!Ykmv1Jc?ALW$wOdb_A9Z{dSRc4$JJ)kuQdDbdjX3!q}0+Lh`8g;7EIn zlbk2&{%H7D8U#i2GwO8O8U;SU}Ho(GEb z&);>iXVjSfgD??}&st1vqpBA3M?aaU@Vw4v30cxlj1BE<5Y>!>&oBrkU~d`t%7x;Z z>3Z%UBmhYX#Jmm|XPIM^^@rHADE&PlnsWq${U`x9FC1u-cKn$@ID?j8>N* z*N-p%So`yb)E}<7L8N4;i672^dE;WE$~M*b+}e_?MOIla6=S^aRfnb#ovOO{@l(# zmB6a8tKFCEJfHS`iIcQY4r{LRxs$RhckTjSi$0mt-bUnkitO#-sNLAlwU=+~yc|zujlC0XY@k`^GpG{9x`J9LK-fg8{|nDB8?bUGoZaN5=e6v{(z8E zG2E027X03<`8mCJFS|xyEolp;`&azHX`RWP+I@nUurrww*z2 z$Fa#ufr#k6xlyokN}6Ft!^Yk5?M~aDi~9tLzFzHEld#CoahB5Tnt>x-8MsOkudfK- zOUiwXJ?6>$J0TXh6VZ{TZ%q47YG)@8G!+O}#nwragAs+~w zQBZ}wEj3gdAC$+1uZY7J;H~}qC0dWQ)ACp9aB{DN0RlH3-#mSQ8bS?-p%=efY#Y9? zJ2MjMLE1@eI#VHC%rfAIgyOZ{5j7sfiwtRJK{{hK-Rp3P^x~amFpUo1!1&cg?J=Zw zdg!1wiXj5CT!jD=sqoJy&txp))z&Wr_sbWY8$0lJG|Cb05Xv!x;1fqiK8fHI`m0u= z(~50C@s8P^i6S#9`0RuZdV4`{wS+T5x-!mXQ@LcuKvEO&G^`9Hz^j%V7rRe(Khg0b zTUM{k^$e(y0XOfyZ+M&NoMmZifYztm{wUUN!I>!adGXLCW%FF8M|ub(dOA32Aa?FG z5Xn$Mk3mD&!6*<QzuYwOWhh2fE#vO(x84%4{zLM?WlW^C&-PcEXuF1bPv*iBgDV9RsvZOeDHgg> zELFvqXEU-A4{D*0+-9ZEGvn50RCR=K01hWdR+r#)QjlJ86`O~1i&c)idj9C5_ms3v z*XWQ(k~S!bA0EuYYvgz{14NfBq(4KecA!y6S=H2JabZ1#bjrhu(NFrIDpc+VrNHz! z&=UjXJ}V@Iacy<`*30b78VymgvC4+f#$6vRTc%ru;X4@Z+4~NzPQ_ z8(I|TS^`Faf{f=D-sp@Gi?8>*dR;lG%V#&)Wn$;XC}m!%Q!Te|&?3b6CIK3p>S0!V zBRNf*>TF8UL82k}vRfLUbA>lAiSZ&uTsB=RZH?$WjIyW>5ha?boHb^rFj`FR@t2UV zVS!+8dY>7xLMS!IE7llX<8SO7E~i_%W4>~PH)uINawpx&BE+px7|EBZd6{X(9OT`W ztl>3^#n&YAIEAax4N~HjM_MUw@hKjYwmJLq7A~XrY*}r~(Gt=Ub84z#WL4@;sEaZM zOr;X!`*_SRZbZN^YK0Qq5-*}O1~B0ki6TVpK4W{04*;*hQG6y4kJ-i8IwiEavLNp` zGH9opy~<&>tNqX|4~wVfrdWmdiXt_mn76o;H>5J4zPw;8o%Bu))KTf9-}`id>&i_L zuQ#%Kix>oM$WOZjZYpPC(4d+%t1L#al2R34m&KPQBRGD-bbk3vTx4=(k zruzy34@?eO{>G`n-Bs5Aucw2KA3G_EsqSFRd*>}96L_!nb-$?itfmVa=Rk@J^p3lR zD@u1gUBYeH`Rsu=i0{=kpV;%4UYeP1XPHZD9mqAFnUYD=$fAxk3nc2p^Qnv55Xb1u z*HlA>X(%jXM8n#tavfB;g&a2Ttg-suy#nkM|HZv>?R)1Ejfj3$jVKd}A5AM;QLC4r zUf$n)4qr2BY3^~I^CAD9+oTe zzxqnPsI{Ic&SSu6;lagB=d$S`@h7^6?MibJn+QFte2*Zl^e1l#1u);LRqEIw+5M*5 zW$w^Il#}UOc+=w863RB18``+|#Ol-{V{<%P=SF3Eq4xka*K0; zP(m3~jY}s!BD0$Z5qgfNioiB(Kj#@F`}1N42+ndB?ULuUW7&xD|Y*dE;#)`f2-4mxY^Y(CID7gr`|>sQ<8O4@O)^mtMU4 z`YZYdHXQ-II>8ommFMm+f;1D2iEz@{uEJjz{bmgodbv{F#&3&PpGGxn!|3A;9L@3g zgdET{jQR1tknLrb+5!g0hOm$KNhNZJelc- z1JlV$43pv&Sc|VqUGU&+#q9uLsYl2WpUS-&AArnH&ss0RmlGO~m*S%qPY`?}5 zN+iOdt17qt;VVP1w(#J{0nX*x&uiWY_t2KY!~#)iMr*r zWOV<^(u0jQJ?v`E^#<2veYoT5UgAUeje-7}YbFqqWj}qK1v;K#@N%dZt3jJBpQiG- zw{CLE0Mw{mE;OsQDWh#@l1MtPxUo5B_iiqyJj^yxAOPq2f~10v~3eK#0>(6FGkt7Kq0czCbTPz zQQ#_pqF(1k1wji%MN*`Oeov#wqCU}7?=epTT0)|=IypH4goE2MsqAz~t`(L_RaNei zGYUm!I?nn~w{uON<+kKEA-&Y4%Jxq?D5}qGa-`Kg8C7#)ka04+XtI1sX6J#81ghMG zNd#p}JW-qB$`EkY4}Y-{09ke?UL_$E`v-ad*{* z>?c&)n`7>{;7mP8DtX|fX}E?aNVJeqB3yd#mDLm9`!e1i#T6nw$84{zjhKEcvVL)G zD-hkCWp?+t3rAy3x?Ivoxp|CMj!u}4_gR!!n4`9iX$*4ss{6&V+XOIy$v*U!^v8Td zv5bXAU$+UtWaUcd{bhS;^i3R>Q%$)J=G&H-AUR9R!V?*#gS1b9TtixenmsJ`jO{J% zMqMKsE_R(buAKK%XonAW*-K<#LlHzk>jPL^;YiK@DFd5+w3|ak8!jdFGqW4BhgOJW zN`+)#3bqIXwvPkcj$j*!KTJoh&DYy-WBb7mdk45u2gSGpU-y-q6nO7yzlY4_s#}2e zR=tO*|8e=6mr+tYkrZEiXH86@_*a_IH<{n8 z1~+=6IG<|;;==^E0RPa1v85!+k!^xi4F9iI238gKN>>S^z}Xz`Z(qJQGibI4%9F(d zm5@RJV3qCUD8=8*5v(e1Aw9#8QxK23;xV$N9Uz>RR8?kOJBwsuL6KEP9q`wde)JfW7iykjqGU zT+n<%R>w>rn#e=(0m#g#+9?~|5CNz-u=DIZ$i~dSo28I_gbt08tOgUV>o?lroSX&| zU&}?Ntp!JHek01TuKDL1_<^~0<&!5ZL%2K&{^k;4;2z^f??ax*J^Ba<$>fid2w(eT z$V>E;n^)-;D5Qw2|Q<4&W_qIY4U@h}Yvpe~riu^=sBOKhioOrz@-mnwK8?sv%2 zwYJ6!@L1SQ^PQ@RX*a*nYIn)Tz{89W&`2e?L3|*PsxPZnOc)EtG44fnHLRQA#tCsb z!+W_Y0T{?<#qFVKZyCRy7MO2MMoaaquwiNGr3(w*op!bE(wXc8_$mY0m^i#;2s6W* zjrHB`#H8nJFy>)G(hkIN@c{KTzDzmXMPRsiw-_;8ixX6-g_%@v zKlj~v5)owQTrixmo8Kl3dcWfHd2ZN!mO&3?d=u>G6LY5W6;+IPqPnQtRu~SfulZU9OL#WQ~qKkdlVYM}# zL1yti71_})Z%|Mk55Z>%VzNOplM&KWIR^l5fzCgSTt%h~XBZkzWi^5XptoEVUWh8p z;h;)NqS?a07{8R6SH`AQa!bSlp6L2u=AdDk(lTWmWKueK{mDSQW(4@r=qcj5o(ai* z@2&7x8W!|sa^@wLN5=+GeNp`Rw@pMKSL^W8t2>Wqn6UpGlK@c-){RG zuE#%?#D6~F)C++JTE%Nri7(Nm7Ov*a5#d(xLbcoHzS86r!OaH6lrXmGP)mf7qEX6* z^kNd0XMg~BPLl!0wW<=-(wBuf%bi}N)@KSww6kGTyagS@xl(ic7RVb??b2#R4?(1YjvbLcyJ=Q4iw60#sQ}U8;VD=~LJ`K&Qz7K48FNGAyEqqF^X$n> zlCYe25yd5HqCAf?_0V2>3kD&W9GmXZ!ssU#k} zFjk$4@Qs~G^_i^`tg^N~EcKGI7t=2pH#CvruXy+NIj7#@xys8+qd~^Y#MgWqZG)`A}wOsbQV5t#IGTpm$&#J$|Legye5_RQ7{-{F?bx>>daf&%IU3!KqWhs)g ziHNjIcp^P8lyYiQ;)OUY!8vneLj-GX)SX}ej|4r&E# zpknDm{FV7qEe`=;l>%VK&*Zo^QXbeZb9Y7VZOMdPROlvZ=WzzFW1o1KCf{P&sB`{X zd4@UK%ASg{cnZz|rPO(cIx>*6+4}sI==(Yy%F*HgkZFm@wdd+IPTTnVY$iQK-gFlP zFH&>)>NYoZZOCoPX)^A8ER}UC&V3+!7i1!{V%e%GCGBj?#i`NrJ!TcN`rKR;kIUXT zRRTEhbC`ECevI!byt%Bca*|%tflPm3BeY2w&K#=~HcR=;LM5i)5*$o~xWyG>+AVCg z{UQBGoo=M_|bwQFv0y04{aQ(8mz=|xurU39a_B=RiWK6iJzLNd9 z`ksi1C3cLXGGcWKQ1tXZ-28a0CXb;utovy97X9R-X<5BMjgm9dSmP^n`&RD!VTIM* z3)|<9J^Z0xhBT^k|48g!9FO~P3);P(9bT-m|55qFe77rm=ZVcOlb*9jjtTd^^HCuI zj2~I+!yT)a%Fy$@I!~3cTh3`UsYCG^`XFpd>|^M-DFh& z5!1UCy`v*38KX^-9yTKSW0Q$FMeT~<%5=@BLQ&RO-;?_xKQugAOfy;q*&gle=9P{Y z3Qe5`2>_$mRy!P|Sk%V6RI+0M_v>%s8#+K}Jt z<5N8PdgLyACxAbyR-($#9-M?vR=G>oGno$?iuOM&?9<$*G=Jf|;~8}Ms)PrAg*6YJ zRFi#2Gf&D{2qIO{mxmk%c2cZGo87IvIdSdxLTyS7NV>LFuhnlqCJ0{U*6}y|@-q$X zF^BJl!>NEVhv%HlgAUq^pA$T|8Z{lQ-~1Gb`3J!UG^{2nd_C=6>>L+Mn3@&0 z>$;TdK5iNc)Z58*=1$+NZ~`2%i0n21XAX`BoDAwOTlYh9U)Y%48(P&g1vU_&xie{@ z(+CI_BibM)qegvTnX|#7Ous`AAcLH%pR~OhNEdQCXoswGvtLQ{P~_t@?(H8u39Lyz zin{d=Puf3j?t9ZHC-1M1D@oc9V-K+OK*6yf*kx5<4nvN5Jxpj=0NgrCVsy#AKR(`ePW_7pO<8o-l7jWEvTQrUAM7En)w9J*#y{R7y$>s zY27}K+{KyXg6+O2hBsyjtGkVsUFJDh?m&~+L8Tb6+fQFrJ;m~?5?Eu*y03?iTYXz- zx9AXRaTkYZF4z;Gm#-0FM&VS9tN_#?9-BTwLnF@k=sOwz<>7yV=Kv@KbVn1$CH2Iz zpNGElL=PDsm;OMW$Eur>#^@KWzj^ReO-?%u)+GlsSMJllO_zc}I2Zmp3l^pMSIF?c z*tE{VX&8)OD`LOoek|Ne)c+j%efWfOKh1H&??Wgq1O^BkYEKxii{|WJB>()cHoL#Y zDgXUL!RvwzF`^SQovB}GPJhg0D@#!&_s_8N!-vNv2n2ySNK+4x5aBT_47H9yaf|XO ze`3Z#wFRI5Fl^zHOrLf7Q2q`jS(9`fL>~dW3jrj@+XDg@WyHWkHeSrizp7#zRCayL+e<-7}0P}&|{U=u! z4K_K&@g$i~9LBe*P~+6MljBBRg0;E>I;&)U=Qf7~3n))DST(zIL$Qom>Z)QRz@5fW z4u2-n=H-VbH|8~HTD#{6aSM+)^bxL}>-d%FnD|EI(Y(+!lEp;I7Bh6TRp?bNoxf;T zH&(Bl+c-LEW7^IwOs-jU@6T#%jj$&UCLkV&x(Ii#M|R)o!Uf({saSe@D-mUwP+i)u zTfmjca;T5NNYmy)_)<>GveNR!(e2fp`+5K8{sM!Q0-vir#2|xw%ko}bAKrJ#d;+An zdU;r9BnEdQNFq6IuRG_h;B0X_sT68^*7bCmu%DDD0MDC*aT$gYRGw@f4J%YFdYp9SNovxM}YFP-)5};|wg?1K>1!lDlNY%VF;3pX_#L;1J59>G9yT zg;b*1xDnFj76BqY%ugjjmu{&iXCTFp6VVJ*9G&-7DKg>e2}5??;TSgm69Cdj1MIL4 zLz0u@+`6Dj0@3%A{dx?=aVKgcZ}P|+KgX$)7Rw4WL$_gQULZ|`H?O(`V(3~jG)i^f zFrP9vb=4*@Y8%RXmfG(KJmcMyw4x{m0jrbFDTPbX1;-&{6!|6TiwJ`9y$y+P3@Iq{ zv*HRN{<|*Kml`ZoXD0l#U4r$(N4j9dnfuICs6t3tO*>cp~0hiw>(_mOHc|->K$$740Ks z(!!%@&T>vNkHuiMZHN^X54~K)M8yjHRMv5uh3!srt-9%q8?h2MUb5O^sLxWyoUbYG zK}-9by2}=I-5rJTk&h|D*Nmi`l@w^M))nB3%OB8G=rYOZNy=kTGkp9ja~|n&BvF;( z^QP@cLG8z52`qd^aThZ+xSn89xi0a%`?|R%qE>XAIzI{{ zq9Qpj9ClxU$h~p#Ab%)9#Bw6+uA$TL)%NKhA!$OhpSe zerBJ@*AocCI5pbjJeQf!cIR)?sfkBr-xdG|#$9i|g%qx%nO{I|ITDOkgeCL)0b*m^ z`9+oNx3M1j&;S%=gS)Qb^|FzML)&=)BrZOj>aW7}FNncEwc+~$!}M$J1}8r%_Xh<` z4*w!{!W*5birP)NUFZm!e%hb^l_s`Nb!llTqbq0{j8OGS`tX(JW7L*K6MI}dzm#b) zr)p(U?E|867VMXD2H5*Vy&}Wx+i_A`i_QWnBCW2-D5!p_&iZZnq%wkxVYIVq%KM@s z#N-;2ZH@Ko5M67rUsRD2{0BxuPO+FTHloKy4G&pY%=MqSJqQV@uGi1&cdRT7Xmu$$ zJ|j|-#v2+gG^@#FdUc5PiW@FDHHu`4g>uXPY{8sgsD`GkTzyG%=6};#kGlS!0UExS z(m#&5_6VQEYa8^s_vKoI*34VrHi^^itA{E|^GjDB)_bx&T*#k;5oe?QMWB5lan(m# zS=8<~D

VK*=6zY?*4WPPbmWd42HlSO$l&%>;4Ai3HJTIb>C>yfwl#i#}`EC=>)i2Twi2v z8h;*XT9FQa9@bqfA9QFy%N9PDK23A1{q6i;Nain!{zl3`=9j;?;rAjTomj6F61tI~ zl4Ahs{s&!uvO%HdIdkk_h8G1R!6V)0B185DodOuRxWyWpQ?|#@6S!WlJMbi&+2a^! zyk6c*EOe;D8_(JnPeC9Yc~UesKM?3L>|T0Cz3(%vJiyilV;v;&$&3A>@2d)0!1?`= z?>%B`iWbg>(O?g-(yu2g6NY#)dd8)SFi*sf6+d#Xcg;N^(YFGXY7~8$H*!uoJkPtJ z>UgcGn$(<)mb59f%voGOM8i(rt2LoLb{zQx#CkdF9Z4oJMI|2w$KkQy_|C|fPP+k` z&%bx{Zx8;ned1qKapL%cD8|VwxbvrCxzKHy7!_Y_7J5p^MF)L+XJqJx{!B89{WOE_ z9~zdZ-W5r)q)0(@A`f;`pyUsoXleUi)8&KigD5*jy*!|W6-G; z>H&b=v4HlMQXF{th9u*T92ICFtBTbh?Q#fZ>u2SJ2|XLPDb#@1I4+MeurxL5s(J|Stdlj;v~u{wRWx}> zrl-MjS8t1D^c0Qn1Sv+`R?&=ST_zWvvA2gXH$FU3c z6DJmG$QBsUKoATnHvC|4X{9&SIl*+?zO6F{46-W}CG&r!!94CPH{Sdf%jS2IOG)q6 zWT_r@seRJHMkTT=8lnuh`dlb6H_fg$CDT{@Hs#BvoDTT&tciUhjNYp4Hqw{xQ|G3H zghKt($@OHNdqZQ3F*aYU{C6zUS8Wnl^i3;wGtW^Mjn?B+Aq=cOzI7o+d59ZnWz;8% zLR=g8O=FX|M#+1vUk?`U| z%R<%QWinA%`XDJ+1h?0R0Qv zIpW{qGFc;{KcIuy%2fEc5IK@+zr={3xjMVl53q?QnO}y!jiv2{2_i6-vMb8~G(S%c z11TbS0vX!QiXzS4PivR7F#_lOkJ04rP$U;(kuG~U>y;KY7g;78K|FmO7@i-j-r6T8 zCsX!H-NqPqZdfBYD=NMG`ZmX|tSN|R=dqsw@#Y`m9J&Xb%{-r3o`nUf(g zF7x!zC;2{EJp@i@c_QF7i}GCX{E+(D!U+;}BXR&sEQ$jWQeE~w(M!62@W@cC)Hk&& zNEt{hH1^;2bFv`4wVqcFS^i2h(lA;_PI44$-VnMQRC-tNE6oeXs)`iJ!qMS8D{1n8 zA?uFyjL3FvJ!PouolS|`h@pRQj<=<7DXt!# zuOWjSPL1SD$WoT7cIZ#>Su)k@K=fXB$mudIp_MKp(J2`RiuZJw%>3+J(Xixf ze!a1^ziA?YYXgV}1g#w=Fkc8}M{A0-S~peq5kXlfg*J$IXJo?%#TKU&^n3^njo@Do z|KCex!UJ^i=}=ytVa73+nFnBce$v3K6z7~(QI$TiR7@A2N}Zx)_jCo~JET?+)hH2@ z^J5G~V!VoHx9&CzWuAR{y9y0YMb^8k`3Zx|1Tma7iwM#g2%&jNS!){;x)ld-tI&8f z>ky(#okGhw&6!EkDg8EOHJ1$tqvz_yDw+e`hs-Oo+A-(wmm8y>uO~DH=D`->*cRC5 zn!v3#N$CCXKolK8o`LY9Gl)~Ca^h(e#N=rhZ?-~f7bpSY>&Ei(_rCn;;78dodcy6- z+H5BQ(Uv&&2-yKk6f&r?;#oNq;PnE7RS|Ygkw5!~T-luI-b~0@c{9s#IxpIL*>RvG z+o#m+QC{uury8r>8x(;Y$g*g~QTxTU&;AdQqRF-I_eC#W-gTRLaR2!i4{AX5-Sao+ zIx^=()JH;$AC8QCW|O$vuw~9oPtyF=oS>C$$UO2oU)@7Efh-_543JyrNhY%(!9egE zN0Q;K#;8vlbDui&cP2(<-e-zFOe1r+uvvViadlnx>H^}gdSBfWL2tcE?<@E)Kj$|& z)1^qQ%(?ATZ+4H8Om;+`-QY1`HgCeGn`T<5Io#!wN|xEzhVp7e>=y_~7Y(14GGQ1% z44M(OqB|j%E?H&h)-?*cYfDy14`TxbcCpyx>3Hu(VX~0Ii-|V}pF9s{@;|#B_0u-j zW%o-&;PG4EHS3SI_P^iR`O=~_wsy#L>Wu$)LX!VObcGEoD~Hj4mKN2bhe4}i$&jCk z_ANCZyKud%A<+9c%3Xw?;(O%{U|)0-Z#6`W&N;2QF6X=%vbyRuHCS? zKA#?5Pe31Td}DyH_$0=(rPFLlAwT$Z{vau0pU_BeLdI*cYi*$o**=0hxWe7mZ?@dK(@j>r@(lja z{?F6$|K$VKq;#5#eEp`?J}WN!G0@83yRbf5DXN#;t92`eP<{qblQj{!Rn^>d>Bv~S z)Ih%$3tHKs%Ew`)c=os!})z#8^``|=^ z5mAT^ND|B%g*L>2f7K4*dYr2x7e;3p@D3oBbzNo`&}EGR@CUd(DT{#bWJI58b^2Ll zYqemNU22GQ*ZC@Q7|tNSkEo>{e*-W{4p#?a5>x7IJNF91)+a%a&J=zpH_q>576Cf~ zBZsv%dmf|COw4u;*iUaflznHz;j53R*nNElVR!3gxF^5xKn0zf`|%l==C6q+d4*wg z9P;~_v}Tun{YqnR>8M@2ZNKinf8~=jUSQzxu%L-+_F2*A_5j5^&t9{9B`qZ4-kYfi z9uQI63Bjqs%~O={KHM*KV*_Dp9CA;|#@Z2sH;qH1apDU1blMAyc;WcVMzqK#FIj<> z%fYEP%_+lL7;rgY_!ARCYKpU}EXyQbF`>ucGZ2Ue`nyFTiCgW-&y(Md4N|@Q_{Vn! zjK0vmQ~4M+5<2hA&9@k2ZXr}#{yEm0-Yb2)HobDpr&gIyV(5!f)ZVCi)0Ze2dHtm6 zT?cO?cE14d(}|{Pv{)EAqe50Ho;MTXvL-WZZvJpx&=MeA$7M=QRGu+6UNNn;sB+tA ze%ipnuoOSq5GVV(T03d*>|43lEUGYWP!?GXGc9l;h$Th$v1)pn&l^@L(U@?0QE3;( zRg+y+wV`u;N$MVP+F`e5!>TZ<8yV0K6@4RWDRAOX5U7IrEZ><$9mypNl9}_n&#A-l zBhD!+`TaVKby-!blS?0MUrFGrZd;UyxS2j}sqASlG>8n(WWC)Q9=S8L6upp=eD}cC zWIBkanwDJtYqg!Dl#ZMotJwyMOc-_6NjFRsHzR{oJvdu^chd}F3h#P5$H@?Q;`6vf z1UfQ{<0RANwYieg9Uot&KK)dqkoUEn1;EGuQ?8htu)mz?6qSEdWV>wJ=joDDA4ue^ zANOgqSuIJlGcUFR!o%+8wP@t?<%Mmd$pt>P*rk=eg-+K?c%&9NL)FBaWPgBV!UB-4 zZ)!?Jvr#kTb`|fAi1XMkONCo*DmhN@S%%vr)tV)_oF18QzZIIQ+{N(`7WtJXSyfq2 zih4!@F6Y6Sg`5hP@ZMG0WHX-Pk7za7W{M4N@;{-KWkCBR4_tWI<~XzPY{p$83R;%t zbA#sC{g;0^oBm$-Un_?{SzT`6HNLyZ@|8yY&STg6O>5u%U$9$ja~Gs87r!~JHbp>{ zaC_Eo0?ChvC1wK~K0CSAl^nGiPxKSDvrz@VZ@J_^j5@?#POtWqxR-jF$ z{Z9J|37dl##LhhGoZkuhk)+Sh0M!=RpLoqDAl#a?<&*?#YpQmXMm%kU+^)oloQ8#2 z@{`+BjGx2tV?|!PyFi9Tn`nCu(?CKWUaz2v%~IQ=`~;iaueuP0BsZQ*I&~8YpLi(a zl^6mhbRdsj!&QPJS-A`4ZwcP&Sgmy!HDq2hcwYq1r3eHjRqUOP-JsvBy?Zf7T!oh) zIo?a*l8~fhIpK92?`3)Ps9-C|sjZQetz6^flxb7DXIsqAkYx$%a`-UtFtZ(&vroTz zV+`CG#bfFzMPyTUx3Mvl`;5kqz{5$b1U0N)0lcPAyGl@~!1;WRQJ14L%D{yaV#4*lRg8n6vyPiv~FYPMq z6=gx-4v1;D6p{pnK-{h~*Mrv1G;USSS?$^X(h{fgAlFlrS?5Kw>PS8$GSQahZKQ+U zn?3MRo-4&Mc`u!-_5<{wBt=!WZxk4Or-aU3xPhfhJ zR3)Xl6lzkjaAPW}^08SiHZ#I|Zn@`y%eD*N{@>%Ts@R;69MA~ooUmA+ibh@5V9PC$b1YY%jE0}EYAQS~geR|=Fj ztkPMHt1JV7APTJ=Z$;dX)rzR2R^`pJZV<}KZ}O@*m5VEudzWo~f@7AJL}f8`n*h~G z0=4uV^U?@^3GSc-u8SDWDT}h7{z?-JCVCazq7AXtN9iprH*hXQS1|-uBy#v7r;772 zs$en+YKpxNgfAo~Cs8OLyJ_l%{4&zx!b%j*Gqmk9aWTi}$CWrN@WIF|bcGUA zarPN1_o%_+06XJ^$XPBR*+*w&$UtAxrd>ozed=JGe$yJhu z=*jVGumvqOcM1$cpvMktbVgF*Tlek}W;yF~C=NR-q9W(t6I5~yvZS@%gwO_-HPR-+ z^o9YZ$i$=w&V`CuMHmr=AnWAa@Zi zwP8@L19ntw;)b5Nhwgzcz*>J$>6|pMWykWoPW#(=ySSnbqQEzeUm^Z0&85!H&Rvy3 zC7lOp(VCA-LI({$M$a_V?~9oEI%UP&OH9`NqI8Ys+z z`+WODHrouqk3nd4e@nQo^dYF-qbu!x=;1oT$kF%~*O#(lhYI2=Z zJfloM41~H%Qyfj`v6DEWCe3|Du~q^gEbzcN9yNGrO4D-G*m zy_$=!Sl58+^%VpM5qXEP;HpztL^f9?@T`iKh!^$FJu6dB2bSPu_$qS`ND2I?vn}^^ ziZ4`CTcLv}#ge7eD$yEgo56~caWqXF-`E>Sn;|YCVK*K(s21P!8A_};B~_tQESJaE zJ18hO37@&i=|rqh941#ziLjav#!Z(=CKO+kYf-&fu+#30s@fLKZJ$4rEog`w)|#6X z9>+`)ltGFKQwsId81lSn4B`nSkE?@{6Urp@WAN*2%bQJA1aNU(q9{~>Ul%jgw?q0u zTOBz&Y>dGmOi}>aLWl_HLq;5vU3O|hNq{~v{HfkN^=3|+wzi1e7V(X9RmA{BZ`f!e zjszx&fKEIRW0?uoYi#($w{2Cr%gu$e`%1&KGyP_J{vfq0d_~TpL&s3AEL;KmCRt5D zbxb)ecSlD?(<^q%F01~$I_wk`YNT@!H^I*O^7XK{cG5bjlE-i+*I;F2=d$dcPkhbCC80-VFFWGuORX^u8f05)O#+5UY+KYz4 za+DL0c9~WYUVe%`_aA1~6WCLRO7?iA6=_u}M0uAeOTzu9D~vF~=8qs0?=!VzHdQ1U z;k@D#U@2`{XTF6^_Hnwo&enMw`}ulGeka8&eA>pSb0na}2VbsOK5UneIX#ydTX4^X5~Myv4769R z@lK?v4ieo3QPG4YX~+(2_KA^Np@@lnoPDBRht6ffXY09P+q438^C-i$keedt$wG5! z!=>}@eWtU?%0-t`V7aVAC|6=$lDKafY8so@0Sjx&aOmR8;if=dhMkizAdA!%gXG$( z*&l72VrE2iAtqoO1#ju5s4^IZ4av=KB0xw+ltbbQ#mkNyq)c_f?3EElT$;9G!t#is za6547n8@x!QooSgj@hj%(=CYQjn2(pa$jMvpJ!`%Yx(MPQxf{M=Tctl^<}EUZn3R} zM_a#(p}}0>r-w*eVgCFft~R*KuG|65$xOEIfp7oor1kGE`02;BimQ*O$itUC%`_!E z5YinI-R(C_+09B3<7iW?wS(me97`nW(If(DkqE~OAR!>;%{8i0Hltp_y3A>Q$cnD*ILTk}&(E}g=a zWwS)3CszD+vbIVHhMdh=8XE7W-@a-)+gK zqvj8$iVtso17@k7xP!~%;PyV+lvCE!=V*a*_}+^{vrGQ7l7kE8O&KL3tGfP2ZfwU<&t&tsvAYiAezrC_-wS4qjF3U zl4YHdxN@Z#P7_90{rtg1Jx7{}urSYRuLrP9AcNC`OPTX2j^2eYCW`w700!<6+^tzn zhlSUEEYi*u6jl|)Xb1zL;#!xl0JvCo+Ti2j``l4y^JT3JqBh)~N(rid;xv{pd~t$> zUV@@J?3LgfH^FvJbPyGM3-+EK3IHyz-D#b>2jBQ-Ulit^j=}h4}&+8bu@-r zCCjqSJJU)J00+)di5)d!2%_WS-yuH%<>5Vg83_$I1|IP6SIt2`BvfEAf8uD!~`63o8(pIoxy_2aOG~4vM(>_&$eAcBMwxrBv?|hzZ5s%E_2t3C%K3pxP&td;a6L{@24l zs{s{XXe81C&K0XnR*V;y^x3BOuBn7$^MF8mBtTb?d~s%VW3OSJC%PwSCB*7c{7XC2 zAvsY=%-yvHNtDgA%j;3HRfV+&SAQM1VmkaOZunyTv%87S0LqoeC-AzH+`);jH2bMc z6EP<{sf*@H{*%xD?Ulz4wIe>ZJeieCyfD63tbXmgYS?DhAND}lvd)g2Z=m@~V_g+xP+YTn(Yf1+-f*Y<9c<755x!>#iL0t6lt+lPiL`i{)IQx7y^Lv%q?##E zRiTiK!MYdM_fK%-=l~b1Qy!TH2qch&39QEtw(LeujPQGIjVD-YtD>GHHJ^0~EZu6e)t4%h|khS~9 z+_B%CWS6+1&WuofAm?PUHG!2a>{y$;>hJ3LKj{|#;bJp@S+c8f zM?%B%(2{R1FQd*{Q=Hp6IT4z-G4UzzYzc1e)=rf_92`wmOx@bilXd-22tW#K8ydXZ zse_mw4;u8}Q$>m!70$mf+5Rgt|J2;kG*p{|EcQ_RYp2Nlwfl|!v#in*aeq_8jo#9< znBqSi469dNyAze(n#NRd$iKfa{wX*raysBUC(0k5l~2IQ-A+j-d+w#bbTYKO{buQ~ z&asu{u=w3OF~#VmM#&&_KC!3{8()D9i~BUlP79pZwmS=rt%y13Ke%%5rf&n69?g%; znBU&V|3DlRe8*q+639RBAN(DoC+V(|1?{c}Bwn~5JsmL1oqBU|-R9v=0x% z=!;cuN}GC^qVa)6DO@_^vKV>VZo_PoSO5&uS~dG_uNK@_!2$raiDpoHGxG|GS2ZRG=`sHE2R3} z9&))(3MnZsFu(Q5^L_Olq)_~r3wQ8%Zxf%2NVYyejT^yJX$m66tdMzo-Q&`FiC2*SK%9=FJea)rPc zJowt6v%tW)GZybHt9}`D#Mr=R^zys<`L6KiwfUY#1%YGj_l#1hD2LAY{La?U&J?JR zvt!lX3QKv7Zbif0Z@AE_9rm`Ce6J(eN8_B}t2NC&2T|`aHr#S7u=wfF>P^pCDS`k2K_A4azUK0tBy6_?F3v$H1-l2+9%*# zPrIF8&4(NUQWEf_*O;O#4j4L9ebkNCzU8zmdmGu8qAN(+2}NNK#PDruUUR49m1>I3nA^?H7avI&&e-1&I*{J0 z5%qK0u|(fz?PS)Ht)oA!;X(Jtt{lRMLpwKmHY>E5;5(O{*OJeb2K)5ev`S`3V@*9FE4aMhP2W3;;fb=cc9T@=yg3QFL%}6ojaeF=~Kay zv+NNZ_)3QR8+rb}d@P+W-Pg1>?iRdTN%Q~XWOl)V$R|1c{ik;_#|!S^O~zb7Rb4f$!2}AZ z$I&Gn%cn84WW2?w1aT7lG^4ZgYQnbHb}Qj^Nd|n)=^L@l|BaK6Ov=25FH6q#Ye!lQ z@wfm6DiAQ%9QETf;iqN}Q^0-xvnn;XTTU}mw=_ooKjywWuBmKYn{litCG5(vErh|Wmw5Fikm0tt{n=)K4&(m^D2LXqAh8akLzIGqi`^5^{Cn2^UOuJr`@!hE+5Hd zg?Ue7vbEw=3z|V;G()jT0;4kgZt4Uf{w-b2U+ky`RUQu?oZc4l=g&Uul`Noo zDo?ptJ!uC&;<`qE%y_Y2On+6#2-70zc|OrEM)8NJRJSV=9p>erctSr;+t7uuuLaG$ zV6y(fs@?~-ES7u4N5floutd+06s?6K-n^Z$RaHcEXG?IZynEsDyjvmJAc|Y>1q*U6 zXUy~SVJKs|XidG3&swg9VL|Ze>V6f4FvW9* z?fLA^dJJtzlI~D9BPc{*ts5sP8kH&`hp!YVy1o;{o15kpx~+Pi6`YMBN&uSgfCfb3 z%Sc@!%?=k(?EBzwm{mkZM-B(iGxs*HunAo9bTk^Pl+STb`p#8Ij_16xyHi#-B5u4g ztMwEZBT~la2Dt?j9B+P?9MH&DtX3!Wor1-gny!jv<`i3Uf`o(+i5U_kPi;Wpvrrg8 zdpr*^{bdO{=b&J=?qJ3wWXR0okiJ8HzdI1H6Zx<V^%-NhLYV%JJpi60mP^qOXA&6PsL2)3SG@m= zOa2R>w*gmQreTDND}}~2XBK8<6a1CCGkhKv*8NJiQL*X8@O77+_GKm+eQ;Do?9m*0 z*?#7}jf{}h)T(6DM~d=<+V6W{pgw>ZiQc8Cc)k?jn{$oV}z=6Nl|UXman4*SV;)yt{s;XBJneqUca zlZd(&h$F#4U~~7c68ksj?Q3%2s>K599SbzYG(23cvgycaKr238q&3i-OO8o6LWIXG z`C?Rv-hQ{kCf>X-?PtYZ>U{E^=mpiFi%Byzk{V?NV#<&$`*(ETCky@acK!O#52kZ| zu0wte{isktLR<1wuJuRU63!a|JG;UpL;E)q=pgfcWsHk()GX7ypd#&UhR61 zoC1`elN&3wn<(@xN#@>au8yt~K_BCXAYP};-3J1tr9l_WkYKZ9=IdF@hZEaWPs1cU zCs}qQ%u4NOzMPm}fkuLd39BT1KWZBD8<5e3sR@6(O{`SM7g8i>y{Bx;JkZv|4nNMC}Mx3aX1{tROt?kEWdFgWx&+UL}Z^okO>Q z#Z+@Ny?@qiD5?_>4L(EpK)tOMT~x}B+YTbpy9H+sqd>cyeL2_2SYN_hHsBiya6Mz6 z8>BVFn+~UY38FXCj)Xp-#Zf*a*1kz6cE(=E!-7^Ul8;;5n3+OBp$Yk60|_7jDuD!% zmX_gVjs1zOyJIkdvgK*%6E$|gu7W^E<4Lr#ZWcuplB#=DGj#xoL?q~Z zJS>};t*wxEgy`FdG6ek9b8kvl2y2&R(?(jAy&!)a>Ke^YI^nxK*>whiW{Guf7_W$e z^G^=O8y zsJq8XSjR*CkO%wSe%b`Lh%XQ1Q#c4^`(VYF-O^%Qy=r)h5t2KIig0v#Rx?^P!*Eu9 zR*x5@5-oN!`6wQN0Ly>;lH~~KqttRqql;F3bmHIV$Yd8B|5%qjzG&cAc+g3UXY>rU ziW&FOuOKV73HIcgL3kwuu$U)Z{NlYwD1Esj`7)9h#TGVl)$?!u%(cX91oz`C7M^)G*q~I&>NcKyO;2PHG``*%3zEUoqRY8M zqEFgpzXB&BtBn!J966=(Uh%N1`op#u;yhix@q9w|G3sdLnI4)Gjwq#A}7(S8RdB8;MrJ zFc-t=p{E$~1mi+zm{WWLStlWE$oo3oy{N;%Mf3DYEYx6c6hoYTfA5P zBfz&XV)0(O;-UU8ACyi=0bmEc>EB*r$KohK9j&&t!TF=($URBep|x)*VxIv zwaow6|CW`-1Yl*!`G%DRP6Ki?cW)7%jAu%;2LLlS{buCc6Wscp90fS5qskomNk(Qd>ap@-MY1c# z$uv|yW@u>m3=JkCDIR<4dnSw2*0$4+WJLzWu#XiNMX^ZT?B_+=IhuA#BWi9V$ooh= zD0M6@??Ni9(tz9xg(vcS%95Q(t}1SFLbt+DmYIV}(A$F?DR7}lN`e=?otUmjKHOU1 z7kFayd=Eor2Z2NnqVxy^@(t%n_g-L9(i$-q$FsB>{Ll$GC-4({?LWL0(u z4s>$MXN(bv!Z<=a2b*R}k?YiVMXS(>0=Q$6l6Z&A+IzMYZp`8k_+!Efvz+h*q!{Ut z(!r66E2I|gBA~|N4eTxQ65;Oxbc_o45=EbV&}#{!x?oQJa~ARXe&D6)EN7>&yXFu9 z^YkrUhQ$^nK=mRVApC}^o2OM<1br^jW*oCNqi$@IIdBgZH!GaDU08v#59UH5R#TaV z%xgHp8pLH^D1%p#AF9l&$>p>RccT7XL+gl+X0ooGwlip;WSTf|5!nF|KxCUY+@Yqg zt~}Pfh)4=0H_z4xm)&WvvdqczDIO4U`=Q%y8SS+l%TYF^U!mm?UX9ZN_h=9icT=Hd zBaIDBWd-29vJaYC4cF1L4VVuV;X5xz)jM9|B{M)Z8;-I9*s-2QF&O0q?@35%x*YNm z%#FEAfeGXIIsHIykR++?CLk)1n;EfY#`IW#2&Q$b1SwH>DD>2d_*CPB6=|7lmjO%Id>=!JjMoP3r2G??#{3UKZvJK!iPPkI)y_f`b{3I&rcTS|gIPTKR+KyR>GvbaY(L0Y)QG3q1=JoH zM@An>*%1(7O4kXsG{$*4QAe{-;*s6jSL#?~IDmQDDqzev5tGl*n;9M8u|#(FG41n9 z*B;m4J_}K0z!^^CnVDy(jsqD2zBsRmsnF^~`gOE7JABW-zkDs ztV&cghr2b(Tm;FaApq_Z7#fC){&b;{lsLA>Lwk#2rudUJB;F^P?Vd=02c`7cX?RIg z=;UX@(}IHK%YBar*W}DVAl#N{it*mZ>Jj4W-Kot<&}iuXOtGI8lu+lZc9nIE!XBr$ zE|;hn(<&s>UN&4dnAcGt-YeefH7{iJ*b3m)8`U_#25B=$bVlPf$^9!XtzI3&tjMC# z5~7GeckQMYMu-HOW#)zmNB+#g1GazREZLz%G4K2jm$f{~z%RWj1+D9-wQ`&8;?4d) z8G8TWlu62GKiAyCV%b#RU+H?UD8>R_x0Q5c^{N8+qLDD}(|P>0eLjiYQ!nHl1?OdF z8tm}`4Ti$+@Lr7GU!5f0m-4pQ{jQ?Gg#*?;j@qRc_l$FliHPNvO6Qiw_=DdZm|EX> zOBjE5J_T*+KKpFCW!e5TI_#56Apgg{_su{S;{Mc4bmF&b+rTd#C_MY?bOUpNxN`A( zMR#)SZg9PB|Ge|S6>v7P4N|-(S^Z;vcIq>L<(tbLrLOA8z(UZWB0BQ#|hgs7>b?;ff>qDH1iBwv`(wN%I3)5xd(Q^neg2*IJ1?$$JJimD``~T#*-`eAECFka1dN*}k zbN@Hz_77is!?$*x7Wk=jrZ(pHI%>1{IsrPY(`r#wxP(NKL^8u^h;URCHG3vO z{2hlEyOLFlM`shGROdXU9ceUeU3n+LbMdZ;gU}IOd5VjJ!%&Xf*#;sMB?@^mu8oP$ zgzBMGBE5tN!~*SXy`pjB%0aNRz|}#)J6(^)9%nG@zk1GN%W$<^1-I&#W}Iw5QOuFy zyV||RR5VdaV$p##2)w+Dcm47S zN$seS1b>@2vW!Td?Js#Z*rM|1XbgIh67xz)*Bf(+FUm8cAz32*c%j0ODAbIh6RvS@ ze|X-EUq>pQzOq5(TllKASOqUMrKjjF;yM?dZW376Z@_wODO?fP<`rDUMpTMe>ElnR_919Po#PX6x@6qg8(Oq zb-rsJ9kxei!GqmOze{bbl%%D1l`!mba)Z*GtIE8N2C5Dr5h$2&4O`$=$OS*u4c0b| zcYu3zg??AGx^m*i3uj#e+NgbT>+txeU8K2pEyC}<((p!iX5v$InuRQ>%xs;(LUyr_YosGIY9Z54f;`KG z{IVeLV%eJJ&#-p4kKLo^B|wehj8dX>l08-*eL-L~`?IsYH_tZ`ct4|JGazN8qnJUU z`@Scp{g0OX*GDJ!ei+<{V2HOUFx?v+zrb?OnOiW$nMi=9*b@`4R*DFO?527Z-(WqO zGmBGIr8~P|b94OcvQ6JUp>flFq71d4MBikk+Z1-z;4bH!l;Hyw^E6g|)C+}5RTBfj zL;?~eL@4YdiU2My-D5_94-+lD-~SkTF!k%O*`nGEg@^xQonET&C~A>nNi+}`F}X&=`B!#niJsWj`ldFVxN`}UAj!@vL>k3l$UusR2>UwP#%JV!hcb5;E= z(aiqS@A+o{o#mVfi75_#1F#QI)_%?Em9J0bg!0I>FmT(Pey#AuTlY;y-x%-jFfXk^ zth;3sRU9KYm7ZSs1?UglNIkOp+~P0?^)o4|_xEz$dG3`JaO1cd!-&sKQY$Vf*$w0- z!+-fxsitvgw3~*}m%BHv(D7LM{9^hu0e z=j8xI0shOs?~E)Ta^6>Yf1}}PY4tI(XhH1ktx!OBD|tx%^^reI=dY*`{`7`F$qK8{ zx4!Lca`koDccE_|A`kAk{hbc@UsL`sJ=w@q;k`ziOXA6g6XqxXLoe~a-two%Mkbe9 zAD_K`Yx8m;<{6+eHv6uBda>hRemA=2Pw_hc1(|bE`!kFGW3D&-$AxTCP)2AYq8YKf z+{Ijcgs?3fd9AXfg&dDVY9fuYM%x-e2>_Veh{Xl=rBr<6b}d?+u{V4;!%cJRvH@|?T;PZk#cSf>BSbq$u1w0N~f{<&l( zjwEWwL=uX}+60^_;9upZPH#zqhEzKf`kwXdiQ!(F#k+6FZEwj{w!2@e)UeW(5QB7i zy^UqPJK!%jN4(7c7AyNcs`c`H?czRBO-%hpwo>T1F?bLY2p|sUd#!G5yIZxu&m)*K zoUzp&b1x#cBF^VU-2pSfoAz{tdrPuOoy1+WSr_)1mV&8`@Qlb*SZ~kMBF(&;(^8(X zdJqm*#!PuJzDu^rHjJV9OsGG^bjy!76AZVSoYjEseRqMciG+U5#Eq&2E@xY+=)I|G zUSRuFk@;&HYEMDKOG*4y6Z^1BG!KUAE6c1ZF0?pcYjZLD#P4ZjO);9j)uqCq{j0eBcgw!}Ux@5T0cs>}7mQD21dMqq*p;7H*4@ev8w)`)Qf~ z?dfzE+X5vL2?O5XDIjO;J)!#^@$xsD`d2%K@vs+s-u*)YIL|6!`N|B{7NYHD8qw%n zw?iO774WVf@Hd&5;$3fCeKzd#p{w}ws@^{A{XIacZY)%ZRblyj$o2hhPg&t-)W{x- z3wC^>1K2bZfB(^hqRri8n}eZ@jjZ4Ia<~rH_>*s*rGH(^_M3z3s}t&9^|zs1EEGS1 zE6{tORX<~Nh3JmLZA0FAo+FS2OT8pELM!NpcMDJm5ZM4wp7KeL{027r#wAA;@W&lB zzWwZlXtzUOnzndZx2WB@F=llL6b=JW2!{r$HvI2!{{8*3IlJzWY#z)P!B+M!c9t{U z?w>0m>9ncREQ+Ae_Glc9pI;C*q>0m9WSZDa8M94DvlyE<`6!xbL(^lI^338qj}|L0 z^~sT-$9mZny?eS~_7mL?r}Y0Y$ect98i~p7IWRd?l2|GY(QT~zBdDa>GGVM0RegmG z+oQ`Llm1rHPT^~_xkN_3GEQw6oDZ4j|viw!cosY zYM4Fizo7EEsj_hmh?}dlx$(iX)E;`paBkbUZKS>-;(1~DEM?uQxS~ zJksoBhrLT^;UORJu@XZSlG*^VZfk9~+k%;6W5|rFSy~pMd=`-QNw_E0Fh@2&s=X#2 ziZbm%47?e#7P}_*!HYh9yrowW;NDlWG+`SENp~v0=HOPD2f2%nNA*g-{I0eQ&70}I zCazl=GSXt=bjN%(nN@zg*fwbqT~yOwL`4kFw#wVD^-;8BVm^{>dpPpr-de~AWLj!b z;SM#icDhEeE1P~t1G5$#FYhfKK#rI@8Tt-%-qglZ)kY6<{QX0l_O2(&!u>PX{S^+t(Zd*6uCA?oT z2bai!gvxt4@^!`dA{gB*^f8qD1Wc5CD__Ohk7Wht3DjkUXlAE&@Cu?W8fM>SD*smK z6bc&Ep9l9sTFS8n+?kwcz(hV}CVOXeXj7Wi$oB3c{an~F%Uk9Qy7+kEFY0gxT0>NP zclyuYjLF4==#&r!5NQREc+`?)jZIocO*=UFt+@dA2_V_qUVOQb-8fuUbEZ)sd#@zRJF@#>vjBhtJd>GW6p4! zywk`Jo$uR+ zz5#cfPaA&wCjI5BmL?tDk+c^=-uKdEGLC2Lgy$GFdT?ANY;>JRcx3(fLH!vIAg$7L z;+FE+6SWsV^BEU_8pZ*~+y#%F=*FcYJ8olq$k0xg52K(WTb8KRSiXmV)3lTX8m-uD z@GzWL1R}DtKzPi|22iN$E1cD!>1}~U2PA3mL6V7!$Y#fr!OSS_^DD5aYxRxBVFNGHkZxk$Ao*(D za%j>l6ILr^CL{1_RS!HIoVbpEgB~py-(@eZ`4rA9B$z)w5q^z*6w0hiMkWz5U=oUW zPIo?Ix!k8U2Tc<8o0^zgo9`u-0{Bb(yqHO@a=yIhmKNEGm}!XZ<|k;hRCJZ4Q@ZbE z;ixn~HsUG*5oin+7<$QTp+Kyw&m17-W}TwU_8cj(5>)Kx6rdOQF1%6O6vcFlWimT+-{Q+D$+^#J&fs&Oa+S zo@YB+d5`tp)*ZvuT@g8mgp5gsp@2jy4JATQ&!R>S}i%R`!gsx>!qIY-J)no>)vZ$dPr&yS)q@aM; zXDo{!n6BS#Qle3DPwdL;Ci>Kwi$V;eP*n@HNK`e02)_+0mIbUagN>qUT23!&ki`{; zcGAtNoIVokjs>b#j_9NiW$wsxVG}<_C-g9#M+>F4zZ2fc1YAYR7Az7RCLxzGrr+gS)Rv+uoV4uP<;UcT3s&^V&mvsz54nO>&5w(VOh!7D$f41)9; z%yd%udsIm!FBETy;0s_+a(D$^R$Iw2mtjN{7(fM(3rxH4Ww}Qc5xSC<*Y`0@1;Qyc zh%jkOU;k`sIK%+rG_E=Z@28xGhycn!d;bC%G~3-;uV3~S4okk;jRscMo{ zNhQ-HVPB%eDeWE+-NNwJ9<3fhvjstmVJWTY2ka)|=$6Xso~d>z+0zHo{k}}Na+xF! zKU@XVlSunySSECv5KIAv0aAZL?526Q0ruA3+854Nf#(YH$-&*ldl=!S%G4i#bsU*z zn)1m;uZs5q#G5Ko<;sF&T+1KW2=@dgt+owMSz98+D2F8mSxU2*12sGyLA4j zjNbyNW~q^dW^aw&|CMeUCq`LKS&>G_lytoR6nAgbyc~EeuwD|T`|5-0NfGE6ny zF7nuN2=xZXJg*=M1Db~qVznA_vssCFDjV@ULq?%p3sdf8dZPg2MreXM1m%}ajBiup z@;MnUDLpI6R5Q(lR_Vrc$usvL>t15(lujH%$RUPG5Srm0iKHyQ{owuxR+JeP4Uc?~ z*J`_MyJ&5z4HCyrb1+Hkk<`uOppf{-t0@P!ayv3^`6Zath>8^T=Dx^#wd_YY&t!pe zQOdTc6%nTW5RE{|afLx)_;_xatV!ek4awg$8?L>(ZQE6`V|-g*vSZ){+|~s7GFDd= zpVvPx@k0LNf-@5#Y}70j4uvssV0FU52KTj^R;mKPZQ(k0{Q0%cAr56dJ0x((1vc&6 z+@1U~>)dl@nfCDop|B)_prrm9G-MR#e7IVL6?abb34TDqRP541y*iA#f;&~};jyd> zt1Kh)pw<|JNJVEH?p07tqfQk#$yH?sTe#YMiFc$XSij0y`GE%ur&Y;(i(o|%(LpcS z)2ZVq1ugR$3t+J*5I#x8NPakTjE}9!sXV*o!=*UmDnLl}Vk*{RAfk`{uB>j-3$^ z+1ckm>{sHl5mhU4SAsaT-Vw^A%=DNbPy5(Ab#sb-!_7uLNT_*GyOFqLH0E{Q>SliO znHrR?ynmd9TZfiCqTF>SD|7%%0Er1EKN*a(om7UrP|?JrE73+vuKX6Ij>696Cwy?z z4IjhQ!aG_7#^6l9l!2JT)eED_SmA*vp$cMd(wOA>YRcY|#)>A&(O>B*2$v47o4(q# zYz@fW>p!>70-O>6PU+i;j73cy*fg```Ndldcf9(M?rML|AI<*1JtC9YFBreG@tCN> zDcc<*zNvcB1JBVTwq-S5@N;HyP@iGMDDE4iT^Z@F3dvAMs+)!SHfS7(yP1?@Y-)qa zfN*xhB;RM;)q>Hg;_+vbE=&)#FB+}UOg3x*ZYlDnLRb?qMsyETuz$rT?S&1sjn!X% z@MDVk6FPEz&&=iJI}wd(=!0(Vg{=!ar7hMyl@AtxxDwH#?R$|QOlOLbV@)0m44Wa< z#6}l#sqymZunV$UZ>KRLURs97XW9or1YD(huJjg4280VWm=!4RQ6y|i(YMNUoxOW0 z)$nYtv|MRw$>5Fm5078e{4C;oBk-rS2eH+%ox9|;1qSCNJ)(j0j+S47o8otKP4Z92 zQ}PYsrnDnfXIIVgWNvb>72+VgXgfA0)%@GKjwj#Dr|j6d)Aws@N0^VPV||eZ2QB=e}<{9f2&j=hnF>8i_D=RncvgvZ`ZbBmLFy1DwjnyvnAEKVO=*|;b60zKa& zefZE9Qcd*YM%4l;D!SR*S{r2dmimI@Y%H|jxzhdOZt*|%|@ z6XFsJdxk6dU%(!#?c6JK*7j@iobWluxj(~qiF_$$!n2~$B4J}@ zHe-oHn@7lx+~s;h9VgjJkV0STcp2e0VUm^I;o~E3a&_FnL)uVJ0#{+es$P$TaJsBe zE1!(?5Uk|o7IHJtMXR|w)#wJ3ko9Hj+)vk-v+78a@}r39b{eO(vuGR!d#XaX!wkjy zs@`rpWTatcGIe*w{PV$@#PZpi*`S>Vt?am#qM0Q*K@(@H6E=-8O*D0l#Aq-($c60} zNvepdESsy@U>m4H_s<9qV5Ji*?!oSxy%iqz>=q$gtwCem+x({TdkN`bXAK3wUbUwM zJHi~agPMa@%s{E~lOnaLq5A8QFS+ctBPU_$RRqf|xIgSA>N(d|eM3s9gCL&C=mUcx z`HFps6WBh2D=t&pZ=6QbRZ+Jm38JA?U;IiZi2vvo%mR^3-(dpl&Zjk4h z6n~}LPxYaua$gZFlhZVoNtk#8K`bk_>TbOI!?+ryFB7g$$murH#9fbeK2bwin*jqN@h+RyFya znvQau_VPOIi3?G?UA|QV1Zgv-Of-!79@ArsU-B=_Eq7v1V0vpK-ych08ldFF;U%TwJ8Y5Au6b^L`SG`JqZ;lI`$0PT2xu*D+ zc;Zn9RhTKhI6WQ=LLDk$&G*0{j)-caLwxUQkYK%}`nQ44*I9q*gJK?dRmUy95VhhcTH znpozf755OaU+Ijtrf9|WHBY6p#yoFM^y@!y;e73(~uL* zEKeoo^1Y%t!i|&J#$K6;!9)b10{D|+6MP$_>WrS79!zgObcyO3<)2&j`f!G1Ff=v; z_P2;cdpPu3tb!qW>KH(Y&^uN$cYRyEEzdm8KDS(ll;%C3-yg77Bqx!jp-UYwt z<`&n%V(z<4Rv&{$I8S*y`*$2sQZt`A%{bs4?8aMF0Ijwu6s0%NbAF;Vgp2O+Yp|2< zP)~qx#0$4~tJ{0lWM!8#W~aH=GWWfcF{?>BT3C%7x-)MO)rrk^D_FdW$owK33 zM`Z zb3%-w&5+LJxqQqy-47JIF5g%JlM8D3q2$lPnsA@-BQbZGpR% zb$=`YATX`7XG;oBOTlN0#Li!#4%GYOkAJ7SvCb)bFzhf#$?b9qb{Nd!teDRXS{zX2 zZj98<$w@O`A1dC3+CH?tJ<$24=b7&N`#INdC2dLVzC&(5Q_gN%>JPS&K~26;IbSFK zAo{A&wtU8#2KoCiB|GvPJs^rucG{$S%%}A1(T&JUIeM)3d-D95zDxxF@BjIIrqt!5 zZ*0!spA5fQJn2+mOxQJ@QI-z>G5pv*yCFo>0P2RL=G-Kmgb{khUnRUf2S_Tn5UEoB z%FdHEsSS^_I$3h}l*J>@tnvAezO)yyICsEsUD(vk4H|i?N_5*ogO$~^e5EA$5;`@} zo3%p6{$2}FaK#)R)nP_Fp5EZp`JM67lupomK)Pa*i~dse_ny1j)p-`}k9Fetq$dWd z%@vqZEio?yI+P5d_LbNgPQBsDW`AaxFA*cSa`@G74KwN3LQhN50EN>iJ#8{#&Ozs; zQ%16L+DqzqJg*CAMLX@8@+n>Bjtb^U~;LYOHvWn3}9$%&RpcZIPi2`0>=mUK~|q}YP~Y^ zD%?>)@q9AJs2&-nIg*x!V!RfgC+CHc-Ocp9+pH4gasEnpw>m{TKdCd~zF%SK7no~XSQHh&lX>S!=4CQ4CMIUByS9UgJwNM_p6kr%xf3nP zL%Ii32m;G!Ovk8K!8I-`rE&@LFT_k$X(KLnrOjaZve8eWW?d*)$I`T5rLGv?V#BO% znaFBEt!8eiwt@64*`JxcW0cDt^xVGIh1igKa|?pU=SAh6b=}yj-V{-2*$=x?#PPC^ zUq98En_J&PO}wu9RDaK+uAcZftUx@nfRB$?1~3aY%@cZ(%l>7H;ps_fUt$VtN+!qb zjB+3EE(`ri$1AK>UCP(2uve3X7CLYSAXV)hY_lRX1_1Lj_=&ao4pBF7=g$Z5pReJuR3-3$yb%nV~ zLIsSFt;@;Ft>x--<&%h*;ilnk#2I!RCyd*!)q(-zt#JaO^w?A?arU3_{4#Tbl|cHnAM=`tH@rnyaSP}fxwKC z)PB6@@Qlf+Xuxs*StEDsYc6YqPeQ%YxU5umKCyWM61yh|4()R_2t2sF9WAdl=;hsn@eeyXQX z4{oOFYC+;Tn6bWITfF99(Lp@HSTf912@QwP8ZX8uvGNFFG}br%Ic7)T5RfA$>vjL~ zPyZB-bcL$FRV3{2?*=2;euV{n1wn;i0gR8Drpfbn3|}yF&95n&OQ60HY-7sTLVgQI zl5+F3_duD{vwo)NTwc{?+58Hk^4PhXJ-qBu6uN3iPn5e{c}q=G{(R|PNCFn!UjadP z!9BVPq?ydnM4_KkqZ0h;P5czzItiA-+;3G+C+E{Ko?Q9q53MB1m_Xl$dS$dw%)f*U zJ_hJi9iu{%ObX0)LY4-?cQ2Sx9sGA(~saUi5Waf(>zZTR7AbHIR zgt|i}*6&Is*(!uJ82H8g(~6gp`vUQTas7_p`nOaWupSu&Ci!p;nz0CC{8p2M-A0r7 zI8kthJyTyjP@eH*G(>fvxdR$i2CJB6%1p9P{+j=9@ml~n!Z$#(uMLIzf?fL?7RB7a z?~jTYDtkC&f9X@y&n#aNWrFO=7^`?U=mcKBqTtm`n>Tdgfsw?X`OS-(R5OYLpZGvF z-|P@VPHI=UK|&j1Bez#Hd*?KLM8o9^mEJB&C3Gj!xx@a@hJW*bT@K34RyD>+f%D!{ z^Ig)(U-C?NY^|QHQqK8u-!x!JqH#1TAfETT7{e?fUyU28!za5e)cqTWF8Q5y@xF5j zp*&%`|D>qh2SI;&ctFD)dEfe^Glir&3 z8N)^8c*Yt`A*f=BRXKOI5|hG`Dyu*~Ki{Z>i04t(lB=0%&?68Bf(%mBpV^Jrcf&0mCRh~t>HbKtNUQ$1^ND% z^2=68{jC7D%I`rX4VALK0$4wtN+H=ylnlRv$UT#zvaTB%IbTb}{d+O_RKI&doVGmI z*IdiA@q5L^ZF+@HKB7&ew2d;KV>ou1{&h$UcYCZ)UGwHm%trT}wb%jM^+#jjn325? zhE`z1R;{iN*+2fC(j{S4E>#PRKq4Uwixm$$luJ4e|IKyyH{1D7E*CBEn7tMDP-xF( zNP)dl=e1g2Uu$b@Uw-~N2|W5G>;O5=Z)`aj-% z23iR}#+W2sZTQfZ{xNq4+=@zcKve+Yb6D`*c&1Es@PpgO9RHvPm6gw)5E^}_Xa8Zw z_D!QonX`pCktP}%#OC;-JU8Oe{tj4HGaO)O)aijr^L4*F4X}azn}2^qOK5xxrT9zE z#J`md_Cfk2r34NGbmhrcIfVAaolyAL$=>{Es5I|;BN8|VzF@TAIC3*)F4>a}{VFIm zmhWw*x?NT{hg`VCgaW^Wc0s#S5~=q&AD^MRKikPJfhTq4eEh~{1u|0A!OwRhtJdO+ zt)GT;VC;XM(mvB@N8>KOv>qBDF1;J+8D9Ws727c6gnTv9^p2ke5lcS^d3BudA&K9BfNn=hMIwK=wZ zw7Yx=AodEl_A6b$37yQ%r+M2SkA2{rP742g&i~Qd2WwSK!FyG$n^kEW1b5pGs_`^9 zLZkTdC;t4nvi2T`pj~Q>$_?VRuW3gf(v@)A+-f}7;g+7xS=F{A^Q*Fz?nKKo#$-Ua@l_c=H1dmZqnar> z-poB>E~lc~n_=%|A(e^9%%2)A=1R44&blsrtFS}qUOWx}@(DK4oag&mWq+k}AF%w1 zjxYMl&@ni)k!CkG@`=A@*^{zp+!l~R+U+u{DYtw}t##!4A?(F|<18@O1tbL4tXIbo9 zapv(dq>+&q_cG1(!Rlz)EHJ_>Kk|BEcf+i^=tujBBEf!Nee^@EM-ty>Cnf{_=H z|46#aE`G;rm5%+HjF=dD33IRUwTtnEBT*VNP{AY zrjmxMa_l9%z}K;QGPYl-oVmb4;hr8^vAa>D8{u5(V zagunuqT~zvzc4Gi-x|PS=WrnSU#r;`$DsZBSiL5xOncSsER9&hr+i(HRtMSi!*y7z zH4qa(z$fmJSF(kiko3@4|4;&7{2>bvFzWrk?vuEd7f~u1A}RXCG;ehPIFxIK&VLT@ z^7(>9AYuSpJs+Q>pdDDG?o%{|+W*+fJ9an7&7DETnnzY5ZOmzURjIn^Hr;o*PJipI z_=k%%wzS{c`#i{cyyF!C(1qAN`$|bHYPFh@*`W`x%}a}264UkN%GksM3?{8$lkWj^YB#F-%h>ZpTvvHYhG7#QE&l}qV4Kac05DksY#CBS z!xveCm4#UB4DxRLxdY(c5_UK3+vKnGo!JI7NTi63b1UUy%e_U(!cArRz=hEo)BHv` z7C;p8GUxys{EGN79&*b-2|;&)&GRQfJJZxIAx-S%n8`;;vd!2twtJ+l@ScnbC$WPJ z!^`Wo3~Q;y5fR((kVSH1fjjTP^}l5&_weQBnaalnL?4#hPCDAFAJOg30nro&HDeTh8=MTSq&1 zMnAx`U8`PVHOG=R6eSRK{>Od{+V(#$(?eh$m<`V31BpO^ASm&45Bcn+hFL(7;7e&Q zrv2EKZc3!i50a%Y!xCxodCsrU5jb~NBmC1Bd8A>8-_H5y+N()eETstim{o&h9q@ij z>Gf0BmuvY=8bs4wK$uaGUjz41I+>GOnxLCF3$Joi?ZMvbf}Dq{$_&`!?Wk(?>+5e3 z-N}Ze+`jkmE7&NF#VmIXia znGH`GU&%~k9rPt90=(q;ZRcSpT0Qzh-?A>`Q)548U39X&aR6XAKG+EUhDmtD4OoAt zXvEUu?Ue{RPhk2po(+tjlf}Gm1V09FPV*x#zX?Eeee#0!0QB9L~ zytYQ*^*E!Qq?+Y2M5AG|fqS5{A<6ro*9@7Tm<}ZnT8C1$qL1f|*t?HH$K{@M$)TV- zDZ?7e3;|X(X(4YMsseW+1s7@o5d2D4O7kKBh^H$vs^QqXl2<=($c?hQd2k2t046pWvpKr1>CNFQyq!M;otw zpHCJ8?PP=M$qnRnY0s95l1JP5<{q}xIo2Z?JS0=Yt`ON%%0kh87J(d7JdSfAJ%A?E zqYf=T7)d?85>JhZ)D=q2r}fH-9&0fq7@Fx|jaQGK0ll+jSAa^U=-hL z5@#O`(27>^UeBLOrnz~gls*~ehY`xUyuj@6o^100YKFbKo`hf(+gOhf?*7RxKIAD= z){E*zc1EhQB-Ya*kLtahOn4+-%@$&ljT4!W$(EQZ3;3>^ngmWG9zU#RzR@C^xE8Sy z-~Tkyl-TKa?C~wqAfueL*L6I&E^C0>3!lXP${Wryii!lC#&t#$kqLwGiqF6q+l_o~ zl9*Zan6B-*Qv8kSSsby8b>*2BltW30ETPOjP-fntDL%{^5?YDX znv|Jq%hr($ohBafN|x?Af~ajXTr)&lhgVF-T^VIj#5@5%&1de{xG z5Y665sdnxQWtYv>mL2Gi#zMznPL@$nXxO$jkpwZnrDtd;R#EhszAmvifA9|5^~tLS zVa#c|6h;ynz6PcAU)GsOtmcW>77=(7T`t~qGI9PfcUD12R8mZ4T(y_{YcjQfO|}&n zqV(^;*L~2X-oiz>+(vO&pGD~iG14rg?= z1j{}<6!9EiE=RO5#-ZTGU5-#JN~IrLDbT~O8F^3R%_Cja{mMT5cn?Utrf)_xBtJ6D zl~o|kPjsj&Y-}m@J*PF4W`;-*h{s2CMjO&B=)nX|$odVGl~56RCyAD*!1N7>)fQAQMG*zx)mS{A!+G7YJa7k0}tm@JpJ94(qwL14QZwDn&*JvuyuwssE0dz zDORzS1;^vn9$`Bm=A)V}Ps8X6N8b4*XlQ7ug*n{GqM~Aky9=DmIZB`nGdsY|?ISvE z6^Me47Es@W9z7uoF$^-8iQ^5wIGS~ybU$GC@=HQhnqwf*^20^Ps?y2VeK)HdqiOkt zJ%xNKn)8KY*K){$xcn__DBRIk9RckSfX>*XZ5#J?Kic;fb+A;CUg`pk#rKQMHo;>uiOSU3NK7HH`3{hoH z!~x!Y-b@y>$bu~G9ZIq^AE9)CYwGQCEXyDSC>qfy`!GO$^Wxn2m~%piGXdNQr!w|7>&qFA zQHI(#VWBOgyc8)_mHxwveDmu31;8hdZwpi#6$UWmF<)qw>N2uJzYZ>5SL@EW=rvTH zg|Nj3JN&$kh>d-M2{^hvdw-(L#!V7^Ka8K1n}UHWM_=ytef|f>!>W)R@o|tE&x;0e z$rii8eB->aLwMe~fHBi?tPFLrkH44uE(zVLj>>6BEYJ4Ur9914d^8k(k)1*NI7w1;nt^`fGZjk z3h|vGJr;)A6a%a1h2W`F_JVW^A%A&er4v%k~@#++ePPFTn@Ac%pOInUTeHow}cU^|vtpUR{Eg#!N8$K1aN|xVnMnYoo6>fUhGYy4bYhFG9 zzob=omXl|iG)Wu%N%gX!C@Ba45cHl+tvuJHr9H;RhtF;)5rFv{|JEV#RP|hq{K8(e zSxrAyA1#nP9HyxSeYnOm!iBZU0eD&U*pobRhWyy6TZ+812QZLvKRBbac#~s0wH-BYf^dpIf%duE$5^?7f@%P@CK3RB&omEU=vGSVGREdGz zJif911ft4nN9p@Nu)PWYK=!i5A-`(BtokRM_n%;1_7VAe{USZ3oxWq0g8s6O5it?B zvYe<@`7(u}ItKX=TCWo{$Sf!qeqm7(O(6+uPibPcNR*e%*n;Fo=|UNFk*-qb$+st* zg@R!BQtaJy;Ie=x6yMq4Fco;Yfz1oqLwj67fk|9DE=|uD(dERo*j_m~+k;}PHL@_x zqGiK*6Emy5^SFu&;tC^*Ews~8LC4Kipl0J(0M`f7nS7o2afdJ27M-=+W{cT{DG%BE z$s#=#uk6yNbOn;L+UgsDv6^Db%F7#c;ti-Uw12Wj9K4W;h+@w>XvOShwo4}RsOykU zDmvwc`@dCJ_`)-$zx~9^;lRE8GD8jNw@HHW`(foQDk2=Uvx!Abqik6P=)Udt`q|3; zfk{ig-saWuwq$3Esse;B7#eZ&s-_%Sjo+@Zo5hT-9}lB(;m!3mMw(sW#TQ#3OA_sS>q|DJ*m)2`xVU?*EvcD za3UpIWnaNacMVmM&xPdE47PQ02Z6XW;H0#AHBXA;TWny;={q)_tjz7_47VKJJUMGt zS)cjM4JNsnCY_pHVg-hb>d7<;dw9946emOkK*Cg+)RQyuEwJ$Z!3^^|^>@BuRU1t3 zglsm1gb)MI$PCO2;_voa$refPl zD;uXrph*D%WZ)iUab3z+$|kQ)eu0lt5$JXTk!hdJw{lf$jQ02s-=t-TciIW6@yF@ zG$|`{H!k^2HtId5w5L6KvPvM)-fxHylK99f$Y>g#1tolKl;lijuldxPpfQo#DvZ8T z%B_HSyh7Ug^ORqlo+#k>8`+m$EIeD~VY?G|jVkO>!}}7^(XAO5wrjaS{AvST=5Tt& zvyH^qHeJm5vu!kg@Nu6VOXyDOQ1xLkAn6wr$frHVxjkUnW7RqU(K%rkwh4U8Tz}OX z)E;SOhxqso)g5-`V_nd^DbeaqPf5Mn8br7{)3nGNdvQUS`1@?`Y%+#Z;P!(9?eUvr z`x-%#Qi@2SOzCh0G?h?r=@TB`DclHN?5NjkLIprFTV}e%l%kF&JGEf^YQ$Qw@2`p5 zmGd;aUygi2&nlBL^gc||=ltZMOP=VJUq|!V1N^Pz@OGmEo}FVf30lA`VJXe|c>#r$akc&Rg_ElJnsAE^^pz=h0Ep$~Q4 zAwe23{e*#6djTEU|1m87C;otJoi{HM0~(qf#;yr+vLTsX$>bXL#>n%9F+xge*_3!z zVrA(2IVGI!PEE%H0Wo09PH={C^+1KHZG0lyes%2J7Zy6C*t4rb4`&UT7NwP-Mx4S` z*_@E_-~~+$h1YLwDd^8k`bZ9e&K^C&WFHk6Z^jAf{{BNOHVvGTFrhER3i`;c(?GFb z5>yr}v2`nnPcktwsh7sGQO71^w$^(`Rh1(ojw);U)YK}*S}Nx=Qq*!QH^m7T%3?Nr z7-y$XF5yYvAC3|wS17`v^6V-NZj5XL8 z%Ejg_T%&5BO51LSjYsA?lC7{JX?f%jJaF?R$Dh^?#gp2MA9=Z~b>+F}X9W%BG1lE` zpX6|r#^V`$7fsBQ)2PHiDH%EheATNE0=Le8bo=e7)!SV+wcS!u@jGhEhTc|gZOshe z<r z84Tl?K70EB^esf~qwyD76v&I;?j~-=q}eCP(H$+Ube`v>JGgGyy8%pB-{N>sl zmy){6nI;cC#YHPTOBwFj3P_M?Sci>NR)tRa6`60YOFvE=aa?h?W#J<65 zzpwf2ky%_qV0nkx!!G5L)vRB%nk`6Jss@gP!yJ zH=^#SpNy%SnEGa(IVK_w%?;4Vct8Z~b|J7X>s&RVaDdp6Jl@^F@5k{nIOWNJ-qiQ@Rt=|!qKYRh;t�@i z6owL2hlAur%E`y4J*(tepV~cy0MM2vpB;WE2AGd_K6S&!NbvS|<0;c|QT$;w-V^|A zQE>`1>AK@E8PrA00ybk!oIy zHTHeIO<8BHHezBEVpZ>0ht{qQPU6A)j6&TECeVw8#@G^9SM5>zD4X&xY~j@&jw-g7 z3Rkvrte#;5)yunjQa%;Bvlgk5Z~;L8typC# zFa)w}d^^Co&-m|C_7cLbyy z8|iKXOi%82@UMAdF(o_L>_M-Bg8=e#>5k`L_wQjWQt&6oXVXW zsZ5_S4>~)QnzfmskHyL&+d==9$o{1~0PqxCV7tfSKRC=6yOP+H@@ul&c1Q27*oWrV zT9SSlcXf`W&l%ed`U*JOh}i~Qu?P;~89~?=qzRkawpOO5f%57V>(6(r=YGNkmG^~- z^@_$Mye^+bGI5y=!t7Ln!X6%;zxDS8z#bKdIUt7(YW%Xib>WlOUsgM&1k9O;P45Xe zt_Rm7>wNy`k=e=;GCJ}ft^i)qsF$^dpl7S_QTRyWCAx+dGT=SG=rzu*{wvnAlg`1M zebqN?!d)qPy*x1NmHCmru@_6aIot!IdQ~Y0 z30+kYc1|dr_TVY@H8W1C!mhqQ@)~DQGNV~_Qe-koW+fNVM$*bQjL*J+$6HxBdNa%U zVmyfsT5*pKZ`iF$x*eEvgOSEABxgod>^>-Niw;}Tdc8YUv7I7?VSp4=!KmHb7N;El zr-8ch*;f9|N(#OaS@m)OCTOU%NGGA3ULk1Y{aYGpD5>b)s393bF|IZ%?+J?*ob^&9 z*JbN=`L9fa&ZsbPt|J{rCPq=`iz3erDau#o#jI{OZ>I*;%^{i~)_bnGI@FxGdbchR2`dam+GR8{tdCy_l{vtx2yfLUl@-!wFbb zJ3Oeor`*7E`ab3?(hMD|-Soy0@Dhs9oP2Rpuu&N()d@gN(dxd2ebiP#WQ!kWG+B#C zw{>mG_2td#A#eYU=gDDt5$? z8(K~>7mdPQQt6D2mHAG?8?6H?wafePC36@PVcSh>I0n<09i+>ul<;oBkmCyHMkWWSy zc^UVq4*O2GK2e#Ied3wifweYQ-1JAR+>?rpXZ^vk>L2Tkm4ZaHMLVZF2s{y2aB-6` z?pZs8HFae#qLbfjJ)!yBs!zILH|>)4PTS=vPz<^^o$X39jAbs|JM$FAo>fd*o=XKc zrPB%;ALD$+7K9@wuI!$hezwY2dGK?iyo$o-@lNYmx!2Pqxq+3$0K)hImv1dHt1-as zp}{#MXxXDoZp1}gxC~03(+g7)Z3c?2M&8WbjLU@%SaCK?)BHI!0*hIEdL*D1R6p z^!ezUK+l83LmF_UuIfNwW85GsVrCNblikDe%XDD}-;G>P3TRX>0C!Bbj^s2%tvc)In41I&HBS;5*$;f%K0zxOzop=oRe``_a1w zx#;14aGd%5GP8=bnVr@SQy;JN8LjDnehHDPICY6pJn8yaR=;guVYNEFM(n#wy5IX| z6(>-8?{$vjobvyv^W%TM{6BzzL!y?o2X{%=0v-3Kxm|Qn*&22x`x3^KUOaC2Yk@`3 zBUQ^@dLzipCMPvpx}SazrlMaaHC)HeN=YJHk?#5@a@WmxpA7hMFI|Kk!e>u1mZg#* zBNIQa;hX<^sJ`t-%p3XRwLc^EPCuk->_Fzf0`+X{9XrVa`NvCYhHUKZpZqkn_J5Cx z)7Eoe)C`OpRm$qO)a2}Q-(e@$Wdtrc%=$k%Q6Fq<+1-T%E!oDKDd`(55Q*Es`RLOy zYip~UZt|qXjr0XJ9=9TpT?72LA*VDj?KZd98dbhOyAj28hWvBBHjPxfABDY-__yr- zs>3zk(Yd^<5Y`#xn|OPjzdzFb`IuOF0RA7_UNoM)QMV_vP;rNTC+i6VMAX?Dr{IcbI(D`Z+Cv9euMf+RPAV*L2@WF4M7dT zh0y)xzC&OL2)nA=;^k^}?5e{HKed`k3S7IR%qYjIlp$XJlB8#GC-BY0uK&k#Qx#Bt zlefm$3nvs_NwfZ}mlRN3>>nJ9DP^uUZ${x*73EsZZxvKROsM@5#;A za8jd8v_thea5X~(G~{DbmCCd%%T3Beui9&&0UM1o{ee7e{oV0Dq-`s zW6(Q?uUW|6l7H(T6PAaX(dib619dy*!2b9@AxuApR!TO16EMB(t|fc0VkG{SZuL8{ z)6};7B``Rp`2}~if%1FjLvJbQK4=lj!4Z7w`iYnf(~2`F)GoB*;@90{F9hD1ooB@E zVM`31&M*DHQ-=yX~D@ZeXds|LISY zHgNb_K3wzsmEYxki0WFMJ&u03SUw)@)0wG>Y0{o$ip6mA&w95p=#O7Ua_8SqZX#ZV zy2{^HK~w#zZe>X(8{*qG65+{(gWmvCyP{MGvP`{EU?FqCPx6t4Efj(n?Xfp9R=L-B zzyI;O1N4jJaV}$nLg2e{oLCl6T^?BUz&y&42HhPi4Y%~i>b$EEVcMQhIj#C(Ucw({an|2D^`8xMj zKL>0!->)R7XnuQoboW4Df^SBhT6U6(wy9EmM^RaVa&y8IPS`=(AbP~7I#4TfX)*xR z6Y*9KTZA08Lw2v*^E`;di`p9(?M*9INu5VW;+x-68C54F%Sq444Qz&ZzIuySvy%#< zZN%tVf_1rpHO}BPjvaM1as-_FYTgfJ&RgSPpO~devN`tdiceJVuvZh=O_2dt`uMZ_ zXPYmH`tNs{hx7V7Yjz|2pyYSJ1qyyBs7qD$iHk3qF-& zQEr1X*FTk0e1)W$ugS?Uz+jOe&+e3w+&_0*%)lC4efY-I?7O$_6c@``seig`RiD(z zMM8p<%ZUGVE8|2VFN;H3`{2^Oo1Xp5wP&)AW_SO7^0&nyeiwgD zwr)|omh=5Vbl*do`GnfNZii>5{<1huxi0YzY_y)GzKqa&cjY}5wx4$Ksle7xe_0$q zO|}|-x~pVZwOzy;J{%#>>~J^TFuwcS-xddE)B0=CQ0D*chm6u!s5ZlmZmPnO!KK8D z^b-jQTB`nc*4HS_GR9Vw4L(<45NS(6*DGodmCt1i5h<<~pWbs^b{K3Kwot=(V^*F6 z2!x+`O&qt=J8v&O;ovaM|Aj5*b8&micP(iPb}=Iuh_8ZqJl=nK_4ukb`XaU)IpjI2 z`<2MS-JrnivV$XaAVO>OI3F!b7nQ$D`nSb#%)B-@Vezwbn*M)fz5Ew75a^A|gHM6W zAHo)}VVbtn*TVoR$w7S#T5k^3rtR6MDNll{>+@q19Du`AOK0c3?80|%7yQ$NtkXT! z+5P_)m0#J)!Y=XiYk(T2+6WH5*?;vt(|yD8T7RS&3YRm-qoUX)H?lznf)tm7q`QX>TEsp==54jLW zGdfiFgMmjrL~h!h-fo&!Lu*))MH-ePy(@IX3)GCl$@1Gtukz%j(rKqmZNl` z!d>I)lM9?ZpW@ltivPt>=Ct}(O{hiv|25qIpLFY2C3~LvX0@<&<=eyk#v0Mc@N-9jsQNap|k5+GXaGfhy=E2Zy@F`~TTp|MbVy Fe* 2/2 + custom-kruise-cloneset-c4z9f 1/1 Running 0 96s 172.18.41.50 spider-worker 2/2 + custom-kruise-cloneset-w9kfm 1/1 Running 0 96s 172.18.41.46 spider-worker 2/2 + ``` + +## 通过 `自动 IPPool` 方式创建 Pod + +1. 创建一个具有 3 个副本的 OpenKruise CloneSet，并通过注释 `ipam.spidernet.io/subnet` 指定子网 + + ```yaml + apiVersion: apps.kruise.io/v1alpha1 + kind: CloneSet + metadata: + name: custom-kruise-cloneset + spec: + replicas: 3 + selector: + matchLabels: + app: custom-kruise-cloneset + template: + metadata: + annotations: + ipam.spidernet.io/subnet: |- + {"ipv4": ["subnet-demo-v4"], "ipv6": ["subnet-demo-v6"]} + ipam.spidernet.io/ippool-ip-number: "5" + labels: + app: custom-kruise-cloneset + spec: + containers: + - name: custom-kruise-cloneset + image: busybox + imagePullPolicy: IfNotPresent + command: ["/bin/sh", "-c", "trap : TERM INT; sleep infinity & wait"] + ``` + + > 注意： + > + > 1. 必须为自动创建的 IPPool 指定固定的 IP 数量，如 `ipam.spidernet.io/ippool-ip-number: "5"`。 + 因为 Spiderpool 无法知道副本数量，所以不支持类似 `ipam.spidernet.io/ippool-ip-number: "+5"` 的注释。 + +2. 检查状态 + + 如预期，Spiderpool 将从 `subnet-demo-v4` 和 `subnet-demo-v6` 对象中创建自动创建的 IPPool。 + OpenKruise CloneSet `custom-kruise-cloneset` 的 Pod 将从创建的 IPPool 中分配 IP 地址。 + + ```text + $ kubectl get sp | grep kruise + NAME VERSION SUBNET ALLOCATED-IP-COUNT TOTAL-IP-COUNT DEFAULT DISABLE APP-NAMESPACE + auto4-custom-kruise-cloneset-eth0-028d6 4 172.16.0.0/16 3 5 false false default + auto6-custom-kruise-cloneset-eth0-028d6 6 fc00:f853:ccd:e790::/64 3 5 false false default + ------------------------------------------------------------------------------------------ + $ kubectl get po -l app=custom-kruise-cloneset -o wide + NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES + custom-kruise-cloneset-f52dn 1/1 Running 0 61s 172.16.41.4 spider-worker 2/2 + custom-kruise-cloneset-mq67v 1/1 Running 0 61s 172.16.41.5 spider-worker 2/2 + custom-kruise-cloneset-nprpf 1/1 Running 0 61s 172.16.41.1 spider-worker 2/2 + ``` diff --git a/docs/usage/readme-zh_CN.md b/docs/usage/readme-zh_CN.md index 431225c9b4..ee82fc5be2 100644 --- a/docs/usage/readme-zh_CN.md +++ b/docs/usage/readme-zh_CN.md @@ -6,7 +6,7 @@ ### 在裸金属环境上安装 Spiderpool -集群网络可以为 Pod 接入 Spiderpool 一个或多个 Underlay 网络的网卡，从而让 Pod 具备接入 underlay 网络的能力，具体可参考 [一个或多个 underlay CNI 协同](../concepts/arch-zh_CN.md#应用场景pod-接入若干个-underlay-cni-网卡) +集群网络可以为 Pod 接入 Spiderpool 一个或多个 Underlay 网络的网卡，从而让 Pod 具备接入 underlay 网络的能力，具体可参考 [一个或多个 underlay CNI 协同](../concepts/arch-zh_CN.md#应用场景pod-接入若干个-underlay-cni-网卡) 以下是安装示例： diff --git a/docs/usage/spider-multus-config-zh_CN.md b/docs/usage/spider-multus-config-zh_CN.md index 2c92a00292..d7cc27a14d 100644 --- a/docs/usage/spider-multus-config-zh_CN.md +++ b/docs/usage/spider-multus-config-zh_CN.md @@ -449,7 +449,7 @@ EOF > `ifacer` 作为 CNI 链式调用顺序的第一个，最先被调用。 根据配置，`ifacer` 将基于 ["ens192","ens224"] 创建一个名为 `bond0` 的 bond 接口，mode 为 1(active-backup)。 > > IPVlan 作为 main CNI，其 master 字段的值为: `bond0`， bond0 承接 Pod 的网络流量。 -> +> > 创建 Bond 如果需要更高级的配置，可以通过配置 SpiderMultusConfig: macvlan-conf.spec.macvlan.bond.options 实现。 输入格式为: "primary=ens160;arp_interval=1",多个参数用";"连接 如果我们需要基于已创建的 Bond 网卡 bond0 创建 Vlan 子接口，以此 Vlan 子接口承接 Pod 的底层网络，可参考以下的配置: @@ -512,7 +512,6 @@ EOF 注意 chainCNIJsonData 每一个元素都必须是合法的 json 字符串。当创建成功，查看对应的 Multus network-attachment-definition 对象: - ```shell ~# kubectl get network-attachment-definitions.k8s.cni.cncf.io -n kube-system macvlan-ens192 -oyaml apiVersion: k8s.cni.cncf.io/v1 diff --git a/docs/usage/spider-multus-config.md b/docs/usage/spider-multus-config.md index c418133411..197dbbd120 100644 --- a/docs/usage/spider-multus-config.md +++ b/docs/usage/spider-multus-config.md @@ -4,7 +4,7 @@ ## Introduction -Spiderpool introduces the SpiderMultusConfig CR to automate the management of Multus NetworkAttachmentDefinition CR and extend the capabilities of Multus CNI configurations. +Spiderpool introduces the SpiderMultusConfig CR to automate the management of Multus NetworkAttachmentDefinition CR and extend the capabilities of Multus CNI configurations. ## SpiderMultusConfig Features diff --git a/docs/usage/underlay_cni_service-zh_CN.md b/docs/usage/underlay_cni_service-zh_CN.md index 5ce8d3154f..14dca0f2dd 100644 --- a/docs/usage/underlay_cni_service-zh_CN.md +++ b/docs/usage/underlay_cni_service-zh_CN.md @@ -99,7 +99,7 @@ default via 10.6.0.1 dev net1 上面我们介绍了在 Spiderpool 中, 我们通过 `coordinator` 将 Pod 访问 Service 的流量劫持到主机转发， 再经过主机上 Kube-proxy 设置的 iptables 规则 DNAT (将目标地址改为目标 Pod) 之后，再转发至目标 Pod。 这可以虽然解决问题，但可能延长了数据访问路径，造成一定的性能损失。 -社区开源的 CNI 项目: Cilium 支持基于 eBPF 技术完全替代 kube-proxy 系统组件。可以帮助我们解析 Service。当访问 Service 时，Service +社区开源的 CNI 项目: Cilium 支持基于 eBPF 技术完全替代 kube-proxy 系统组件。可以帮助我们解析 Service。当访问 Service 时，Service 地址会被 Cilium 挂载的 eBPF 程序直接解析为目标 Pod 的 IP，这样源 Pod 直接对目标 Pod 发起访问，而不需要经过主机的网络协议栈，极大的缩短了访问路径，从而实现加速访问 Service。借助于强大的 Cilium， 我们也可以通过它实现加速 Underlay CNI的 Service 访问。 @@ -174,7 +174,7 @@ spiderpool-controller-7df784cdb7-bsfwv 1/1 Running 0 1m spiderpool-init 0/1 Completed 0 1m ``` -创建 macvlan 相关的 multus 配置，并创建配套的 ippool 资源: +创建 macvlan 相关的 multus 配置，并创建配套的 ippool 资源: ```shell ~# cat < 10.6.185.218.43550: Flags [S.], cksum 0x9d1a (correct), seq 2119742376, ack 1391704017, win 65160, options [mss 1460,sackOK,TS val 3827707465 ecr 2667940841,nop,wscale 7], length 0 ``` -> `10.6.185.218` 是源 Pod 的 IP, `10.6.185.210` 是目标 Pod 的 IP，可以确认 Cilium 解析了 Service 的 IP。 +> `10.6.185.218` 是源 Pod 的 IP, `10.6.185.210` 是目标 Pod 的 IP，可以确认 Cilium 解析了 Service 的 IP。 使用 sockperf 工具测试使用 Cilium 加速前后， 测得 Pod 跨节点访问 ClusterIP 的数据对比: diff --git a/docs/usage/underlay_cni_service.md b/docs/usage/underlay_cni_service.md index 6113bb3b4c..3184a5486e 100644 --- a/docs/usage/underlay_cni_service.md +++ b/docs/usage/underlay_cni_service.md @@ -12,7 +12,7 @@ correctly, resulting in packet loss. Spiderpool provides the following two solutions to solve the problem of Underlay CNI accessing Service: -- Use `kube-proxy` to access Service +- Use `kube-proxy` to access Service - Use `cgroup eBPF` to access Service Both of these ways solve the problem that Underlay CNI cannot access Service, but the implementation principle is