Skip to content

Commit

Permalink
supplement IP GC ENV for various scenarios
Browse files Browse the repository at this point in the history 
Signed-off-by: Icarus9913 <[email protected]>
  • Loading branch information
@Icarus9913
Icarus9913 committed Jan 24, 2024
1 parent ca8606e commit 63213d4
Show file tree
Hide file tree
Showing 19 changed files with 367 additions and 101 deletions.
3 changes: 2 additions & 1 deletion .github/.spelling
View file
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,6 @@ SPIDERPOOL_CLI_PORT
SPIDERPOOL_UPDATE_CR_MAX_RETRIES
SPIDERPOOL_WORKLOADENDPOINT_MAX_HISTORY_RECORDS
SPIDERPOOL_IPPOOL_MAX_ALLOCATED_IPS
SPIDERPOOL_GC_TERMINATING_POD_IP_ENABLED
SPIDERPOOL_GC_DEFAULT_INTERVAL_DURATION
kubernetes
vlan
Expand Down Expand Up @@ -294,3 +293,5 @@ auto_pool_scale_max_duration_seconds
auto_pool_scale_min_duration_seconds
auto_pool_scale_latest_duration_seconds
auto_pool_scale_duration_seconds_histogram
SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED
SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED
25 changes: 13 additions & 12 deletions charts/spiderpool/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,18 +127,19 @@ helm install spiderpool spiderpool/spiderpool --wait --namespace kube-system \

### ipam parameters

| Name | Description | Value |
| -------------------------------------- | --------------------------------------------------------------------------- | ------ |
| `ipam.enableIPv4` | enable ipv4 | `true` |
| `ipam.enableIPv6` | enable ipv6 | `true` |
| `ipam.enableStatefulSet` | the network mode | `true` |
| `ipam.enableKubevirtStaticIP` | the feature to keep kubevirt vm pod static IP | `true` |
| `ipam.enableSpiderSubnet` | SpiderSubnet feature gate. | `true` |
| `ipam.subnetDefaultFlexibleIPNumber` | the default flexible IP number of SpiderSubnet feature auto-created IPPools | `1` |
| `ipam.gc.enabled` | enable retrieve IP in spiderippool CR | `true` |
| `ipam.gc.gcAll.intervalInSecond` | the gc all interval duration | `600` |
| `ipam.gc.GcDeletingTimeOutPod.enabled` | enable retrieve IP for the pod who times out of deleting graceful period | `true` |
| `ipam.gc.GcDeletingTimeOutPod.delay` | the gc delay seconds after the pod times out of deleting graceful period | `0` |
| Name | Description | Value |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------ |
| `ipam.enableIPv4` | enable ipv4 | `true` |
| `ipam.enableIPv6` | enable ipv6 | `true` |
| `ipam.enableStatefulSet` | the network mode | `true` |
| `ipam.enableKubevirtStaticIP` | the feature to keep kubevirt vm pod static IP | `true` |
| `ipam.enableSpiderSubnet` | SpiderSubnet feature gate. | `true` |
| `ipam.subnetDefaultFlexibleIPNumber` | the default flexible IP number of SpiderSubnet feature auto-created IPPools | `1` |
| `ipam.gc.enabled` | enable retrieve IP in spiderippool CR | `true` |
| `ipam.gc.gcAll.intervalInSecond` | the gc all interval duration | `600` |
| `ipam.gc.statelessPod.zombieOnReadyNode` | enable reclaim IP for the stateless pod who is over deleting graceful period on a ready node | `true` |
| `ipam.gc.statelessPod.zombieOnNotReadyNode` | enable reclaim IP for the stateless pod who is over deleting graceful period on a not-ready node | `true` |
| `ipam.gc.gcDeletingTimeOutPodDelay` | the gc delay seconds after the pod times out of deleting graceful period | `0` |

### grafanaDashboard parameters

Expand Down
8 changes: 5 additions & 3 deletions charts/spiderpool/templates/deployment.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -161,10 +161,12 @@ spec:
value: {{ .Values.spiderpoolController.httpPort | quote }}
- name: SPIDERPOOL_GC_IP_ENABLED
value: {{ .Values.ipam.gc.enabled | quote }}
- name: SPIDERPOOL_GC_TERMINATING_POD_IP_ENABLED
value: {{ .Values.ipam.gc.GcDeletingTimeOutPod.enabled | quote }}
- name: SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED
value: {{ .Values.ipam.gc.statelessPod.zombieOnReadyNode | quote }}
- name: SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED
value: {{ .Values.ipam.gc.statelessPod.zombieOnNotReadyNode | quote }}
- name: SPIDERPOOL_GC_ADDITIONAL_GRACE_DELAY
value: {{ .Values.ipam.gc.GcDeletingTimeOutPod.delay | quote }}
value: {{ .Values.ipam.gc.gcDeletingTimeOutPodDelay | quote }}
- name: SPIDERPOOL_GC_DEFAULT_INTERVAL_DURATION
value: {{ .Values.ipam.gc.gcAll.intervalInSecond | quote }}
- name: SPIDERPOOL_MULTUS_CONFIG_ENABLED
Expand Down
13 changes: 8 additions & 5 deletions charts/spiderpool/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,15 @@ ipam:
## @param ipam.gc.gcAll.intervalInSecond the gc all interval duration
intervalInSecond: 600

GcDeletingTimeOutPod:
## @param ipam.gc.GcDeletingTimeOutPod.enabled enable retrieve IP for the pod who times out of deleting graceful period
enabled: true
statelessPod:
## @param ipam.gc.statelessPod.zombieOnReadyNode enable reclaim IP for the stateless pod who is over deleting graceful period on a ready node
zombieOnReadyNode: true

## @param ipam.gc.GcDeletingTimeOutPod.delay the gc delay seconds after the pod times out of deleting graceful period
delay: 0
## @param ipam.gc.statelessPod.zombieOnNotReadyNode enable reclaim IP for the stateless pod who is over deleting graceful period on a not-ready node
zombieOnNotReadyNode: true

## @param ipam.gc.gcDeletingTimeOutPodDelay the gc delay seconds after the pod times out of deleting graceful period
gcDeletingTimeOutPodDelay: 0

## @section grafanaDashboard parameters
##
Expand Down
3 changes: 2 additions & 1 deletion cmd/spiderpool-controller/cmd/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,8 @@ var envInfo = []envConf{
{"SPIDERPOOL_PYROSCOPE_PUSH_SERVER_ADDRESS", "", false, &controllerContext.Cfg.PyroscopeAddress, nil, nil},

{"SPIDERPOOL_GC_IP_ENABLED", "true", true, nil, &gcIPConfig.EnableGCIP, nil},
{"SPIDERPOOL_GC_TERMINATING_POD_IP_ENABLED", "true", true, nil, &gcIPConfig.EnableGCForTerminatingPod, nil},
{"SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_READY_NODE_ENABLED", "true", true, nil, &gcIPConfig.EnableGCStatelessTerminatingPodOnReadyNode, nil},
{"SPIDERPOOL_GC_STATELESS_TERMINATING_POD_ON_NOT_READY_NODE_ENABLED", "true", true, nil, &gcIPConfig.EnableGCStatelessTerminatingPodOnNotReadyNode, nil},
{"SPIDERPOOL_GC_IP_WORKER_NUM", "3", true, nil, nil, &gcIPConfig.ReleaseIPWorkerNum},
{"SPIDERPOOL_GC_CHANNEL_BUFFER", "5000", true, nil, nil, &gcIPConfig.GCIPChannelBuffer},
{"SPIDERPOOL_GC_MAX_PODENTRY_DB_CAP", "100000", true, nil, nil, &gcIPConfig.MaxPodEntryDatabaseCap},
Expand Down
1 change: 1 addition & 0 deletions cmd/spiderpool-controller/cmd/daemon.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,7 @@ func initGCManager(ctx context.Context) {
controllerContext.PodManager,
controllerContext.StsManager,
controllerContext.KubevirtManager,
controllerContext.NodeManager,
controllerContext.Leader,
)
if nil != err {
Expand Down
38 changes: 20 additions & 18 deletions docs/concepts/coordinator-zh_CN.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,24 +24,24 @@ Spiderpool 内置一个叫 `coordinator` 的 CNI meta-plugin, 它在 Main CNI

## CNI 配置字段说明

| Field | Description | Schema | Validation | Default |
|-----------|---------------------------------------------------|--------|------------|---------|
| type | CNI 的类型 | 字符串 | required |coordinator |
| mode | coordinator 运行的模式. "auto": coordinator 自动判断运行在 Underlay 或者 Overlay; "underlay": 为 Pod 创建一对 Veth 设备，用于转发集群东西向流量。由 Pod 的 Underlay 网卡转发南北向流量; "overlay": 不额外创建 veth 设备，运行在多网卡模式。由 overlay 类型的 CNI(calico，cilium) 转发集群东西向流量，由 underlay 网卡转发南北向流量; "disable": 禁用 coordinator | 字符串 | optional | auto |
| tunePodRoutes | Pod 多网卡模式下，是否调协 Pod 的路由，解决访问来回路径不一致的问题 | 布尔型 | optional | true |
| podDefaultRouteNic | Pod 多网卡时，配置 Pod 的默认路由网卡。默认为 "", 其 value 实际为 Pod 第一张拥有默认路由的网卡| 字符串 | optional | "" |
| podDefaultCniNic | K8s 中 Pod 默认的第一张网卡 | 布尔型 | optional | eth0 |
| detectGateway | 创建 Pod 时是否检查网关是否可达 | 布尔型 | optional | false |
| detectIPConflict | 创建 Pod 时是否检查 Pod 的 IP 是否可达 | 布尔型 | optional | false |
| podMACPrefix | 是否固定 Pod 的 Mac 地址前缀 | 字符串 | optional | "" |
| overlayPodCIDR | 默认的集群 Pod 的子网，会注入到 Pod 中。不需要配置，自动从 Spidercoordinator default 中获取 | []stirng | optional | 默认从 Spidercoordinator default 中获取 |
| serviceCIDR | 默认的集群 Service 子网， 会注入到 Pod 中。不需要配置，自动从 Spidercoordinator default 中获取 | []stirng | optional | 默认从 Spidercoordinator default 中获取 |
| hijackCIDR | 额外的需要从主机转发的子网路由。比如nodelocaldns 的地址: 169.254.20.10/32 | []stirng | optional | 空 |
| hostRuleTable | 策略路由表号，同主机与 Pod 通信的路由将会存放于这个表号 | 整数型 | optional | 500 |
| hostRPFilter | 设置主机上的 sysctl 参数 rp_filter | 整数型 | optional | 0 |
| txQueueLen | 设置 Pod 的网卡传输队列 | 整数型 | optional | 0 |
| detectOptions | 检测地址冲突和网关可达性的高级配置项: 包括重试次数(默认为 3 次), 探测间隔(默认为 1s) 和 超时时间(默认为 1s) | 对象类型 | optional | 空 |
| logOptions | 日志配置，包括 logLevel(默认为 debug) 和 logFile(默认为 /var/log/spidernet/coordinator.log) | 对象类型 | optional | - |
| Field | Description | Schema | Validation | Default |
|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|------------|-----------------------------------|
| type | CNI 的类型 | 字符串 | required | coordinator |
| mode | coordinator 运行的模式. "auto": coordinator 自动判断运行在 Underlay 或者 Overlay; "underlay": 为 Pod 创建一对 Veth 设备，用于转发集群东西向流量。由 Pod 的 Underlay 网卡转发南北向流量; "overlay": 不额外创建 veth 设备，运行在多网卡模式。由 overlay 类型的 CNI(calico，cilium) 转发集群东西向流量，由 underlay 网卡转发南北向流量; "disable": 禁用 coordinator | 字符串 | optional | auto |
| tunePodRoutes | Pod 多网卡模式下，是否调协 Pod 的路由，解决访问来回路径不一致的问题 | 布尔型 | optional | true |
| podDefaultRouteNic | Pod 多网卡时，配置 Pod 的默认路由网卡。默认为 "", 其 value 实际为 Pod 第一张拥有默认路由的网卡 | 字符串 | optional | "" |
| podDefaultCniNic | K8s 中 Pod 默认的第一张网卡 | 布尔型 | optional | eth0 |
| detectGateway | 创建 Pod 时是否检查网关是否可达 | 布尔型 | optional | false |
| detectIPConflict | 创建 Pod 时是否检查 Pod 的 IP 是否冲突 | 布尔型 | optional | false |
| podMACPrefix | 是否固定 Pod 的 Mac 地址前缀 | 字符串 | optional | "" |
| overlayPodCIDR | 默认的集群 Pod 的子网，会注入到 Pod 中。不需要配置，自动从 Spidercoordinator default 中获取 | []stirng | optional | 默认从 Spidercoordinator default 中获取 |
| serviceCIDR | 默认的集群 Service 子网， 会注入到 Pod 中。不需要配置，自动从 Spidercoordinator default 中获取 | []stirng | optional | 默认从 Spidercoordinator default 中获取 |
| hijackCIDR | 额外的需要从主机转发的子网路由。比如nodelocaldns 的地址: 169.254.20.10/32 | []stirng | optional | 空 |
| hostRuleTable | 策略路由表号，同主机与 Pod 通信的路由将会存放于这个表号 | 整数型 | optional | 500 |
| hostRPFilter | 设置主机上的 sysctl 参数 rp_filter | 整数型 | optional | 0 |
| txQueueLen | 设置 Pod 的网卡传输队列 | 整数型 | optional | 0 |
| detectOptions | 检测地址冲突和网关可达性的高级配置项: 包括重试次数(默认为 3 次), 探测间隔(默认为 1s) 和 超时时间(默认为 1s) | 对象类型 | optional | 空 |
| logOptions | 日志配置，包括 logLevel(默认为 debug) 和 logFile(默认为 /var/log/spidernet/coordinator.log) | 对象类型 | optional | - |

> 如果您通过 `SpinderMultusConfig CR` 帮助创建 NetworkAttachmentDefinition CR，您可以在 `SpinderMultusConfig` 中配置 `coordinator` (所有字段)。参考: [SpinderMultusConfig](../reference/crd-spidermultusconfig.md)。
>
Expand Down Expand Up @@ -75,6 +75,8 @@ spec:
detectIPConflict: true # Enable detectIPConflict
```
> 若 IP 冲突检查发现某 IP 已被占用，请检查是否被集群中其他处于 `Terminating` 阶段的 **无状态** Pod 所占用，并配合 [IP 回收机制](./ipam-des-zh_CN.md#ip-回收机制) 相关参数进行配置。

## 支持检测 Pod 的网关是否可达(alpha)

在 Underlay 网络下，Pod 访问外部需要通过网关转发。如果网关不可达，那么在外界看来，这个 Pod 实际是失联的。有时候我们希望创建 Pod 时，其网关是可达的。 我们可借助 `coordinator` 检测 Pod 的网关是否可达，
Expand Down
Loading

0 comments on commit 63213d4

Please sign in to comment.