Spidercoordinator: It able to get CIDR from kubeadm-config

If the kube-controller-manager Pod is running as systemd precess rather than Pod, In this case, We can't get the CIDR from the KCM Pod. We can get the CIDR from the kubeadm-config configMap.

Signed-off-by: cyclinder <[email protected]>

docs/usage/install/overlay/get-started-calico-zh_cn.md

@@ -87,10 +87,31 @@ status:
   serviceCIDR:
   - 10.233.0.0/18
 ```
+
+> 目前 Spiderpool 优先通过查询 `kube-system/kubeadm-config` ConfigMap 获取集群的 Pod 和 Service 子网。 如果 kubeadm-config 不存在导致无法获取集群子网，那么 Spiderpool 会从 Kube-controller-manager Pod 中获取集群 Pod 和 Service 的子网。 如果您集群的 Kube-controller-manager 组件以 `systemd` 方式而不是以静态 Pod 运行。那么 Spiderpool 仍然无法获取集群的子网信息。
 
-> 1.如果 phase 不为 Synced, 那么将会阻止 Pod 被创建
->
-> 2.如果 overlayPodCIDR 不正常, 可能会导致通信问题
+如果上面两种方式都失败，Spiderpool 会同步 status.phase 为 NotReady, 这将会阻止 Pod 被创建。我们可以通过下面解决异常情况:
+
+- 手动创建 kubeadm-config ConfigMap, 并正确配置集群的子网信息:
+
+```shell
+export POD_SUBNET=<YOUR_POD_SUBNET>
+export SERVICE_SUBNET=<YOUR_SERVICE_SUBNET>
+cat << EOF | kubectl apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kubeadm-config
+  namespace: kube-system
+data:
+  ClusterConfiguration: 
+    networking:
+      podSubnet: ${POD_SUBNET}
+      serviceSubnet: ${SERVICE_SUBNET}
+EOF
+```
+
+一旦创建完成，Spiderpool 将会自动同步其状态。
 
 ### 创建 SpiderIPPool
 

docs/usage/install/overlay/get-started-calico.md

@@ -83,9 +83,30 @@ status:
   - 10.233.0.0/18
 ```
 
-> 1.If the phase is not synced, the pod will be prevented from being created.
->
-> 2.If the overlayPodCIDR does not meet expectations, it may cause pod communication issue.
+> At present, Spiderpool prioritizes obtaining the cluster's Pod and Service subnets by querying the kube-system/kubeadm-config ConfigMap. If the kubeadm-config does not exist, causing the failure to obtain the cluster subnet, Spiderpool will attempt to retrieve the cluster Pod and Service subnets from the kube-controller-manager Pod. If the kube-controller-manager component in your cluster runs in systemd mode instead of as a static Pod, Spiderpool still cannot retrieve the cluster's subnet information.
+
+If both of the above methods fail, Spiderpool will synchronize the status.phase as NotReady, preventing Pod creation. To address such abnormal situations, we can take either of the following approaches:
+
+- Manually create the kubeadm-config ConfigMap and correctly configure the cluster's subnet information:
+
+```shell
+export POD_SUBNET=<YOUR_POD_SUBNET>
+export SERVICE_SUBNET=<YOUR_SERVICE_SUBNET>
+cat << EOF | kubectl apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kubeadm-config
+  namespace: kube-system
+data:
+  ClusterConfiguration: 
+    networking:
+      podSubnet: ${POD_SUBNET}
+      serviceSubnet: ${SERVICE_SUBNET}
+EOF
+```
+
+Once created, Spiderpool will automatically synchronize its status.
 
 ### Create SpiderIPPool
 

docs/usage/install/overlay/get-started-cilium-zh_cn.md

@@ -84,9 +84,30 @@ status:
   - 10.233.0.0/18
 ```
 
-> 1.如果 phase 不为 Synced, 那么将会阻止 Pod 被创建
->
-> 2.如果 overlayPodCIDR 不正常, 可能会导致通信问题
+> 目前 Spiderpool 优先通过查询 `kube-system/kubeadm-config` ConfigMap 获取集群的 Pod 和 Service 子网。 如果 kubeadm-config 不存在导致无法获取集群子网，那么 Spiderpool 会从 Kube-controller-manager Pod 中获取集群 Pod 和 Service 的子网。 如果您集群的 Kube-controller-manager 组件以 `systemd` 方式而不是以静态 Pod 运行。那么 Spiderpool 仍然无法获取集群的子网信息。
+
+如果上面两种方式都失败，Spiderpool 会同步 status.phase 为 NotReady, 这将会阻止 Pod 被创建。我们可以通过下面的方式解决异常情况:
+
+- 手动创建 kubeadm-config ConfigMap, 并正确配置集群的子网信息:
+
+```shell
+export POD_SUBNET=<YOUR_POD_SUBNET>
+export SERVICE_SUBNET=<YOUR_SERVICE_SUBNET>
+cat << EOF | kubectl apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kubeadm-config
+  namespace: kube-system
+data:
+  ClusterConfiguration: |
+  networking:
+    podSubnet: ${POD_SUBNET}
+    serviceSubnet: ${SERVICE_SUBNET}
+EOF
+```
+
+一旦创建完成，Spiderpool 将会自动同步其状态。
 
 ### 创建 SpiderIPPool
 

docs/usage/install/overlay/get-started-cilium.md

@@ -84,9 +84,30 @@ status:
   - 10.233.0.0/18
 ```
 
-> 1.If the phase is not synced, the pod will be prevented from being created.
->
-> 2.If the overlayPodCIDR does not meet expectations, it may cause pod communication issue.
+> At present, Spiderpool prioritizes obtaining the cluster's Pod and Service subnets by querying the kube-system/kubeadm-config ConfigMap. If the kubeadm-config does not exist, causing the failure to obtain the cluster subnet, Spiderpool will attempt to retrieve the cluster Pod and Service subnets from the kube-controller-manager Pod. If the kube-controller-manager component in your cluster runs in systemd mode instead of as a static Pod, Spiderpool still cannot retrieve the cluster's subnet information.
+
+If both of the above methods fail, Spiderpool will synchronize the status.phase as NotReady, preventing Pod creation. To address such abnormal situations, we can take either of the following approaches:
+
+- Manually create the kubeadm-config ConfigMap and correctly configure the cluster's subnet information:
+
+```shell
+export POD_SUBNET=<YOUR_POD_SUBNET>
+export SERVICE_SUBNET=<YOUR_SERVICE_SUBNET>
+cat << EOF | kubectl apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kubeadm-config
+  namespace: kube-system
+data:
+  ClusterConfiguration: 
+  networking:
+    podSubnet: ${POD_SUBNET}
+    serviceSubnet: ${SERVICE_SUBNET}
+EOF
+```
+
+Once created, Spiderpool will automatically synchronize its status.
 
 ### Create SpiderIPPool
 

pkg/coordinatormanager/coordinator_informer.go

@@ -334,36 +334,8 @@ func (cc *CoordinatorController) syncHandler(ctx context.Context, coordinatorNam
 }
 
 func (cc *CoordinatorController) fetchPodAndServerCIDR(ctx context.Context, logger *zap.Logger, coordCopy *spiderpoolv2beta1.SpiderCoordinator) (*spiderpoolv2beta1.SpiderCoordinator, error) {
-	var err error
-	var cmPodList corev1.PodList
-	if err := cc.APIReader.List(ctx, &cmPodList, client.MatchingLabels{"component": "kube-controller-manager"}); err != nil {
-		event.EventRecorder.Eventf(
-			coordCopy,
-			corev1.EventTypeWarning,
-			"ClusterNotReady",
-			err.Error(),
-		)
-
-		setStatus2NoReady(logger, coordCopy)
-		return coordCopy, err
-	}
-	if len(cmPodList.Items) == 0 {
-		msg := `Failed to get kube-controller-manager Pod with label "component: kube-controller-manager"`
-		event.EventRecorder.Eventf(
-			coordCopy,
-			corev1.EventTypeWarning,
-			"ClusterNotReady",
-			msg,
-		)
-
-		setStatus2NoReady(logger, coordCopy)
-		return coordCopy, err
-	}
-
-	k8sPodCIDR, k8sServiceCIDR := extractK8sCIDR(&cmPodList.Items[0])
 	if *coordCopy.Spec.PodCIDRType == auto {
-		var podCidrType string
-		podCidrType, err = fetchType(cc.DefaultCniConfDir)
+		podCidrType, err := fetchType(cc.DefaultCniConfDir)
 		if err != nil {
 			if apierrors.IsNotFound(err) {
 				event.EventRecorder.Eventf(
@@ -381,6 +353,30 @@ func (cc *CoordinatorController) fetchPodAndServerCIDR(ctx context.Context, logg
 		coordCopy.Spec.PodCIDRType = &podCidrType
 	}
 
+	var err error
+	var cm *corev1.ConfigMap
+	var k8sPodCIDR, k8sServiceCIDR []string
+	if err := cc.APIReader.Get(ctx, types.NamespacedName{Namespace: metav1.NamespaceSystem, Name: "kubeadm-config"}, cm); err == nil && cm != nil {
+		logger.Sugar().Infof("Trying to fetch the ClusterCIDR from kube-system/kubeadm-config")
+		k8sPodCIDR, k8sServiceCIDR = ExtractK8sCIDRFromKubeadmConfigMap(cm)
+	} else {
+		logger.Sugar().Warn("kube-system/kubeadm-config is no found, trying to fetch the ClusterCIDR from kube-controller-manager Pod")
+		var cmPodList corev1.PodList
+		err = cc.APIReader.List(ctx, &cmPodList, client.MatchingLabels{"component": "kube-controller-manager"})
+		if err != nil {
+			logger.Sugar().Errorf("failed to get kube-controller-manager Pod with label \"component: kube-controller-manager\": %v", err)
+			event.EventRecorder.Eventf(
+				coordCopy,
+				corev1.EventTypeWarning,
+				"ClusterNotReady",
+				"Neither kubeadm-config ConfigMap nor kube-controller-manager Pod can be found",
+			)
+			setStatus2NoReady(logger, coordCopy)
+			return coordCopy, err
+		}
+		k8sPodCIDR, k8sServiceCIDR = ExtractK8sCIDRFromKCMPod(&cmPodList.Items[0])
+	}
+
 	switch *coordCopy.Spec.PodCIDRType {
 	case cluster:
 		if cc.caliCtrlCanncel != nil {
@@ -538,7 +534,42 @@ func (cc *CoordinatorController) fetchCiliumCIDR(ctx context.Context, logger *za
 	return nil
 }
 
-func extractK8sCIDR(kcm *corev1.Pod) ([]string, []string) {
+func ExtractK8sCIDRFromKubeadmConfigMap(cm *corev1.ConfigMap) ([]string, []string) {
+	var podCIDR, serviceCIDR []string
+
+	podReg := regexp.MustCompile(`--cluster-cidr=(.*)`)
+	serviceReg := regexp.MustCompile(`--service-cluster-ip-range=(.*)`)
+
+	var podSubnets, serviceSubnets []string
+	for _, data := range cm.Data {
+		podSubnets = podReg.FindStringSubmatch(data)
+		serviceSubnets = serviceReg.FindStringSubmatch(data)
+	}
+
+	if len(podSubnets) != 0 {
+		for _, cidr := range strings.Split(podSubnets[1], ",") {
+			_, _, err := net.ParseCIDR(cidr)
+			if err != nil {
+				continue
+			}
+			podCIDR = append(podCIDR, cidr)
+		}
+	}
+
+	if len(serviceSubnets) != 0 {
+		for _, cidr := range strings.Split(serviceSubnets[1], ",") {
+			_, _, err := net.ParseCIDR(cidr)
+			if err != nil {
+				continue
+			}
+			serviceCIDR = append(serviceCIDR, cidr)
+		}
+	}
+
+	return podCIDR, serviceCIDR
+}
+
+func ExtractK8sCIDRFromKCMPod(kcm *corev1.Pod) ([]string, []string) {
 	var podCIDR, serviceCIDR []string
 
 	podReg := regexp.MustCompile(`--cluster-cidr=(.*)`)

test/doc/spidercoodinator.md

@@ -1,12 +1,13 @@
 # E2E Cases for spidercoordinator
 
-| Case ID | Title                                                                    | Priority | Smoke | Status | Other |
-| ------- | ------------------------------------------------------------------------ | -------- | ----- | ------ | ----- |
-| V00001  | Switch podClusterType to `auto`, see if it could auto fetch the type       | p3       |       |        |       |
-| V00002  | Switch podClusterType to `auto` but no cni files in /etc/cni/net.d, see if the phase is NotReady       | p3       |       |        |       |
-| V00003  | Switch podClusterType to `calico`, see if it could auto fetch the cidr from calico ippools   | p3       |       |        |       |
-| V00004  | Switch podClusterType to `cilium`, see if it works in ipam-mode: [cluster-pool,kubernetes,multi-pool]   | p3       |       |        |       |
-| V00005  | Switch podClusterType to `none`, expect the cidr of status to be empty   | p3       |       |        |       |
-| V00006  | status.phase is not-ready, expect the cidr of status to be empty         | p3       |       |        |       |
-| V00007  | spidercoordinator has the lowest priority                                | p3       |       |        |       |
-| V00008  | status.phase is not-ready, pods will fail to run                         | p3       |       |        |       |
+| Case ID | Title                                                                                                     | Priority | Smoke | Status | Other |
+| ------- | --------------------------------------------------------------------------------------------------------- | -------- | ----- | ------ | ----- |
+| V00001  | Switch podCIDRType to `auto`, see if it could auto fetch the type                                         | p3       |       |  done  |       |
+| V00002  | Switch podCIDRType to `auto` but no cni files in /etc/cni/net.d, Viewing should be consistent with `none` | p3       |       |  done  |       |
+| V00003  | Switch podCIDRType to `calico`, see if it could auto fetch the cidr from calico ippools                   | p3       |       |  done  |       |
+| V00004  | Switch podCIDRType to `cilium`, see if it works in ipam-mode: [cluster-pool,kubernetes,multi-pool]        | p3       |       |  done  |       |
+| V00005  | Switch podCIDRType to `none`, expect the cidr of status to be empty                                       | p3       |       |  done  |       |
+| V00006  | status.phase is not-ready, expect the cidr of status to be empty                                          | p3       |       |  done  |       |
+| V00007  | spidercoordinator has the lowest priority                                                                 | p3       |       |  done  |       |
+| V00008  | status.phase is not-ready, pods will fail to run                                                          | p3       |       |  done  |       |
+| V00009 | it can get the clusterCIDR from kubeadmConfig or kube-controller-manager pod | p3 |  | done|

test/e2e/common/constant.go

@@ -40,6 +40,8 @@ var ForcedWaitingTime = time.Second
 const (
 	SpiderPoolConfigmapName      = "spiderpool-conf"
 	SpiderPoolConfigmapNameSpace = "kube-system"
+	SpiderPoolLeases             = "spiderpool-controller-leases"
+	SpiderPoolLeasesNamespace    = "kube-system"
 )
 
 // Kubeadm configurations
@@ -88,6 +90,17 @@ var (
 	NIC2 string = "net1"
 	NIC3 string = "eth0.100"
 	NIC4 string = "eth0.200"
+	NIC5 string = "eth1"
+
+	// Spidercoodinator podCIDRType
+	PodCIDRTypeAuto    = "auto"
+	PodCIDRTypeCluster = "cluster"
+	PodCIDRTypeCalico  = "calico"
+	PodCIDRTypeCilium  = "cilium"
+	PodCIDRTypeNone    = "none"
+
+	// Spidercoodinator default config
+	SpidercoodinatorDefaultName = "default"
 )
 
 // Error
@@ -104,7 +117,9 @@ const (
 
 // Webhook Port
 const (
-	WebhookPort = "5722"
+	WebhookPort                 = "5722"
+	SpiderControllerMetricsPort = "5721"
+	SpiderAgentMetricsPort      = "5711"
 )
 
 func init() {

test/e2e/common/mode.go

@@ -11,6 +11,7 @@ import (
 const (
 	ENV_INSTALL_OVERLAY          = "INSTALL_OVERLAY_CNI"
 	E2E_SPIDERPOOL_ENABLE_SUBNET = "E2E_SPIDERPOOL_ENABLE_SUBNET"
+	INSTALL_CALICO               = "INSTALL_CALICO"
 	INSTALL_CILIUM               = "INSTALL_CILIUM"
 )
 
@@ -31,6 +32,10 @@ func CheckSubnetFeatureOn() bool {
 	return checkBoolEnv(E2E_SPIDERPOOL_ENABLE_SUBNET)
 }
 
+func CheckCalicoFeatureOn() bool {
+	return checkBoolEnv(INSTALL_CALICO)
+}
+
 func CheckCiliumFeatureOn() bool {
 	return checkBoolEnv(INSTALL_CILIUM)
 }