Many CCM in one cluster.

[sergelogvinov]

Hello. I'd like to launch two CCM in one k8s cluster (one control plane)

Currently, CCM is trying to manage all nodes in the cluster, even if they have different ProviderName.
And one CCM can delete a node which uninitialized because it does not find the node in the cloud provider and does not have ProviderID yet.

Can we skip managing the foreign nodes? And add delete delay for uninitialized nodes?

a few changes here sergelogvinov@8468b74

So, what do you think, is it possible to solve the problem like this?

[cheftako]

Having multiple copies of the same controller operating simultaneously is asking for race conditions. Currently we prevent this using controller manager level leader election. I am assuming that you have already given one of the two CCMs an alternate leader election name to circumvent the leader election protection. If you want to go that route I would suggest using the controllers flag (https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/controller-manager/options/generic.go#L65) to control the controllers in each CCM to ensure that each controller is only running in one of the two CCMs.

[sergelogvinov]

Sorry, I'll add more details:

I have two cloud providers with a different ProviderName (nameA/NameB). I have deployments for each cloud-provider.
Deployment(CCM) has its own lock resource --leader-elect-resource-name=nameA/nameB. It solves race conditions for one type of CCM (for one cloud provider).

All CCMs can initialize the nodes and set providerID (nameA://instanceId or nameB://instanceId)

And based on providerName(nameX://), CCMs can choose the right node and make a node lifecycle.
In this case CCM have to skip the nodes from another cloud provider. sergelogvinov@8468b74#diff-807a869d357013a377e7a6153dc3133491ca7dfb909690f0dfa2b1f2b873fa60R131-R135

And problem her - when the node joins the cluster, it does not have a providerID yet.
One of CCMs will delete the node because a cloud provider does not have this node. It belongs to another cloud provider.
If wait a few second, another CCM initialize the node and set providerID. sergelogvinov@8468b74#diff-807a869d357013a377e7a6153dc3133491ca7dfb909690f0dfa2b1f2b873fa60R162-R169

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[james-callahan]

I have this same question: how has anyone implemented a multi-cloud kubernetes cluster without this?

Related: #35