Process request got errors when two MirrorMaker2 deployed along with two different source Kafka clusters.

[flyfax]

Source1 and source2 MM2 deployed with Kafka cluster are running in two dedicated K3s env, and are trying to sync up data to the same target destination. The data sync up from source1 to target working well. However, there are errors in source2 MM2 deployment showing that it's trying to process the request for the connector of source1.

2024-05-03 15:54:42,459 INFO 10.42.0.7 - - [03/May/2024:15:54:42 +0000] "DELETE /connectors/source1-cluster-1->target-cluster.MirrorSourceConnector HTTP/1.1" 500 180 "-" "-" 26 (org.apache.kafka.connect.runtime.rest.RestServer) [qtp726350104-68320]
2024-05-03 15:54:42,461 ERROR IO error forwarding REST request:  (org.apache.kafka.connect.runtime.rest.RestClient) [qtp726350104-107071]
java.util.concurrent.ExecutionException: java.net.UnknownHostException: source1-mirror-maker-1-mirrormaker2-0.source1-mirror-maker-1-mirrormaker2.kafka.svc
        at org.eclipse.jetty.client.util.FutureResponseListener.getResult(FutureResponseListener.java:118)
        at org.eclipse.jetty.client.util.FutureResponseListener.get(FutureResponseListener.java:101)

I'm not sure will this scenario be supported? Or Do I have any configuration issue? Any suggestions are welcome. Thanks

Here are the configurations of two deployments
Source-1 cluster configurations yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: source1-mirror-maker-1
spec:
  version: 3.7.0
  replicas: 1
  connectCluster: "target-cluster" # Must be the target custer
  clusters:
  - alias: "source1-cluster-1" # Source cluster
    bootstrapServers: source1-cluster-bootstrap:30743
  - alias: "target-cluster" # Target cluster
    bootstrapServers: target-cluster-bootstrap:30333
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "source1-cluster-1"
    targetCluster: "target-cluster"
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        sync.topic.acls.enabled: "false"
        refresh.topics.interval.seconds: 600
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        sync.group.offsets.enabled: "false"
        refresh.groups.interval.seconds: 600
    topicsPattern: ".*"
    groupsPattern: ".*"

Source-2 cluster configurations yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: source2-mirror-maker-2
spec:
  version: 3.7.0
  replicas: 1
  connectCluster: "target-cluster" # Must be the target cluster
  clusters:
  - alias: "source2-cluster-2" # Source cluster
    bootstrapServers: source2-cluster-bootstrap:32306
  - alias: "target-cluster" # Target cluster
    bootstrapServers: target-cluster-bootstrap:30333
    config:
      # -1 means it will use the default replication factor configured in the broker
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
  mirrors:
  - sourceCluster: "source2-cluster-2"
    targetCluster: "target-cluster"
    sourceConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        replication.factor: -1
        offset-syncs.topic.replication.factor: -1
        sync.topic.acls.enabled: "false"
        refresh.topics.interval.seconds: 600
    checkpointConnector:
      tasksMax: 1
      config:
        # -1 means it will use the default replication factor configured in the broker
        checkpoints.topic.replication.factor: -1
        sync.group.offsets.enabled: "false"
        refresh.groups.interval.seconds: 600
    topicsPattern: ".*"
    groupsPattern: ".*"

[scholzj]

That is likely because you didn't properly configured the MM2 instances to be separate. You should check the docs for how to configure multiple instances of MM2 with the same target cluster.

[flyfax]

@scholzj Yeah, find the information at here: https://strimzi.io/docs/operators/latest/deploying#con-config-mm2-multiple-instances-str
Thank you!