Deploying Milvus CDC

[ayushk5-ai]

I am trying to install Milvus CDC, on Milvus Standalone v2.4.x
I have followed through the steps mentioned here: https://milvus.io/docs/v2.4.x/deploy-cdc-server.md

But after creating the docker image of milvus-cdc, how to proceed and what next steps to follow?

[ayushk5-ai]

"For a Docker image, mount the compiled file to /app/server/configs/cdc.yaml within the container."

What does this line means in the official documentation?

[yhmo]

No need to build the image by yourself.
There is a docker-compose.yaml under the cdc repo: https://github.com/zilliztech/milvus-cdc/blob/main/deployment/docker/docker-compose.yaml

In this docker-compose.yaml, we can see:

services:
  milvus-cdc:
    container_name: milvus-cdc
    image: milvusdb/milvus-cdc:latest
    tty: true
    volumes:
      - type: bind
        source: ./cdc.yaml
        target: /app/configs/cdc.yaml
        read_only: true
    network_mode: "host"

This docker-compose.yaml download the official image from dockhub, and launch a cdc container for you.
The "volumes" defines a path mapping for the cdc.yaml
The inner cdc.yaml is mapped to the outside cdc.yaml: https://github.com/zilliztech/milvus-cdc/blob/main/deployment/docker/cdc.yaml

The cdc.yaml defines the source etcd and source pulsar address, and the metastore for milvus-cdc to store tasks infomation.

• sourceConfig.etcd is the etcd service of the source milvus cluster
• sourceConfig.pulsar is the pulsar service of the source milvus cluster
• metaStoreConfig.etcd is the etcd service to store milvus-cdc tasks

Once the milvus-cdc container started successfully, you can create tasks to sync data to target milvus. Read this doc for more info:
https://github.com/zilliztech/milvus-cdc/blob/main/doc/cdc-usage.md

[ayushk5-ai]

services:
  etcd:
    container_name: milvus-etcd
    image: custom-etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

This is a snippet of docker-compose file of the source etcd milvus.
In cdc.yaml file, I am using the etcd address as 127.0.0.1:2379, but still not working, what should be there?

Milvus-CDC and Milvus standalone are on the same server
Also, the metastore etcd is on the same server

Where would I get the address for metaStoreConfig.etcd and sourceConfig.etcd ??
Is it the address of the linux server on which it is installed?

[yhmo]

Expose the port 2379 of the "milvus-etcd" container to the host port 12379:

services:
  etcd:
    container_name: milvus-etcd
    ......
    ports:
      - "12379:2379"

Set 127.0.0.1:12379 as the sourceConfig.etcd.address
Set 127.0.0.1:2379 as the metaStoreConfig.etcd.address

[ayushk5-ai]

Thanks, that worked

Now, while running the container, it gives this and container isn't starting

This is my cdc.yaml file

address: 0.0.0.0:8444
maxTaskNum: 100
metaStoreConfig:
  storeType: etcd
  etcd:
    address:
      - 127.0.0.1:2379
    enableAuth: false
    username: analytics
    password:
    enableTLS: false
    tlsCertPath: deployment/cert/client.pem # path to your cert file
    tlsKeyPath: deployment/cert/client.key # path to your key file
    tlsCACertPath: deployment/cert/ca.pem # path to your CACert file
    tlsMinVersion: 1.3
#  mysqlSourceUrl: root:root@tcp(127.0.0.1:3306)/milvuscdc?charset=utf8
  rootPath: /data/analytics/milvus/volumes/etcd-cdc
sourceConfig:
  etcd:
    address:
      - 127.0.0.1:2379
    rootPath: /data/analytics/milvus/volumes/etcd
    metaSubPath: meta
    enableAuth: false
    username: analytics
    password:
    enableTLS: false
    tlsCertPath: deployment/cert/client.pem # path to your cert file
    tlsKeyPath: deployment/cert/client.key # path to your key file
    tlsCACertPath: deployment/cert/ca.pem # path to your CACert file
    tlsMinVersion: 1.3
  readChanLen: 4
  defaultPartitionName: _default
  replicateChan: by-dev-replicate-msg
  pulsar:
    address: pulsar://localhost:6650
    webAddress: localhost:80
    maxMessageSize: 5242880
    tenant: public
    namespace: default
#    authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken
#    authParams: token:xxx
#  kafka:
#    address: 127.0.0.1:9092
maxNameLength: 256
logLevel: info
detectDeadLock: false

sourceConfig.rootpath is the location of my milvus etcd storage
metaStoreConfig.rootpath is the location of my cdc etcd storage

What is causing my current issue?