the old server wander too long after patched by the new instance

[3AceShowHand]

1. deploy a cluster with new ticdc arch instances
2. patch the cdc cluster, cause all instances restarts

[2024/12/30 17:49:24.711 +08:00] [INFO] [version.go:48] ["Welcome to Change Data Capture (CDC)"] [release-version=v8.2.0-master] [git-hash=9f7f3e3ac0395dd1d10630cb4e38703469e848f8] [git-branch=read-server-code] [utc-build-time="2024-12-30 09:48:31"] [go-version="go version go1.23.4 linux/amd64"] [failpoint-build=false]
...
[2024/12/30 17:49:24.868 +08:00] [INFO] [reactor_state.go:116] ["remote capture online"] [info="{\"id\":\"67873c92-0183-4c08-bfe3-77138d9532c8\",\"address\":\"10.2.6.80:8300\",\"version\":\"v8.2.0-master\",\"git-hash\":\"65f34ab1af943cf517863e89ae5dfb87d96ec009\",\"deploy-path\":\"/root/tidb-deploy/cdc-8300/bin/cdc\",\"start-timestamp\":1735551641}"] [role=capture-manager]
[2024/12/30 17:49:24.868 +08:00] [INFO] [reactor_state.go:116] ["remote capture online"] [info="{\"id\":\"740e2769-063c-4170-889a-f5bbcd02a71f\",\"address\":\"10.2.6.80:8300\",\"version\":\"v8.2.0-master\",\"git-hash\":\"9f7f3e3ac0395dd1d10630cb4e38703469e848f8\",\"deploy-path\":\"/root/tidb-deploy/cdc-8300/bin/cdc\",\"start-timestamp\":1735552164}"] [role=capture-manager]
[2024/12/30 17:49:24.919 +08:00] [INFO] [message_center.go:167] ["Add remote target"] [local=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [epoch=0] [addr=10.2.6.80:8300]
[2024/12/30 17:49:24.919 +08:00] [INFO] [target.go:143] ["Create remote target"] [local=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [addr=10.2.6.80:8300] [localEpoch=0] [targetEpoch=0]
[2024/12/30 17:49:24.921 +08:00] [ERROR] [message_center.go:363] ["Target not found"] [error="ErrorType: Unknown, Reason: Target 67873c92-0183-4c08-bfe3-77138d9532c8 not found"]
[2024/12/30 17:49:24.921 +08:00] [ERROR] [message_center.go:363] ["Target not found"] [error="ErrorType: Unknown, Reason: Target 67873c92-0183-4c08-bfe3-77138d9532c8 not found"]
...
[2024/12/30 17:49:38.945 +08:00] [ERROR] [message_center.go:363] ["Target not found"] [error="ErrorType: Unknown, Reason: Target 67873c92-0183-4c08-bfe3-77138d9532c8 not found"]
[2024/12/30 17:49:38.945 +08:00] [ERROR] [message_center.go:363] ["Target not found"] [error="ErrorType: Unknown, Reason: Target 67873c92-0183-4c08-bfe3-77138d9532c8 not found"]
...
[2024/12/30 17:49:38.945 +08:00] [WARN] [target.go:202] ["received message from remote failed, will be reconnect"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [error="ErrorType: MessageReceiveFailed, Reason: rpc error: code = Unknown desc = ErrorType: Unknown, Reason: Target 67873c92-0183-4c08-bfe3-77138d9532c8 not found"]
...
[2024/12/30 17:49:38.963 +08:00] [INFO] [controller.go:514] ["send coordinator bootstrap request"] [to=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:38.968 +08:00] [INFO] [reactor_state.go:77] ["remote capture offline"] [info="{\"id\":\"67873c92-0183-4c08-bfe3-77138d9532c8\",\"address\":\"10.2.6.80:8300\",\"version\":\"v8.2.0-master\",\"git-hash\":\"65f34ab1af943cf517863e89ae5dfb87d96ec009\",\"deploy-path\":\"/root/tidb-deploy/cdc-8300/bin/cdc\",\"start-timestamp\":1735551641}"] [role=capture-manager]
[2024/12/30 17:49:38.968 +08:00] [INFO] [module_node_manager.go:93] ["server change detected"]
[2024/12/30 17:49:38.968 +08:00] [INFO] [message_center.go:176] ["remove remote target from message center"] [local=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:38.969 +08:00] [INFO] [target.go:182] ["Close remote target"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [addr=10.2.6.80:8300]
[2024/12/30 17:49:39.463 +08:00] [INFO] [controller.go:514] ["send coordinator bootstrap request"] [to=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:40.946 +08:00] [INFO] [target.go:282] ["reconnect to remote target"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:40.946 +08:00] [INFO] [target.go:252] ["Cannot establish event grpc stream"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [error="rpc error: code = Canceled desc = context canceled"]
[2024/12/30 17:49:40.946 +08:00] [WARN] [target.go:202] ["received message from remote failed, will be reconnect"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [error="ErrorType: ConnectionFailed, Reason: Cannot open event grpc stream, error: rpc error: code = Canceled desc = context canceled"]
[2024/12/30 17:49:42.947 +08:00] [INFO] [target.go:282] ["reconnect to remote target"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:42.948 +08:00] [INFO] [target.go:252] ["Cannot establish event grpc stream"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [error="rpc error: code = Canceled desc = context canceled"]
[2024/12/30 17:49:42.948 +08:00] [WARN] [target.go:202] ["received message from remote failed, will be reconnect"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [error="ErrorType: ConnectionFailed, Reason: Cannot open event grpc stream, error: rpc error: code = Canceled desc = context canceled"]
[2024/12/30 17:49:44.948 +08:00] [INFO] [target.go:282] ["reconnect to remote target"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:44.949 +08:00] [INFO] [target.go:252] ["Cannot establish event grpc stream"] [messageCenterID=740e2769-063c-4170-889a-f5bbcd02a71f] [remote=67873c92-0183-4c08-bfe3-77138d9532c8] [error="rpc error: code = Canceled desc = context canceled"]
[2024/12/30 17:49:44.949 +08:00] [INFO] [coordinator.go:150] ["log coordinaotr detect node removed"] [nodeId=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:44.949 +08:00] [WARN] [etcd_worker.go:294] ["EtcdWorker reactor tick took too long"] [duration=5.980571678s] [role=capture-manager]
[2024/12/30 17:49:44.949 +08:00] [INFO] [controller.go:140] ["coordinator is too slow"] [type=1] [duration=5.485943287s]
[2024/12/30 17:49:44.949 +08:00] [INFO] [controller.go:197] ["node changed"] [new=0] [removed=1]
[2024/12/30 17:49:44.949 +08:00] [INFO] [bootstrap.go:98] ["remove node from bootstrap"] [changefeed=coordinator] [status=1] [id=67873c92-0183-4c08-bfe3-77138d9532c8]
[2024/12/30 17:49:44.949 +08:00] [INFO] [controller.go:203] ["bootstrap done after removed some nodes"]
[2024/12/30 17:49:44.949 +08:00] [INFO] [controller.go:230] ["all nodes have sent bootstrap response"] [size=1]
[2024/12/30 17:49:44.949 +08:00] [INFO] [controller.go:234] ["received bootstrap response"] [server=740e2769-063c-4170-889a-f5bbcd02a71f] [size=0]

After restart the server, the old server still alive for more than 20 seconds, this should be treated as unexpected.