[Bug]: milvus load stuck at 0% with querynode error: [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=by-dev-rootcoord-dml_6]

[eskinmi]

Is there an existing issue for this?

• [Bug]: Loading collection stuck at 0% after server's sudden electricity lost #30501

Environment

- Milvus version: 2.5.0
- Deployment mode(standalone or cluster): Cluster (EKS)
- MQ type(rocksmq, pulsar or kafka):  pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.9
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: Xeon Scalable 3.5 GHz 16vCPU & 128Gi
- GPU: N/A

Current Behavior

When I try to load my collection in Milvus, from Attu I can see that the load is stuck at 0%. I have about 20M entities in the collections that I'm trying to load. When I look into the querynode logs, I see:

[2025/01/23 21:42:50.576 +00:00] [ERROR] [msgdispatcher/dispatcher.go:122] ["seek failed"] [pchannel=by-dev-rootcoord-dml_6] [subName=querynode-48-by-dev-rootcoord-dml_6_455282050800631750v0-true] [isMain=true] [error="failed to seek, error server error: UnknownError: Error when resetting subscription: unable to persist readPosition for cursor reset 464:805"] [stack="github.com/milvus-io/milvus/pkg/mq/msgdispatcher.NewDispatcher\n\t/workspace/source/pkg/mq/msgdispatcher/dispatcher.go:122\ngithub.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*dispatcherManager).Add\n\t/workspace/source/pkg/mq/msgdispatcher/manager.go:106\ngithub.com/milvus-io/milvus/pkg/mq/msgdispatcher.(*client).Register\n\t/workspace/source/pkg/mq/msgdispatcher/client.go:94\ngithub.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).ConsumeMsgStream\n\t/workspace/source/internal/util/pipeline/stream_pipeline.go:122\ngithub.com/milvus-io/milvus/internal/querynodev2.(*QueryNode).WatchDmChannels\n\t/workspace/source/internal/querynodev2/services.go:327\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).WatchDmChannels\n\t/workspace/source/internal/distributed/querynode/service.go:296\ngithub.com/milvus-io/milvus/pkg/proto/querypb._QueryNode_WatchDmChannels_Handler.func1\n\t/workspace/source/pkg/proto/querypb/query_coord_grpc.pb.go:2020\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ServerIDValidationUnaryServerInterceptor.func7\n\t/workspace/source/pkg/util/interceptor/server_id_interceptor.go:54\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ClusterValidationUnaryServerInterceptor.func6\n\t/workspace/source/pkg/util/interceptor/cluster_interceptor.go:48\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\ngithub.com/milvus-io/milvus/pkg/util/logutil.UnaryTraceLoggerInterceptor\n\t/workspace/source/pkg/util/logutil/grpc_interceptor.go:23\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8.1.1\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:25\ngithub.com/milvus-io/milvus/internal/distributed/querynode.(*Server).startGrpcLoop.ChainUnaryServer.func8\n\t/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:34\ngithub.com/milvus-io/milvus/pkg/proto/querypb._QueryNode_WatchDmChannels_Handler\n\t/workspace/source/pkg/proto/querypb/query_coord_grpc.pb.go:2022\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1379\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1790\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1029"]

Expected Behavior

The collection should load successfully.

Steps To Reproduce

No response

Milvus Log

milvus-log.tar.gz

Anything else?

No response

[yanliang567]

/assign @bigsheeper
/unassign

[xiaofan-luan]

@eskinmi

I think this might be a different issue.
The error message showed seek failure. Which mean pulsar report an error on subscribe by-dev-rootcoord-dml_6

1. restart pulsar cluster see it works
2. try to use pulsarctl to connect to pulsar cluster, and subscribe from by-dev-rootcoord-dml_6 this topic, see if can read data
3. upgrade Milvus to 2.5.4

[xiaofan-luan]

If your Pulsar TTL (Time-To-Live) is set to a very short duration, it can also cause similar issues. When the system recovers from a failure, the corresponding position may have already been garbage collected.

[xiaofan-luan]

check which collection watch failed and use birdwatcher to see the checkpoints, if some of the checkpoints is old, we need to reset it to newer timestamp

[eskinmi]

@xiaofan-luan

Thank you very much for your help! Restarting the pulsar cluster did work indeed.
I will make sure to upgrade to the latest version.

🎉

[eskinmi]

@xiaofan-luan btw I can't see the TTL configuration in here: https://milvus.io/docs/configure_pulsar.md