[Bug]: Error Milvus can't load Segment

[danielelongo]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: milvusdb/milvus:v2.5.0
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.5.4
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: m5.2xlarge aws machine
- GPU: No
- Others:

Current Behavior

I don't know how to replicate this issue, but basically after a long period of inactivity (such as weekend) the query engine doesn't work anymore.

I'm loading the collection in memory and releasing at each query, could the problem be in this logic?

Expected Behavior

No response

Steps To Reproduce

Milvus Log

[2025/01/27 10:23:55.511 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=a-bucket, object=files/index_files/455409384957061896/0/455409384953479403/455409384957061895/index_null_offset]"] [errorVerbose="At LoadSegment: Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=a-bucket, object=files/index_files/455409384957061896/0/455409384953479403/455409384957061895/index_null_offset]\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:336\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=a-bucket, object=files/index_files/455409384957061896/0/455409384953479403/455409384957061895/index_null_offset]\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]

Anything else?

No response

[danielelongo]

Collection loading still at 0%

[yanliang567]

@danielelongo it seems that the bucket is broken or the files in bucket is dirty. did you meet some unplanned reboot for something like that? could you please attache the full milvus logs?
If you install Milvus with docker-compose, please use docker-compose logs > milvus.log to export the logs.
/assign @danielelongo

[xiaofan-luan]

lang.org/toolchain@v0.0.1-go1.22.0.linux-amd64/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) Error in GetObjectSize[errcode:404, exception:, errmessage:No response body., params:params, bucket=a-bucket, object=files/index_files/455409384957061896/0/455409384953479403/455409384957061895/index_null_offset

this seems to be obviously a minio/S3 setup issue.

if this is your first time to setup the cluster, you need to check your config of object storage access

[danielelongo]

Thanks for the answer, but why does it happen after some time?

I mean it works fine then suddenly i get this error.

Is there something i need to setup with minio for data persistency?

[danielelongo]

milvus.log

I've added the first 50k rows of the logs.

This isn't the first time it happend on different machines so maybe there is some issue with my configuration, if it is ok i can send you the docker-compose file

[xiaofan-luan]

#38832

you might hit this issue.

you can drop the index of 455409384957061896 collection and recreated. or upgrade to 2.5.4

[xiaofan-luan]

My recommendation is upgrading to latest 2.5.4 the problem should be solved.

[danielelongo]

Ok thanks you I'll do it.
I'll close the issue.