[Bug]: When building an index for the same data, the CPU usage of the master image is 100%, but that of the 2.4 image is only 60%

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20250106-9e221063-amd64 vs master-20250108-f0dae814-amd64
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Test index cost for sift 10m data on master and 2.4 branch. The cpu usage of master is 100% while the 2.4 is only 60%

1. standalone config

    standalone:
      env:
      - name: GOTRACEBACK
        value: crash
      replicas: 1
      resources:
        limits:
          cpu: "8" 
          memory: 32Gi
        requests:
          cpu: "8" 
          memory: 30Gi

2. index params:

{'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 30, 'efConstruction': 360}}

3. index cost and cpu usage

• 2.4-20250106-9e221063-amd64: cost 2220.5526 s, cpu usage 60%
  metrics of stats-24-op-55-4263

pyroscope of stats-24-op-55-4263

• master-20250108-f0dae814-amd64: cost 1510.0654 s, cpu usage 100%
  metrics of stats-master-op-62-8296

pyroscope of stats-master-op-62-8296

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/8a74abd1-0140-4ce5-af21-f146e2275f1f?nodeId=zong-stats-index-4

Milvus Log

• pods of 2.4:

stats-24-op-55-4263-etcd-0                                        1/1     Running                  0               5h20m   10.104.25.5     4am-node30   <none>           <none>
stats-24-op-55-4263-kafka-0                                       1/1     Running                  1 (5h19m ago)   5h19m   10.104.30.233   4am-node38   <none>           <none>
stats-24-op-55-4263-kafka-1                                       1/1     Running                  1 (5h19m ago)   5h19m   10.104.32.33    4am-node39   <none>           <none>
stats-24-op-55-4263-kafka-2                                       1/1     Running                  0               5h19m   10.104.19.161   4am-node28   <none>           <none>
stats-24-op-55-4263-kafka-zookeeper-0                             1/1     Running                  0               5h19m   10.104.32.31    4am-node39   <none>           <none>
stats-24-op-55-4263-kafka-zookeeper-1                             1/1     Running                  0               5h19m   10.104.17.211   4am-node23   <none>           <none>
stats-24-op-55-4263-kafka-zookeeper-2                             1/1     Running                  0               5h19m   10.104.19.160   4am-node28   <none>           <none>
stats-24-op-55-4263-milvus-standalone-b95fd7954-x65ss             1/1     Running                  0               5h18m   10.104.9.159    4am-node14   <none>           <none>
stats-24-op-55-4263-minio-64c5f5f586-zdqkb                        1/1     Running                  0               5h19m   10.104.25.7     4am-node30   <none>           <none>

• pods of master

stats-master-op-62-8296-etcd-0                                    1/1     Running                  0              5h6m    10.104.25.3     4am-node30   <none>           <none>
stats-master-op-62-8296-kafka-0                                   1/1     Running                  1 (5h6m ago)   5h6m    10.104.19.156   4am-node28   <none>           <none>
stats-master-op-62-8296-kafka-1                                   1/1     Running                  1 (5h6m ago)   5h6m    10.104.25.4     4am-node30   <none>           <none>
stats-master-op-62-8296-kafka-2                                   1/1     Running                  1 (5h6m ago)   5h6m    10.104.30.231   4am-node38   <none>           <none>
stats-master-op-62-8296-kafka-zookeeper-0                         1/1     Running                  0              5h6m    10.104.19.157   4am-node28   <none>           <none>
stats-master-op-62-8296-kafka-zookeeper-1                         1/1     Running                  0              5h6m    10.104.32.28    4am-node39   <none>           <none>
stats-master-op-62-8296-kafka-zookeeper-2                         1/1     Running                  0              5h6m    10.104.20.211   4am-node22   <none>           <none>
stats-master-op-62-8296-milvus-standalone-77c5ddc7d6-gzj5t        1/1     Running                  0              5h5m    10.104.6.4      4am-node13   <none>           <none>
stats-master-op-62-8296-minio-7fdf8fbc77-r4rnz                    1/1     Running                  0              5h6m    10.104.32.27    4am-node39   <none>           <none>
stats-master-op-62-8296-minio-update-prometheus-secret-zzh49      0/1     Completed                0              5h6m    10.104.6.3      4am-node13   <none>           <none>

Anything else?

No response

[yanliang567]

/unassign

[xiaocai2333]

@ThreadDao In standalone mode, the default CPU usage is limited to 75%, so the usage of 2.4 is expected. However, I'm still investigating why the master can utilize 100% of the CPU.

[yanliang567]

@ThreadDao In standalone mode, the default CPU usage is limited to 75%, so the usage of 2.4 is expected. However, I'm still investigating why the master can utilize 100% of the CPU.

is there any config for this cpu usage limitation? I remember it was 50%? @xiaocai2333

[xiaocai2333]

@ThreadDao In standalone mode, the default CPU usage is limited to 75%, so the usage of 2.4 is expected. However, I'm still investigating why the master can utilize 100% of the CPU.

is there any config for this cpu usage limitation? I remember it was 50%? @xiaocai2333

yeah， the index-building pool size is usually set to 50% in standalone mode.
When starting Milvus, both indexNode and querynode set their respective index building pool sizes based on the configuration:

• indexnode uses configuration paramaters related to:
  

  milvus/configs/milvus.yaml

  Line 809 in 461c376

  buildIndexThreadPoolRatio: 0.75

• querynode uses configuration paramaters related to:
  

  milvus/configs/milvus.yaml

  Line 410 in 461c376

  buildParallelRate: 0.5 # the ratio of building interim index parallel matched with cpu num

In standalone mode, the component that starts later will overwrite the corresponding configuration. Typically, the querynode starts last, so the index-building pool size is usually set to 50% in standalone mode.

[cqy123456]

The cpu util of building diskann index in master branch is normal: 50%( = build thread number / total core number).

[cqy123456]

The implementation of hnsw was switched from hnswlib to faiss when the milvus version switch from 2.4 to master. It is possible that the omp setter does not work in faiss_hnsw train and add function ？/assign @alexanderguzhva

[foxspy]

https://github.com/zilliztech/knowhere/blob/main/src/index/hnsw/faiss_hnsw.cc#L113

From the test results, build_pool is effective and can limit the CPU resources used to build the index; the code also shows that it is limited by omp https://github.com/zilliztech/knowhere/blob/main/src/index/hnsw/faiss_hnsw.cc#L113

[foxspy]

[yanliang567]

verified on 2.5-20250120-44649664-amd64 which cpu usage keeps on 50% in standalone mode.