Why does the performance of the existing collection degrade when large data are inserted into a new collection?

[pipi-olo]

Hi.

I installed Milvus on kubernets using helm chart. (v2.5.15)

While inserting data into Collection B using Bulk Insert, the QPS for Collection A drops.
What could be the reason for this?
(I used VectorDBBench to measure the QPS performance.)

I thought the reason is that the query node's growing segments have increased due to subscribe kafka. I created a new resource group and deployed Collection A.
However, even with this setup, when i insert large data into a new collection B, the performance of the existing collection A drops.
Also, I read that bulk insert does not publish data to Kafka.

What is the reason for this, and how can it be resolved?

[yhmo]

Theoretically, bulkinsert doesn't affect search performance in cluster mode. Bulkinsert doesn't generate growing segments, data is read and sealed segment is generated, the new segment is loaded only after it is fully indexed.

Your milvus is a cluster, not standalone?
How many qps dropped during bulkinsert?

[pipi-olo]

milvus is a cluster.

During bulk insert, the QPS for collection A dropped from 150 to 40.
After the indexing for collection B finished, the QPS for A returned to 150.

cf. I used diskann index.
cf. I performed a bulk insert through http

[xiaofan-luan]

How much querynode and indexnode do youn have?
Could it possible that indexnode and querynode actually on the same physical machine?

[pipi-olo]

The query node and index node can be placed on the same physical machine. You think a lack of resource on that machine, right?

Let me check it right away.

[pipi-olo]

Following your advice, I have separated the physical machines for query node and index node.

However,
the QPS for the existing collection A still drops until the indexing job for the new Collection B is completed.

I hava index node with 16 cores, 32GB mem, 2 to 8 replicas (HPA) and query node with 4 cores, 16GB mem, 5 replicas.

The test dataset is openai_large_set with 5 million and 1,546 dimensions.

[yhmo]

You can deploy a monitoring service to observe: https://milvus.io/docs/v2.4.x/monitor.md
Observe these metrics:

[pipi-olo]

I had focused on cpu and memory.
However, the cause is disk i/o.

Thank you!