feat: [Sparse Float Vector] segcore basics and index building #30357

zhengbuqian · 2024-01-30T04:09:34Z

This commit adds sparse float vector support to segcore with the following:

data type enum declarations
Adds corresponding data structures for handling sparse float vectors in various scenarios, including:

FieldData as a bridge between the binlog and the in memory data structures
mmap::Column as the in memory representation of a sparse float vector column of a sealed segment;
ConcurrentVector as the in memory representation of a sparse float vector of a growing segment which supports inserts.

Adds logic in payload reader/writer to serialize/deserialize from/to binlog
Adds the ability to allow the index node to build sparse float vector index
Adds the ability to allow the query node to build growing index for growing segment and temp index for sealed segment without index built

This commit also includes some code cleanness, comment improvement, and some unit tests for sparse vector.

#29419

mergify · 2024-01-30T04:28:20Z

@zhengbuqian E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-01-30T04:59:17Z

@zhengbuqian ut workflow job failed, comment rerun ut can trigger the job again.

mergify · 2024-01-31T07:48:46Z

@zhengbuqian E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-02-18T07:02:30Z

@zhengbuqian E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-02-20T12:29:48Z

@zhengbuqian ut workflow job failed, comment rerun ut can trigger the job again.

codecov · 2024-02-20T13:44:53Z

Codecov Report

Attention: Patch coverage is 48.55072% with 213 lines in your changes are missing coverage. Please review.

Project coverage is 80.94%. Comparing base (7e17f24) to head (1abc16c).
Report is 18 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #30357      +/-   ##
==========================================
- Coverage   80.99%   80.94%   -0.05%     
==========================================
  Files         977      965      -12     
  Lines      141490   142009     +519     
==========================================
+ Hits       114594   114948     +354     
- Misses      23045    23209     +164     
- Partials     3851     3852       +1

Files	Coverage Δ
internal/core/src/common/FieldData.cpp	`72.64% <100.00%> (+2.53%)`	⬆️
internal/core/src/common/FieldData.h	`100.00% <100.00%> (ø)`
internal/core/src/common/Schema.cpp	`74.28% <100.00%> (+5.71%)`	⬆️
internal/core/src/common/Schema.h	`94.11% <ø> (-0.17%)`	⬇️
internal/core/src/common/Span.h	`100.00% <ø> (ø)`
internal/core/src/index/IndexFactory.cpp	`77.18% <100.00%> (+3.43%)`	⬆️
internal/core/src/indexbuilder/IndexFactory.h	`43.47% <ø> (-2.36%)`	⬇️
internal/core/src/segcore/SegmentGrowing.h	`100.00% <ø> (ø)`
internal/core/src/segcore/SegmentGrowingImpl.cpp	`73.03% <100.00%> (+0.28%)`	⬆️
internal/core/src/segcore/SegmentGrowingImpl.h	`81.03% <ø> (+0.67%)`	⬆️
... and 24 more

... and 222 files with indirect coverage changes

mergify · 2024-02-28T07:21:46Z

@zhengbuqian E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-02-29T03:36:27Z

@zhengbuqian E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-03-08T04:57:15Z

@zhengbuqian E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

zhengbuqian · 2024-03-08T05:18:29Z

/run-cpu-e2e

xiaofan-luan · 2024-03-09T20:18:03Z

internal/core/src/index/Utils.cpp

@@ -68,6 +68,10 @@ unsupported_index_combinations() {
    static std::vector<std::tuple<IndexType, MetricType>> ret{
        std::make_tuple(knowhere::IndexEnum::INDEX_FAISS_BIN_IVFFLAT,
                        knowhere::metric::L2),
+        std::make_tuple(knowhere::IndexEnum::INDEX_SPARSE_INVERTED_INDEX,


Don't quite understand.
Do we support cosine metrics?

No, IP is the only supported metric. Updated.

and bin ivf index don't support IP, cosine as well.

I think this check is meaningless.... what about INDEX_SPARSE_INVERTED_INDEX plus hamming

agreed.. this check is never completed since the beginning and we always rely on checks elsewhere to guarantee the precondition. Since this check originally has only 1 pair, I'll see if I can remove it in a later PR.

sparse index has no hamming distance support.

xiaofan-luan · 2024-03-09T20:21:56Z

internal/core/src/index/VectorMemIndex.cpp

+        auto buf = std::shared_ptr<uint8_t[]>(new uint8_t[total_size]);
+        int64_t offset = 0;
+        for (auto data : field_datas) {
+            std::memcpy(buf.get() + offset, data->Data(), data->Size());


We can atually avoid this copy by introducing a wrapper?
This may save tons of memories.
Leave it for optimization

that is very true. Currently we copy for both dense and sparse. I tried updating this but it requires some major refactoring to allow the field data to give up ownership of its underlying data. such a change will affect lots of code(most of them should be beneficial) and is way beyond the scope of this PR so I just keep it as is.

I am adding a TODO.

xiaofan-luan · 2024-03-09T20:23:03Z

internal/core/src/mmap/Utils.h

+                // TODO(SPARSE): this is for mmap to write data to disk so that
+                // the file can be mmaped into memory.
+                throw std::runtime_error(
+                    "WriteFieldData for VECTOR_SPARSE_FLOAT not implemented");


this means mmap is not supported for mmap yet?

yes, no mmap support for sparse yet.

xiaofan-luan · 2024-03-09T20:25:12Z

internal/core/src/segcore/segment_c.cpp

-        auto suc = insert_data->ParseFromArray(data_info, data_info_len);
+        auto insert_record_proto =
+            std::make_unique<milvus::InsertRecordProto>();
+        auto suc =


might introduce too much copy here.
but so far is good

This is only a naming change from using InsertData = proto::segcore::InsertRecord; to using InsertRecordProto = proto::segcore::InsertRecord;. the name InsertData is overloaded for several purposes and ambiguous.

xiaofan-luan · 2024-03-09T20:25:52Z

/ltgm
/approve

sre-ci-robot · 2024-03-09T20:26:04Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xiaofan-luan, zhengbuqian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~internal/core/OWNERS~~ [xiaofan-luan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xiaofan-luan · 2024-03-09T20:26:21Z

Except for not supporting mmap for now, the rest of the pr looks good to me.
Let's ship it and avoid unnecessary blokcing

This commit adds sparse float vector support to segcore with the following: 1. data type enum declarations 2. Adds corresponding data structures for handling sparse float vectors in various scenarios, including: * FieldData as a bridge between the binlog and the in memory data structures * mmap::Column as the in memory representation of a sparse float vector column of a sealed segment; * ConcurrentVector as the in memory representation of a sparse float vector of a growing segment which supports inserts. 3. Adds logic in payload reader/writer to serialize/deserialize from/to binlog 4. Adds the ability to allow the index node to build sparse float vector index 5. Adds the ability to allow the query node to build growing index for growing segment and temp index for sealed segment without index built This commit also includes some code cleanness, comment improvement, and some unit tests for sparse vector. Signed-off-by: Buqian Zheng <[email protected]>

czs007 · 2024-03-11T06:43:24Z

/lgtm

sre-ci-robot requested review from czs007 and godchen0212 January 30, 2024 04:09

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. area/compilation labels Jan 30, 2024

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Jan 30, 2024

buqian-zilliz force-pushed the sparse-v7-pr1-official branch from e706b12 to 049e907 Compare February 4, 2024 12:02

sre-ci-robot added the area/dependency Pull requests that update a dependency file label Feb 4, 2024

buqian-zilliz force-pushed the sparse-v7-pr1-official branch from 049e907 to 10149dd Compare February 18, 2024 06:50

zhengbuqian mentioned this pull request Feb 18, 2024

feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id #30629

Merged

zhengbuqian mentioned this pull request Feb 18, 2024

[Feature]: Add Sparse Float Vector support #29419

Closed

1 task

buqian-zilliz force-pushed the sparse-v7-pr1-official branch from 10149dd to c5101b4 Compare February 18, 2024 08:39

zhengbuqian mentioned this pull request Feb 21, 2024

feat: [Do Not Mrege] Add Sparse Float Vector support to milvus #29421

Closed

buqian-zilliz force-pushed the sparse-v7-pr1-official branch from c5101b4 to f9bf5e4 Compare February 28, 2024 07:10

buqian-zilliz force-pushed the sparse-v7-pr1-official branch from f9bf5e4 to b64864d Compare February 29, 2024 03:05

buqian-zilliz force-pushed the sparse-v7-pr1-official branch from b64864d to 43c15e0 Compare February 29, 2024 05:19

zhengbuqian force-pushed the sparse-v7-pr1-official branch 2 times, most recently from 53ec559 to 2c23d04 Compare March 8, 2024 04:38

xiaofan-luan reviewed Mar 9, 2024

View reviewed changes

sre-ci-robot added the approved label Mar 9, 2024

zhengbuqian force-pushed the sparse-v7-pr1-official branch from 2c23d04 to 1abc16c Compare March 10, 2024 06:32

czs007 added ci-passed manual-pass manually set pass before ci-passed labeled labels Mar 11, 2024

sre-ci-robot assigned czs007 Mar 11, 2024

sre-ci-robot added the lgtm label Mar 11, 2024

sre-ci-robot merged commit 070dfc7 into milvus-io:master Mar 11, 2024
13 of 14 checks passed

zhengbuqian deleted the sparse-v7-pr1-official branch March 11, 2024 06:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: [Sparse Float Vector] segcore basics and index building #30357

feat: [Sparse Float Vector] segcore basics and index building #30357

zhengbuqian commented Jan 30, 2024

mergify bot commented Jan 30, 2024

mergify bot commented Jan 30, 2024

mergify bot commented Jan 31, 2024

mergify bot commented Feb 18, 2024

mergify bot commented Feb 20, 2024

codecov bot commented Feb 20, 2024 •

edited

Loading

mergify bot commented Feb 28, 2024

mergify bot commented Feb 29, 2024

mergify bot commented Mar 8, 2024

zhengbuqian commented Mar 8, 2024

xiaofan-luan Mar 9, 2024

zhengbuqian Mar 10, 2024

xiaofan-luan Mar 10, 2024

zhengbuqian Mar 11, 2024

xiaofan-luan Mar 9, 2024

zhengbuqian Mar 10, 2024

xiaofan-luan Mar 9, 2024

zhengbuqian Mar 10, 2024

xiaofan-luan Mar 9, 2024

zhengbuqian Mar 10, 2024

xiaofan-luan commented Mar 9, 2024

sre-ci-robot commented Mar 9, 2024

xiaofan-luan commented Mar 9, 2024

czs007 commented Mar 11, 2024

feat: [Sparse Float Vector] segcore basics and index building #30357

feat: [Sparse Float Vector] segcore basics and index building #30357

Conversation

zhengbuqian commented Jan 30, 2024

mergify bot commented Jan 30, 2024

mergify bot commented Jan 30, 2024

mergify bot commented Jan 31, 2024

mergify bot commented Feb 18, 2024

mergify bot commented Feb 20, 2024

codecov bot commented Feb 20, 2024 • edited Loading

Codecov Report

mergify bot commented Feb 28, 2024

mergify bot commented Feb 29, 2024

mergify bot commented Mar 8, 2024

zhengbuqian commented Mar 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaofan-luan commented Mar 9, 2024

sre-ci-robot commented Mar 9, 2024

xiaofan-luan commented Mar 9, 2024

czs007 commented Mar 11, 2024

codecov bot commented Feb 20, 2024 •

edited

Loading