Simple usage tutorial (with Docker)

[soulteary]

This project is very useful, but the current way of using it is a bit complicated. So, I made a simple docker image for quick use. (base on v2.1.0)

The use of this docker image is relatively simple, just two steps.

Let's start with a simple command to start a local Milvus docker container and name it "milvus":

docker run --rm -it --name=milvus soulteary/milvus:embed-2.1.0

After the container starts, about 1-2 seconds later, we will see a log output, telling us that the service is started, and we can start the "second step".

---Milvus Proxy successfully initialized and ready to serve!---

For example, we can open a new "command line terminal window" and let the container we just named "milvus" run the "Hello World" similar to the "MySQL CRUD" operation in the Milvus SDK:

docker exec -it milvus python hello-world.py

We will get results similar to the following:

=== start connecting to Milvus     ===

Does collection hello_milvus exist in Milvus: False

=== Create collection `hello_milvus` ===


=== Start inserting entities       ===

Number of entities in Milvus: 3000

=== Start Creating index IVF_FLAT  ===


=== Start loading                  ===


=== Start searching based on vector similarity ===

hit: (distance: 0.0, id: 2998), random field: -13.0
hit: (distance: 0.16608965396881104, id: 1059), random field: -12.0
hit: (distance: 0.17376846075057983, id: 976), random field: -18.0
hit: (distance: 0.0, id: 2999), random field: -15.0
hit: (distance: 0.09368982911109924, id: 760), random field: -15.0
hit: (distance: 0.1466047316789627, id: 263), random field: -12.0
search latency = 0.2489s

=== Start querying with `random > -14` ===

query result:
-{'pk': 16, 'random': -11.0, 'embeddings': [0.73126, 0.492735, 0.017334, 0.219567, 0.872692, 0.050455, 0.072988, 0.322331]}
search latency = 0.2517s

=== Start hybrid searching with `random > -12` ===

hit: (distance: 0.2523949146270752, id: 2281), random field: -11.0
hit: (distance: 0.34272605180740356, id: 2146), random field: -11.0
hit: (distance: 0.34762364625930786, id: 467), random field: -11.0
hit: (distance: 0.249067485332489, id: 819), random field: -11.0
hit: (distance: 0.3513686954975128, id: 2786), random field: -11.0
hit: (distance: 0.4150209426879883, id: 1486), random field: -11.0
search latency = 0.3481s

=== Start deleting with expr `pk in [0, 1]` ===

query before delete by expr=`pk in [0, 1]` -> result: 
-{'pk': 0, 'random': -17.0, 'embeddings': [0.336848, 0.42659, 0.09273, 0.587923, 0.559363, 0.854384, 0.348711, 0.16934]}
-{'pk': 1, 'random': -20.0, 'embeddings': [0.273309, 0.255625, 0.597282, 0.181107, 0.198166, 0.758371, 0.186847, 0.120927]}

query after delete by expr=`pk in [0, 1]` -> result: []


=== Drop collection `hello_milvus` ===

Of course, we can also start an interactive Python terminal directly and do our own thing in it:

docker exec -it milvus python

---

The data in the docker image will be saved in the /etc/milvus directory. If you need to persist the data during local debugging or testing, you can map the data directory to the local when executing the container:

docker run --rm -it --name=milvus \
           -v `pwd`/data:/etc/milvus/data \
           -v `pwd`/logs:/etc/milvus/logs \
           -v `pwd`/etcd:/etc/milvus/etcd/data \
           -v `pwd`/rdbs:/etc/milvus/rdb/data \
           -v `pwd`/data:/etc/milvus/data \
           soulteary/milvus:embed-2.1.0

If you need to test SDKs in various languages, you can also expose some network ports of the docker locally. As for which ports are available, you can refer to the configuration file soulteary/portable-docker-app/milvus/config.yml, and again Not much to say.

---

Blog: https://soulteary.com/2022/09/17/the-docker-tool-image-of-the-entry-vector-database-milvus.html (in chinese, maybe need to translate)

[eolivelli]

This work will be very useful for us at LangStream.ai

LangStream/langstream#470

It is not possible to write an integration test easily.

@soulteary could you please create a new docker image with the very latest version (2.3.0) ? I need support for UPSERTS, that is missing in 2.1.0.
Your docker image works pretty well

[matrixji]

This work will be very useful for us at LangStream.ai

LangStream/langstream#470

It is not possible to write an integration test easily.

@soulteary could you please create a new docker image with the very latest version (2.3.0) ? I need support for UPSERTS, that is missing in 2.1.0. Your docker image works pretty well

@eolivelli Previously we needed some preparations for the setup of milvus-embd, now we could simply install/run milvus-lite with pip in a few steps. I could provide a simple Dockerfile for it later. The Milvus lite already supports 2.3.0(Linux & Mac), maybe 2.3.1 days later.