diff --git a/examples/qdrant/Qdrant_similarity_search.ipynb b/examples/qdrant/Qdrant_similarity_search.ipynb new file mode 100644 index 000000000..a9f8e8a75 --- /dev/null +++ b/examples/qdrant/Qdrant_similarity_search.ipynb @@ -0,0 +1,516 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "lTx8eQlc3cP-" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "4HZoi8yf4GEU" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "M9I7LG483nXB" + }, + "source": [ + "# Gemini API: Similarity Search using Qdrant" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "awKO767lQIWh" + }, + "source": [ + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b1xoF_bU4NCP" + }, + "source": [ + "## Overview" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CWedABji6bXJ" + }, + "source": [ + "The [Gemini API](https://ai.google.dev/models/gemini) provides access to a family of generative AI models for generating content and solving problems. These models are designed and trained to handle both text and images as input.\n", + "\n", + "[Qdrant](https://qdrant.tech/) is a vector similarity search engine that offers an easy-to-use API for managing, storing, and searching vectors, with an additional payload. It is a production-ready service.\n", + "\n", + "In this notebook, you'll learn how to perform a similarity search on data from a website with the help of Gemini API and Qdrant." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dIAarGkG8VwC" + }, + "source": [ + "## Setup\n", + "\n", + "First, you must install the packages and set the necessary environment variables.\n", + "\n", + "### Installation" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "70wYOKUC8q1m" + }, + "source": [ + "Install google's python client SDK for the Gemini API, `google-generativeai`. Next, install Qdrant's Python client SDK, `qdrant-client`." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "mnQbBnA1GKha" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m294.6/294.6 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m230.1/230.1 kB\u001b[0m \u001b[31m14.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m17.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m27.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.5/62.5 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.9/15.9 MB\u001b[0m \u001b[31m61.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m87.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m77.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m92.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m86.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m109.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m10.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.5/57.5 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", + "tensorflow-metadata 1.15.0 requires protobuf<4.21,>=3.20.3; python_version < \"3.11\", but you have protobuf 4.25.1 which is incompatible.\n", + "transformers 4.41.2 requires huggingface-hub<1.0,>=0.23.0, but you have huggingface-hub 0.20.3 which is incompatible.\n", + "transformers 4.41.2 requires tokenizers<0.20,>=0.19, but you have tokenizers 0.15.2 which is incompatible.\u001b[0m\u001b[31m\n", + "\u001b[0m" + ] + } + ], + "source": [ + "!pip install -q google-generativeai\n", + "!pip install -q protobuf==4.25.1 qdrant-client[fastembed]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eomJzCa6lb90" + }, + "source": [ + "## Configure your API key\n", + "\n", + "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "v-JZzORUpVR2" + }, + "outputs": [], + "source": [ + "from google.colab import userdata\n", + "GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n", + "\n", + "genai.configure(api_key=GOOGLE_API_KEY)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-go1BAF-_GtV" + }, + "source": [ + "## Basic steps\n", + "\n", + "Semantic search is the process using which search engines interpret and match keywords to a user's intent in organic search results. It goes beyond surface-level keyword matching. It uses the meaning of words, phrases, and context using advanced algorithms resulting in more relevant and user-friendly search experiences.\n", + "\n", + "Semantic searches rely on vector embeddings which can best match the user query to the most similar result.\n", + "\n", + "In this tutorial, you'll implement the three main components of semantic search:\n", + "\n", + "1. Create an index\n", + "\n", + " Create and store the index for the data in the Qdrant vector store. You will use a Gemini API embedding model to create embedding vectors that can be stored in the Qdrant vector store.\n", + "\n", + "2. Query the index\n", + "\n", + " Query the index using a query string to return the top `n` neighbors of the query.\n", + "\n", + "You'll learn more about these stages in the upcoming sections while implementing the application." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0egnCR92JKsj" + }, + "source": [ + "## Import the required libraries" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "LfJN5QosJQqD" + }, + "outputs": [], + "source": [ + "import google.generativeai as genai\n", + "\n", + "from bs4 import BeautifulSoup\n", + "from qdrant_client import models, QdrantClient\n", + "from urllib.request import urlopen" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lL7J7BtyJsNQ" + }, + "source": [ + "## 1. Create an index\n", + "\n", + "In this stage, you will perform the following steps:\n", + "\n", + "1. Read and parse the website data using Python's BeautifulSoup library.\n", + "\n", + "2. Create embeddings of the website data.\n", + "\n", + "3. Store the embeddings in Qdrant's vector database.\n", + " \n", + " Qdrant is a vector similarity search engine. Along with a convenient API to store, search, and manage points(i.e. vectors), it also provides an option to add an additional payload. The payloads are essentially extra bits of data that you can utilize to refine your search and obtain relevant information that you can then share with your users." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kFlGmkKbRebP" + }, + "source": [ + "### Read and parse the website data\n", + "\n", + "To read the website data as text, you will use the `BeautifulSoup` library from Python." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "oMs-ux1gtxOa" + }, + "outputs": [], + "source": [ + "url = \"https://blog.google/outreach-initiatives/sustainability/\"\\\n", + " \"report-ai-sustainability-google-cop28/\"\n", + "html = urlopen(url).read()\n", + "soup = BeautifulSoup(html, features=\"html.parser\")\n", + "\n", + "# Remove all script and style elements\n", + "for script in soup([\"script\", \"style\"]):\n", + " script.extract() # Self-destruct\n", + "\n", + "# Get the text\n", + "text_content = soup.get_text()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "toC03rqUSfom" + }, + "source": [ + "If you only want to select a specific portion of the website data to add context to the prompt, you can use regex, text slicing, or text splitting.\n", + "\n", + "In this example, you'll use Python's `split()` function to extract the required portion of the text." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "cHJq059duxj7" + }, + "outputs": [], + "source": [ + "# The text content between the substrings \"Later this month at COP28\" to\n", + "# \"POSTED IN:\" is relevant for this tutorial. You can use Python's `split()`\n", + "# to select the required content.\n", + "text_content_1 = text_content.split(\"Later this month at COP28\",1)[1]\n", + "final_text = text_content_1.split(\"POSTED IN:\",1)[0]\n", + "\n", + "texts = final_text.split(\".\")\n", + "\n", + "documents = []\n", + "\n", + "# Convert text into a chunk of 3 sentences.\n", + "for i in range(0, len(texts), 3):\n", + " documents.append({\"content\": \" \".join(texts[i:i+3])})" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-CVPdm0h6aTd" + }, + "source": [ + "### Initialize the embedding model\n", + "\n", + "To create the embeddings from the website data, you'll use the **embedding-001** model, which supports creating embeddings from text.\n", + "\n", + "To use the embedding model, you have to use the `embed_content` function from the `google-generativeai` package. To learn more about the embedding model, read the [model documentation](https://ai.google.dev/gemini-api/docs/models/gemini#embedding).\n", + "\n", + "One of the arguments passed to the embedding function is `task_type`. Speciefying the `task_type` parameter ensures the model produces appropriate embeddingsfor the expected task and inputs. It is a string that can take on one of the following values:\n", + "\n", + "| task_type\t | Description |\n", + "|---|---|\n", + "| `RETRIEVAL_QUERY` | Specifies the given text is a query in a search or retrieval setting. |\n", + "| `RETRIEVAL_DOCUMENT` | Specifies the given text is a document in a search or retrieval setting. | \n", + "| `SEMANTIC_SIMILARITY` | Specifies the given text will be used for Semantic Textual Similarity (STS). | \n", + "| `CLASSIFICATION` | Specifies that the embeddings will be used for classification. |\n", + "| `CLUSTERING` | Specifies that the embeddings will be used for clustering. |" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "EeSW5NOBTS0a" + }, + "outputs": [], + "source": [ + "# Default embedding model\n", + "embedding_model = \"models/embedding-001\"\n", + "\n", + "# Function to convert text to embeddings\n", + "def make_embed_text_fn(text, model=embedding_model,\n", + " task_type=\"retrieval_document\"):\n", + " embedding = genai.embed_content(model=model,\n", + " content=text,\n", + " task_type=task_type)\n", + " return embedding['embedding']" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9ajByZTxVXal" + }, + "source": [ + "### Store the data using Qdrant\n", + "\n", + " Next, you'll store the embeddings of the website data in Qdrant's vector store.\n", + "\n", + " First, you have to initiate a Qdrant client by creating an instance of `QdrantClient`. In this tutorial, you will store the embeddings in memory. To create an in-memory Qdrant client specify `:memory:` for the `location` argument of the `QdrantClient` class initializer. You can read more about the different types of storage in Qdrant in the [storage reference guide](https://qdrant.tech/documentation/concepts/storage/).\n", + "\n", + "After initializing the client, you have to create a Qdrant collection using the `recreate_collection` function of `QdrantClient`. You can specify your vector configuration inside the `recreate_collection` function. Pass an instance of `VectorParams` with the `size` set to `768` to match the embedding model and `distance` set to cosine.\n", + "\n", + "**Note**: Since you will run the script several times during your experiments, `recreate_collection` is appropriate for this tutorial. `recreate_collection` will first try to remove an existing collection with the same name." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "pnURtmtZTImC" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + ":5: DeprecationWarning: `recreate_collection` method is deprecated and will be removed in the future. Use `collection_exists` to check collection existence and `create_collection` instead.\n", + " qdrant.recreate_collection(\n" + ] + }, + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Initialize Qdrant client.\n", + "qdrant = QdrantClient(\":memory:\")\n", + "\n", + "# Create a collection named \"GeminiCollection\".\n", + "qdrant.recreate_collection(\n", + " collection_name=\"GeminiCollection\",\n", + " vectors_config=models.VectorParams(\n", + " size=768, # Vector size of `embedding-001`\n", + " distance=models.Distance.COSINE,\n", + " ),\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YQu7FIhLeC0O" + }, + "source": [ + "You will now insert the `documents` you parsed from the website data into the Qdrant collection you created earlier and index them using the `upsert` function of `QdrantClient`.\n", + "\n", + "The `upsert` function takes the data to be stored and indexed as an array of `PointsStruct`s.\n", + "\n", + "Points are the main entity in Qdrant operations. A point is a record consisting of a vector and an optional payload. You can perform a similarity search among the points in one collection. Read more about points in [Qdrant's points documentation](https://qdrant.tech/documentation/concepts/points/).\n", + "\n", + "You'll create an array of points by enumerating over the documents you prepared earlier from the website data." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "uOqivudxSyR9" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "UpdateResult(operation_id=0, status=)" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Qdrant uses batch loading of points to optimize performance.\n", + "# You can create a batch in two ways - record-oriented and column-oriented.\n", + "# Here you are using the record-oriented approach.\n", + "\n", + "qdrant.upsert(\n", + " collection_name=\"GeminiCollection\",\n", + " points=[\n", + " # Use PointStruct function to intialize the point.\n", + " models.PointStruct(\n", + " # Use `make_embed_text_fn` to convert text to embeddings.\n", + " # Pass the same data as payload for a refined search.\n", + " id=idx, vector=make_embed_text_fn(doc[\"content\"]), payload = doc\n", + " )\n", + " for idx, doc in enumerate(documents)\n", + " ]\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JdVrKZZ0cTkV" + }, + "source": [ + "## 2. Query the index\n", + "\n", + "You'll now query the Qdrant index you created earlier with a question related to the data contained in the website documents.\n", + "To query the index, you have to mention the collection name and the query vector. The query vector should be first converted to an embedding vector using the Gemini API embedding model you leveraged to create embedding vectors for the website data. Use the `make_embed_text_fn` you defined earlier for creating an embedding vector from your query. Since you are embedding a query string that is being used to search `retrieval_document` embeddings, the `task_type` must be set to `retrieval_query`." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "6LQVKNfMTyOx" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'content': ' Already, it is starting to address climate challenges in three key areas: providing people and organizations with better information to make more sustainable choices, delivering improved predictions to help adapt to climate change, and finding recommendations to optimize climate action for high-impact applications Here’s a look at how, at Google, we’ve used AI to address climate challenges:Providing helpful information: People are looking for information to reduce their environmental footprint Fuel-efficient routing in Google Maps uses AI to suggest routes that have fewer hills, less traffic, and constant speeds with the same or similar ETA'} score: 0.7711945535904017\n", + "{'content': ' Policymakers, in particular, have a central role to play both in harnessing the potential of AI for climate action and in ensuring its sustainable and equitable use Policymakers can make a difference in accelerating three outcomes:Enabling AI for climate progress by encouraging data sharing, ensuring affordable technology access, building awareness, and supporting the creation and expansion of AI and climate-related upskilling programs for corporations Accelerating the deployment of AI for climate by defining public and private sector priorities, delivering on public sector use cases, and encouraging private sector action'} score: 0.7458781382056137\n", + "{'content': '\\n\\n\\n\\n\\nManaging the environmental impact of AIWhile scaling these applications of AI and finding new ways to use it to accelerate climate action is crucial, we need to build AI responsibly and manage the environmental impact associated with it As AI is at an inflection point, predicting the future growth of energy use and emissions from AI compute in our data centers is challenging Historically, data center energy consumption has grown much more slowly than demand for computing power'} score: 0.7405380973240167\n" + ] + } + ], + "source": [ + "hits = qdrant.search(\n", + " collection_name=\"GeminiCollection\",\n", + " query_vector=make_embed_text_fn(\"How can AI address climate challenges?\",\n", + " task_type=\"retrieval_query\"),\n", + " limit=3,\n", + ")\n", + "for hit in hits:\n", + " print(hit.payload, \"score:\", hit.score)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Tt1wSSMIxsf2" + }, + "source": [ + "##Conclusion\n", + "\n", + "That's it. You have successfully performed a similarity search using Qdrant with the help of a Gemini API embedding model." + ] + } + ], + "metadata": { + "colab": { + "name": "Qdrant_similarity_search.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}