diff --git a/quickstarts/Caching.ipynb b/quickstarts/Caching.ipynb new file mode 100644 index 000000000..921a260d2 --- /dev/null +++ b/quickstarts/Caching.ipynb @@ -0,0 +1,514 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "navG4j6eqc_o" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "0lpR41ozqBFp" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "k9g4WhqtqiV-" + }, + "source": [ + "# Gemini API: Context Caching Quickstart\n", + "\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WCC-LFNtqogI" + }, + "source": [ + "This notebook introduces context caching with the Gemini API and provides examples of interacting with the Apollo 11 transcript using the Python SDK. For a more comprehensive look, check out [the caching guide](https://ai.google.dev/gemini-api/docs/caching?lang=python)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5yRLvyPhrXSf" + }, + "outputs": [], + "source": [ + "!pip install -qU 'google-generativeai>=0.7.0'" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4xdJSvLerazn" + }, + "outputs": [], + "source": [ + "import google.generativeai as genai\n", + "from google.generativeai import caching" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xrJYxEpOrc3d" + }, + "source": [ + "## Configure your API key\n", + "\n", + "To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "EgO56yWoriI0" + }, + "outputs": [], + "source": [ + "from google.colab import userdata\n", + "\n", + "genai.configure(api_key=userdata.get(\"GOOGLE_API_KEY\"))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2T-ZorTqrsoz" + }, + "source": [ + "## Upload a file\n", + "\n", + "A common pattern with the Gemini API is to ask a number of questions of the same document. Context caching is designed to assist with this case, and can be more efficient by avoiding the need to pass the same tokens through the model for each new request.\n", + "\n", + "Start by uploading a large file with the [File API](https://github.com/google-gemini/cookbook/blob/main/quickstarts/File_API.ipynb)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Sa-r2s_ltBXy" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "INTRODUCTION\n", + "\n", + "This is the transcription of the Technical Air-to-Ground Voice Transmission (GOSS NET 1) from the Apollo 11 mission.\n", + "\n", + "Communicators in the text may be identified according to the following list.\n", + "\n", + "Spacecraft:\n", + "CDR\tCommander\tNeil A. Armstrong\n", + "CMP\tCommand module pilot \tMichael Collins\n", + "LMP\tLunar module pilot\tEdwin E. ALdrin, Jr.\n" + ] + } + ], + "source": [ + "!wget -q https://storage.googleapis.com/generativeai-downloads/data/a11.txt\n", + "!head a11.txt" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b3bU9AvcvZ_x" + }, + "outputs": [], + "source": [ + "document = genai.upload_file(path=\"a11.txt\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N4LP7unAvmce" + }, + "source": [ + "## Cache the prompt\n", + "\n", + "Next create a [`CachedContent`](https://ai.google.dev/api/python/google/generativeai/protos/CachedContent) object specifying the prompt you want to use, including the file and other fields you wish to cache. In this example the `system_instruction` has been set, and the document was provided in the prompt." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "55V-QkaWv4tb" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "CachedContent(\n", + " name='cachedContents/7k9fow3y6xx5',\n", + " model='models/gemini-1.5-flash-001',\n", + " display_name='',\n", + " usage_metadata={\n", + " 'total_token_count': 323394,\n", + " },\n", + " create_time=2024-06-18 06:03:36.541562+00:00,\n", + " update_time=2024-06-18 06:03:36.541562+00:00,\n", + " expire_time=2024-06-18 07:03:35.195945+00:00\n", + ")" + ] + }, + "execution_count": 45, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Note that caching requires a frozen model, e.g. one with a `-001` version suffix.\n", + "model_name = \"gemini-1.5-flash-001\"\n", + "\n", + "apollo_cache = caching.CachedContent.create(\n", + " model=model_name,\n", + " system_instruction=\"You are a NASA transcript analyst. You understand spaceflight and can provide important context to transcripts.\",\n", + " contents=[document],\n", + ")\n", + "\n", + "apollo_cache" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9j5yzay5xyPC" + }, + "source": [ + "## Manage the cache expiry\n", + "\n", + "Once you have a `CachedContent` object, you can update the expiry time to keep it alive while you need it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cUJT2ESUyTGb" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "CachedContent(\n", + " name='cachedContents/7k9fow3y6xx5',\n", + " model='models/gemini-1.5-flash-001',\n", + " display_name='',\n", + " usage_metadata={\n", + " 'total_token_count': 323394,\n", + " },\n", + " create_time=2024-06-18 06:03:36.541562+00:00,\n", + " update_time=2024-06-18 06:03:38.612328+00:00,\n", + " expire_time=2024-06-18 08:03:38.593924+00:00\n", + ")" + ] + }, + "execution_count": 46, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import datetime\n", + "\n", + "apollo_cache.update(ttl=datetime.timedelta(hours=2))\n", + "apollo_cache" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t_PWabuayrf-" + }, + "source": [ + "## Use the cache for generation\n", + "\n", + "As the `CachedContent` object refers to a specific model and parameters, you must create a [`GenerativeModel`](https://ai.google.dev/api/python/google/generativeai/GenerativeModel) using [`from_cached_content`](https://ai.google.dev/api/python/google/generativeai/GenerativeModel#from_cached_content). Then, generate content as you would with a directly instantiated model object." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "EG8VNpuIzGwT" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The transcript is full of technical jargon, but there's a moment of levity when Buzz Aldrin, the Lunar Module Pilot, responds to a request for a crew status report. \n", + "\n", + "**02 00 20 31 CMP**\n", + "If we're late in answering you, it's because we're munching sandwiches.\n", + "\n", + "**02 00 20 36 CC**\n", + "Roger. I wish I could do the same here.\n", + "\n", + "**02 00 20 40 CMP**\n", + "No. Don't leave the console!\n", + "\n", + "**02 00 20 42 CC**\n", + "Don't worry. I won't.\n", + "\n", + "**02 00 20 47 CMP**\n", + "FLIGHT doesn't like it.\n", + "\n", + "**02 00 20 54 CMP**\n", + "How is FLIGHT today?\n", + "\n", + "**02 00 20 58 CC**\n", + "Oh, he's doing quite well. \n", + "\n", + "This humorous exchange highlights the fact that even during a monumental mission like Apollo 11, crew members and ground control find time to lighten the mood and share a laugh. It's a reminder that even when facing incredible pressure, the human spirit still finds ways to connect and inject some humor. \n", + "\n" + ] + } + ], + "source": [ + "apollo_model = genai.GenerativeModel.from_cached_content(cached_content=apollo_cache)\n", + "\n", + "response = apollo_model.generate_content(\"Find a lighthearted moment from this mission\")\n", + "print(response.text)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SzixLQhC3AO2" + }, + "source": [ + "You can inspect token usage through `usage_metadata`. Note that the cached prompt tokens are included in `prompt_token_count`, but excluded from the `total_token_count`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "MLFd8DFZ29lC" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "prompt_token_count: 323403\n", + "candidates_token_count: 283\n", + "total_token_count: 292\n", + "cached_content_token_count: 323394" + ] + }, + "execution_count": 29, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response.usage_metadata" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "o-24t14t302N" + }, + "source": [ + "You can ask new questions of the model, and the cache is reused." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pZngmGj13k9O" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The most important part of this transcript is when Neil Armstrong takes his \"one small step\" on the moon. Here is that quote: \n", + "\n", + "\"THAT'S ONE SMALL STEP FOR (A) MAN, ONE GIANT LEAP FOR MANKIND.\"\n", + "\n", + "This iconic quote is spoken at 04:13:24 in the transcript and marks the moment when humankind first set foot on the moon.\n", + "\n" + ] + } + ], + "source": [ + "chat = apollo_model.start_chat()\n", + "response = chat.send_message(\"Give me a quote from the most important part.\")\n", + "print(response.text)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "GhGTutW65u7h" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "After Neil Armstrong's iconic statement, \"That's one small step for (a) man, one giant leap for mankind,\" he goes on to describe the lunar surface in more detail:\n", + "\n", + "* **The surface is fine and powdery.** Armstrong describes the moon's surface as \"fine and powdery\" and says he can pick it up loosely with his toe. He also notes that it adheres in fine layers to his boots, similar to powdered charcoal.\n", + "* **Footprints are clearly visible.** He mentions seeing the footprints of his boots and the tread marks in the sand-like particles. \n", + "* **The surface is easy to move around on.** Armstrong says there's no difficulty in moving around, and it's even easier than the simulations on Earth. \n", + "* **The descent engine did not leave a crater.** He mentions that the engine has about 1 foot of clearance from the ground, and there's no significant crater from the landing.\n", + "\n", + "He also briefly mentions seeing rays emanating from the descent engine, suggesting that the engine exhaust did affect the surface in a minor way.\n", + "\n", + "This initial description of the moon's surface is crucial, as it gives the first-hand account of what it's like to walk on another celestial body. \n", + "\n" + ] + } + ], + "source": [ + "response = chat.send_message(\"What was recounted after that?\")\n", + "print(response.text)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "SB5Ywx2D6cOn" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "prompt_token_count: 323497\n", + "candidates_token_count: 257\n", + "total_token_count: 360\n", + "cached_content_token_count: 323394" + ] + }, + "execution_count": 38, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "response.usage_metadata" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LlU43ByS7Kt2" + }, + "source": [ + "## Counting tokens\n", + "\n", + "The `GenerativeModel` object can be used to count the tokens of a request in the same manner as a direct, uncached, model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "1zzHVCAK7ahI" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "total_tokens: 9" + ] + }, + "execution_count": 41, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "apollo_model.count_tokens(\"How many people are involved in this transcript?\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kfeAxehx79ng" + }, + "source": [ + "## Delete the cache\n", + "\n", + "A cache object will be deleted automatically when it expires. You can also explicitly delete a cache object." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HdP83dj08Nb1" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "cachedContents/7k9fow3y6xx5\n" + ] + } + ], + "source": [ + "print(apollo_cache.name)\n", + "apollo_cache.delete()" + ] + } + ], + "metadata": { + "colab": { + "name": "Caching.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}