From b8dc35947b30979d64fbd03a5b028fe1de90913e Mon Sep 17 00:00:00 2001 From: kinarr Date: Thu, 10 Oct 2024 05:54:13 +0530 Subject: [PATCH 01/43] FIx small typo in ES Gemma RAG Colab --- Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb b/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb index 93d4572..8f70741 100644 --- a/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb +++ b/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb @@ -1254,7 +1254,7 @@ "\n", "Finally, you'll set up the RAG chain that ties everything together while relying on LangChain's [LCEL paradigm](https://python.langchain.com/v0.1/docs/expression_language/why/).\n", "\n", - "[The prompt](https://smith.langchain.com/hub/rlm/rag-prompt) you're using expects an input that includes two keys: `context` and `question`. The user only provides the question, so uoi need to obtain and format the relevant context using our retriever. Here's how it works:\n", + "[The prompt](https://smith.langchain.com/hub/rlm/rag-prompt) you're using expects an input that includes two keys: `context` and `question`. The user only provides the question, so you need to obtain and format the relevant context using our retriever. Here's how it works:\n", "\n", "* **Retrieve Relevant Documents**: Use `retriever` to get the relevant documents based on the user's question.\n", "\n", From 2917398362bd5467e86dd159d9492dc1a8880308 Mon Sep 17 00:00:00 2001 From: "nidhin.devan" Date: Fri, 11 Oct 2024 01:27:51 +0530 Subject: [PATCH 02/43] LocalGemma in Colab notebook added --- Gemma/Using_Gemma_with_LocalGemma.ipynb | 336 ++++++++++++++++++++++++ 1 file changed, 336 insertions(+) create mode 100644 Gemma/Using_Gemma_with_LocalGemma.ipynb diff --git a/Gemma/Using_Gemma_with_LocalGemma.ipynb b/Gemma/Using_Gemma_with_LocalGemma.ipynb new file mode 100644 index 0000000..d76c27c --- /dev/null +++ b/Gemma/Using_Gemma_with_LocalGemma.ipynb @@ -0,0 +1,336 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "7QCqLJkVc2Fh" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "bkysk5wedUe5" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "F6DdwaTLdj6f" + }, + "source": [ + "# Getting Started with Gemma 2 and Local Gemma\n", + "\n", + "[Gemma](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n", + "Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or your cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.\n", + "\n", + "[Local Gemma](https://github.com/huggingface/local-gemma) provides an easy and fast way to run Gemma-2 locally from your CLI or via a Python library. The Huggingface's [Transformers](https://github.com/huggingface/transformers) and [bitsandbytes](https://huggingface.co/docs/bitsandbytes/index) libraries serve as the foundation for it.\n", + "\n", + "In this notebook, you will learn how to run Gemma 2 models using **Local Gemma** in a Google Colab environment. You'll install the necessary packages, set up the model, and run a sample prompt.\n", + "\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aKZoU1elgfSF" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n", + "\n", + "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **T4 GPU**.\n", + "\n", + "### Gemma setup\n", + "\n", + "**Before you dive into the tutorial, let's get you set up with Gemma:**\n", + "\n", + "1. **Hugging Face Account:** If you don't already have one, you can create a free Hugging Face account by clicking [here](https://huggingface.co/join).\n", + "2. **Gemma Model Access:** Head over to the [Gemma model page](https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315) and accept the usage conditions.\n", + "3. **Colab with Gemma Power:** For this tutorial, you'll need a Colab runtime with enough resources to handle the Gemma 2B model. Choose an appropriate runtime when starting your Colab session.\n", + "4. **Hugging Face Token:** Generate a Hugging Face access (preferably `write` permission) token by clicking [here](https://huggingface.co/settings/tokens). You'll need this token later in the tutorial.\n", + "\n", + "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ceBbRmolghKS" + }, + "source": [ + "### Configure your HF token\n", + "\n", + "Add your Hugging Face token to the Colab Secrets manager to securely store it.\n", + "\n", + "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + "2. Create a new secret with the name `HF_TOKEN`.\n", + "3. Copy/paste your token key into the Value input box of `HF_TOKEN`.\n", + "4. Toggle the button on the left to allow notebook access to the secret." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "aa_KvIwqgkM4" + }, + "outputs": [], + "source": [ + "import os\n", + "from google.colab import userdata\n", + "\n", + "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n", + "# vars as appropriate for your system.\n", + "os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_uaNoIIovh0P" + }, + "source": [ + "### Install dependencies\n", + "\n", + "You'll need to install a few Python packages and dependencies for Local Gemma and interact with HuggingFace.\n", + "\n", + "Run the following cell to install or upgrade it:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "luCD2jFZvik8" + }, + "outputs": [], + "source": [ + "# The huggingface_hub library allows us to download models and other files from Hugging Face.\n", + "!pip install --upgrade -q huggingface_hub\n", + "\n", + "# Install local-gemma \"CUDA\" version\n", + "!pip install -q local-gemma\"[cuda]\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ver08PM1P1AB" + }, + "source": [ + "## Run Local Gemma directly\n", + "\n", + "If you want to chat with the Gemma-2 through an interactive session, run the `local-gemma` command directly. If you are running this command for the first time, it will download the model to the cache.\n", + "\n", + "`local-gemma` command comes with a set of options. You will use `--model` and `--silent` here.\n", + "\n", + "| Option | Details |\n", + "|--------|---------|\n", + "| `--model`| Size of Gemma 2 instruct model to be used in the application ('2b', '9b', or '27b') or, a Hugging Face repo. Defaults to '9b'.|\n", + "|`--silent`| Does NOT print any output except for the model outputs.|\n", + "\n", + "**Note:**\n", + "Input `!exit` to leave the program and input `!new session` to reset the conversation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vDbcZJTSO3w9" + }, + "source": [ + "Use `local-gemma` to run the 2b version of Gemma 2 in command line." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Nw10uOtHQhqV" + }, + "outputs": [], + "source": [ + "!local-gemma --silent --model 2b" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aHNBAV5QOB37" + }, + "source": [ + "To know about all the available options for `local-gemma`, run the following command." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "EICej-FROMxg" + }, + "outputs": [], + "source": [ + "!local-gemma -h" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GvNTOw_vQnCA" + }, + "source": [ + "Alternatively, you can pass a prompt to generate a single response from the model.\n", + "\n", + "`!local-gemma `" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "FcbqapZOzXif" + }, + "outputs": [], + "source": [ + "!local-gemma List top 5 popular programming languages" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "17G2kX_1SVuh" + }, + "source": [ + "## Running the Gemma 2 model using the `local_gemma` library\n", + "\n", + "You can use **Local Gemma** in your Python code using the `local_gemma` library.\n", + "\n", + "You will initialize the `LocalGemma2ForCausalLM` from the `local_gemma` library by loading a pre-trained Gemma 2 model from HuggingFace. Here's what each argument of `from_pretrained` does:\n", + "\n", + "- `pretrained_model_name_or_path`: Path to the model.\n", + "- `preset`: Sets the optimization strategy for the local model deployment. Defaults to 'auto', which selects the best strategy for your device.\n", + "\n", + "For this example, you will use the 9b version of Gemma 2." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "DuKCJqPpLkce" + }, + "outputs": [], + "source": [ + "from local_gemma import LocalGemma2ForCausalLM\n", + "from transformers import AutoTokenizer\n", + "\n", + "model = LocalGemma2ForCausalLM.from_pretrained(\n", + " \"google/gemma-2-2b\", preset=\"memory\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "OpCbI_fAQbsD" + }, + "source": [ + "### Interacting with the Gemma model\n", + "Finally, you can prompt the Gemma 2 model to generate a response.\n", + "\n", + "To do this, you will first tokenize the input prompt by initializing the tokenizer for the selected model(`google/gemma-2-9b`) and passing the prompt as its argument. To learn more about tokenizer and its arguments, read [HuggingFace's Tokenizer documentation](https://huggingface.co/docs/transformers/en/main_classes/tokenizer).\n", + "\n", + "You can then query the model using the tokenized prompt. The response generated by the model is tokenized. It must be decoded into sentences." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qzdHfI13QS_L" + }, + "outputs": [], + "source": [ + "tokenizer = AutoTokenizer.from_pretrained(\"google/gemma-2-2b\")\n", + "\n", + "model_inputs = tokenizer(\"What is your favourite condiment?\",\n", + " return_attention_mask=True,\n", + " return_tensors=\"pt\")\n", + "generated_ids = model.generate(**model_inputs.to(model.device), max_new_tokens=30)\n", + "\n", + "decoded_text = tokenizer.batch_decode(generated_ids)\n", + "decoded_text" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZGisEuosrOe4" + }, + "source": [ + "To learn more about the Python usage of **Local Gemma**, visit [Local Gemma's GitHub page](https://github.com/huggingface/local-gemma?tab=readme-ov-file#python-usage)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pCwGdjKdVEBN" + }, + "source": [ + "## Conclusion" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9-htC4GdoPNV" + }, + "source": [ + "Congratulations! You've successfully set up the Gemma 2 model using **Local Gemma** in a Colab environment. You can now experiment with the model, generate text, and explore its capabilities." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Using_Gemma_with_LocalGemma.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} \ No newline at end of file From d5c9f22df91a8b01fcfc2d1d5e8a5b57983aaf6a Mon Sep 17 00:00:00 2001 From: "nidhin.devan" Date: Fri, 11 Oct 2024 01:30:49 +0530 Subject: [PATCH 03/43] nbformat applied --- Gemma/Using_Gemma_with_LocalGemma.ipynb | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/Gemma/Using_Gemma_with_LocalGemma.ipynb b/Gemma/Using_Gemma_with_LocalGemma.ipynb index d76c27c..e5f00ab 100644 --- a/Gemma/Using_Gemma_with_LocalGemma.ipynb +++ b/Gemma/Using_Gemma_with_LocalGemma.ipynb @@ -326,11 +326,8 @@ "kernelspec": { "display_name": "Python 3", "name": "python3" - }, - "language_info": { - "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +} From 931d14cf94d98b7b4d6256416be2787aae8557d7 Mon Sep 17 00:00:00 2001 From: "nidhin.devan" Date: Fri, 11 Oct 2024 02:03:17 +0530 Subject: [PATCH 04/43] README and wishlist updated --- README.md | 3 ++- WISHLIST.md | 1 - 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index d27a73a..2829a67 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,8 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Gemma2_on_Groq.ipynb](Gemma/Gemma2_on_Groq.ipynb) | Leverage the free Gemma 2 9B IT model hosted on [Groq](https://groq.com/) (super fast speed). | | [Run_with_Ollama.ipynb](Gemma/Run_with_Ollama.ipynb) | Run Gemma models using [Ollama](https://www.ollama.com/). | | [Using_Gemma_with_Llamafile.ipynb](Gemma/Using_Gemma_with_Llamafile.ipynb) | Run Gemma models using [Llamafile](https://github.com/Mozilla-Ocho/llamafile/). | -| [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | +| [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). +| [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | | [Integrate_with_Mesop.ipynb](Gemma/Integrate_with_Mesop.ipynb) | Integrate Gemma with [Google Mesop](https://google.github.io/mesop/). | | [Integrate_with_OneTwo.ipynb](Gemma/Integrate_with_OneTwo.ipynb) | Integrate Gemma with [Google OneTwo](https://github.com/google-deepmind/onetwo). | | [Deploy_with_vLLM.ipynb](Gemma/Deploy_with_vLLM.ipynb) | Deploy a Gemma model using [vLLM](https://github.com/vllm-project/vllm). | diff --git a/WISHLIST.md b/WISHLIST.md index 2f1b776..ea337d3 100644 --- a/WISHLIST.md +++ b/WISHLIST.md @@ -2,7 +2,6 @@ A wish list of cookbooks showcasing: * Inference * Integration with [Google GenKit](https://firebase.google.com/products/genkit) - * HF local-gemma demo * Gemma+Gemini with [routerLLM](https://github.com/lm-sys/RouteLLM) * [SGLang](https://github.com/sgl-project/sglang) integration From db37affb0c81f3ce77ac4eec1d6bd2fd989beda0 Mon Sep 17 00:00:00 2001 From: nidhinpd-YML <78841202+nidhinpd-YML@users.noreply.github.com> Date: Fri, 11 Oct 2024 12:20:30 +0530 Subject: [PATCH 05/43] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2829a67..a8729b7 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Gemma2_on_Groq.ipynb](Gemma/Gemma2_on_Groq.ipynb) | Leverage the free Gemma 2 9B IT model hosted on [Groq](https://groq.com/) (super fast speed). | | [Run_with_Ollama.ipynb](Gemma/Run_with_Ollama.ipynb) | Run Gemma models using [Ollama](https://www.ollama.com/). | | [Using_Gemma_with_Llamafile.ipynb](Gemma/Using_Gemma_with_Llamafile.ipynb) | Run Gemma models using [Llamafile](https://github.com/Mozilla-Ocho/llamafile/). | -| [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). +| [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | | [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | | [Integrate_with_Mesop.ipynb](Gemma/Integrate_with_Mesop.ipynb) | Integrate Gemma with [Google Mesop](https://google.github.io/mesop/). | | [Integrate_with_OneTwo.ipynb](Gemma/Integrate_with_OneTwo.ipynb) | Integrate Gemma with [Google OneTwo](https://github.com/google-deepmind/onetwo). | From cf3a4c7a8809af0ba26c72e499ae88c7fcc66ce3 Mon Sep 17 00:00:00 2001 From: "nidhin.devan" Date: Fri, 11 Oct 2024 12:24:27 +0530 Subject: [PATCH 06/43] README updated --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index a8729b7..ca0c73b 100644 --- a/README.md +++ b/README.md @@ -50,8 +50,8 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Gemma2_on_Groq.ipynb](Gemma/Gemma2_on_Groq.ipynb) | Leverage the free Gemma 2 9B IT model hosted on [Groq](https://groq.com/) (super fast speed). | | [Run_with_Ollama.ipynb](Gemma/Run_with_Ollama.ipynb) | Run Gemma models using [Ollama](https://www.ollama.com/). | | [Using_Gemma_with_Llamafile.ipynb](Gemma/Using_Gemma_with_Llamafile.ipynb) | Run Gemma models using [Llamafile](https://github.com/Mozilla-Ocho/llamafile/). | -| [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | -| [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | +| [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | +| [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | | [Integrate_with_Mesop.ipynb](Gemma/Integrate_with_Mesop.ipynb) | Integrate Gemma with [Google Mesop](https://google.github.io/mesop/). | | [Integrate_with_OneTwo.ipynb](Gemma/Integrate_with_OneTwo.ipynb) | Integrate Gemma with [Google OneTwo](https://github.com/google-deepmind/onetwo). | | [Deploy_with_vLLM.ipynb](Gemma/Deploy_with_vLLM.ipynb) | Deploy a Gemma model using [vLLM](https://github.com/vllm-project/vllm). | From 569b1093140fe72849eed747f2725155388851ac Mon Sep 17 00:00:00 2001 From: NSTiwari Date: Fri, 11 Oct 2024 13:33:38 +0530 Subject: [PATCH 07/43] Removed HF_TOKEN from test.py --- .../PaliGemma-on-Android/Python_Server/apiserver/api/test.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/PaliGemma/PaliGemma-on-Android/Python_Server/apiserver/api/test.py b/PaliGemma/PaliGemma-on-Android/Python_Server/apiserver/api/test.py index e59d575..3bb8fe8 100644 --- a/PaliGemma/PaliGemma-on-Android/Python_Server/apiserver/api/test.py +++ b/PaliGemma/PaliGemma-on-Android/Python_Server/apiserver/api/test.py @@ -2,7 +2,7 @@ def detect(): - client = Client("big-vision/paligemma", hf_token="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") + client = Client("big-vision/paligemma") result = client.predict( handle_file('../media/images/person_and_car.jpg'), # filepath in 'Image' Image component "Detect person and car", From 23f3196748586ae2336eb7e5bf1a798150709ca6 Mon Sep 17 00:00:00 2001 From: Wei Wei <5577741+windmaple@users.noreply.github.com> Date: Sat, 12 Oct 2024 13:26:02 +0800 Subject: [PATCH 08/43] Update README a bit --- README.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ca0c73b..c60cfc1 100644 --- a/README.md +++ b/README.md @@ -21,9 +21,20 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Hugging Face | [Utilize Hugging Face Transformers and TRL for fine-tuning and inference tasks with Gemma models.](partner-quickstarts/gemma-huggingface.md) | | NVIDIA | [Fine-tune Gemma models with NVIDIA NeMo Framework and export to TensorRT-LLM for production.](partner-quickstarts/Gemma-NVidia/) | -| LangChain | This [tutorial](partner-quickstarts/gemma-langchain.ipynb) shows you how to get started with Gemma and LangChain, running in Google Cloud or in your Colab environment. | | +| LangChain | This [tutorial](partner-quickstarts/gemma-langchain.ipynb) shows you how to get started with Gemma and LangChain, running in Google Cloud or in your Colab environment. | | MongoDB | This [article](partner-quickstarts/rag_with_hugging_face_gemma_mongodb.ipynb) presents how to leverage Gemma as the foundation model in a retrieval-augmented generation pipeline or system. | +## Workshops +| Notebook | Description | +| --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | +| [Workshop_How_to_Fine_tuning_Gemma.ipynb](Workshops/Workshop_How_to_Fine_tuning_Gemma.ipynb/Keras_Gemma_2_Quickstart.ipynb) | Recommended finetuning notebook for getting started | + +## Accompanying notebooks for the Build with AI video series +| Folder | +| ----------------------------------------------------------- | +| [Business email assistant](Gemma/business-email-assistant/) | +| [Personal code assistant](Gemma/personal-code-assistant/) | +| [Spoken language tasks](Gemma/spoken-language-tasks/) | ## Cookbook table of contents From 22ef3d6b8f927d67211065f4d0a4c51890426a15 Mon Sep 17 00:00:00 2001 From: Wei Wei <5577741+windmaple@users.noreply.github.com> Date: Mon, 14 Oct 2024 09:40:34 +0800 Subject: [PATCH 09/43] Update README --- README.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c60cfc1..d8cfe18 100644 --- a/README.md +++ b/README.md @@ -24,10 +24,12 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | LangChain | This [tutorial](partner-quickstarts/gemma-langchain.ipynb) shows you how to get started with Gemma and LangChain, running in Google Cloud or in your Colab environment. | | MongoDB | This [article](partner-quickstarts/rag_with_hugging_face_gemma_mongodb.ipynb) presents how to leverage Gemma as the foundation model in a retrieval-augmented generation pipeline or system. | -## Workshops -| Notebook | Description | -| --------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | -| [Workshop_How_to_Fine_tuning_Gemma.ipynb](Workshops/Workshop_How_to_Fine_tuning_Gemma.ipynb/Keras_Gemma_2_Quickstart.ipynb) | Recommended finetuning notebook for getting started | +## Workshops and technical talks +| Notebook | Description | +| --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| [Workshop_How_to_Fine_tuning_Gemma.ipynb](Workshops/Workshop_How_to_Fine_tuning_Gemma.ipynb/Keras_Gemma_2_Quickstart.ipynb) | Recommended finetuning notebook for getting started | +| [Self_extend_Gemma.ipynb](Gemma/Self_extend_Gemma.ipynb) | Self-extend context window for Gemma in the I/O 2024 [Keras talk](https://www.youtube.com/watch?v=TV7qCk1dBWA) | +| [Gemma_control_vectors.ipynb](Gemma/Gemma_control_vectors.ipynb) | Implement [control vectors](https://arxiv.org/abs/2310.01405) with Gemma in the I/O 2024 [Keras talk](https://www.youtube.com/watch?v=TV7qCk1dBWA) | ## Accompanying notebooks for the Build with AI video series | Folder | @@ -51,7 +53,6 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Chat_and_distributed_pirate_tuning.ipynb](Gemma/Chat_and_distributed_pirate_tuning.ipynb) | Chat with Gemma 7B and finetune it so that it generates responses in pirates' tone. | | [gemma_inference_on_tpu.ipynb](Gemma/gemma_inference_on_tpu.ipynb) | Basic inference of Gemma with JAX/Flax on TPU. | | [gemma_data_parallel_inference_in_jax_tpu.ipynb](Gemma/gemma_data_parallel_inference_in_jax_tpu.ipynb) | Parallel inference of Gemma with JAX/Flax on TPU. | -| [Gemma_control_vectors.ipynb](Gemma/Gemma_control_vectors.ipynb) | Implement [control vectors](https://arxiv.org/abs/2310.01405) with Gemma in the I/O 2024 [Keras talk](https://www.youtube.com/watch?v=TV7qCk1dBWA). | | [Gemma_Basics_with_HF.ipynb](Gemma/Gemma_Basics_with_HF.ipynb) | Load, run, finetune and deploy Gemma using [Hugging Face](https://huggingface.co/). | | [Gemma_with_Langfun_and_LlamaCpp.ipynb](Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb) | Leverage [Langfun](https://github.com/google/langfun) to seamlessly integrate natural language with programming using Gemma 2 and [LlamaCpp](https://github.com/ggerganov/llama.cpp). | | [Gemma_with_Langfun_and_LlamaCpp_Python_Bindings.ipynb](Gemma/Gemma_with_Langfun_and_LlamaCpp_Python_Bindings.ipynb) | Leverage [Langfun](https://github.com/google/langfun) for smooth language-program interaction with Gemma 2 and [llama-cpp-python](https://github.com/abetlen/llama-cpp-python). | @@ -71,8 +72,6 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Prompt_chaining.ipynb](Gemma/Prompt_chaining.ipynb) | Illustrate prompt chaining and iterative generation with Gemma. | | [LangChain_chaining.ipynb](Gemma/LangChain_chaining.ipynb) | Illustrate LangChain chaining with Gemma. | | [Advanced_Prompting_Techniques.ipynb](Gemma/Advanced_Prompting_Techniques.ipynb) | Illustrate advanced prompting techniques with Gemma. | -| **Long context** | | -| [Self_extend_Gemma.ipynb](Gemma/Self_extend_Gemma.ipynb) | Self-extend context window for Gemma in the I/O 2024 [Keras talk](https://www.youtube.com/watch?v=TV7qCk1dBWA). | | **RAG** | | | [RAG_with_ChromaDB.ipynb](Gemma/RAG_with_ChromaDB.ipynb) | Build a Retrieval Augmented Generation (RAG) system with Gemma using [ChromaDB](https://www.trychroma.com/) and [Hugging Face](https://huggingface.co/). | | [Minimal_RAG.ipynb](Gemma/Minimal_RAG.ipynb) | Minimal example of building a RAG system with Gemma using [Google UniSim](https://github.com/google/unisim) and [Hugging Face](https://huggingface.co/). | From b5cca0456826d335b2945f4c9cf014d3c8b8cf2a Mon Sep 17 00:00:00 2001 From: Kinar R Date: Wed, 16 Oct 2024 23:43:33 +0530 Subject: [PATCH 10/43] Remove unused blocks of code/text --- Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb | 14 -------------- ...ith_Langfun_and_LlamaCpp_Python_Bindings.ipynb | 15 --------------- ...g_Gemma_with_Elasticsearch_and_LangChain.ipynb | 2 +- 3 files changed, 1 insertion(+), 30 deletions(-) diff --git a/Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb b/Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb index 0fe612d..8f222db 100644 --- a/Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb +++ b/Gemma/Gemma_with_Langfun_and_LlamaCpp.ipynb @@ -672,20 +672,6 @@ "In this example, you'll demonstrate the model's capability in performing CoT reasoning for a simple puzzle.\n" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "pQW9oUO0ch8O" - }, - "outputs": [], - "source": [ - "question = (\n", - " 'Janetโ€™s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. '\n", - " 'She sells the remainder at the farmers\\' market daily for $2 per fresh duck egg. '\n", - " 'How much in dollars does she make every day at the farmers\\' market? ')" - ] - }, { "cell_type": "code", "execution_count": null, diff --git a/Gemma/Gemma_with_Langfun_and_LlamaCpp_Python_Bindings.ipynb b/Gemma/Gemma_with_Langfun_and_LlamaCpp_Python_Bindings.ipynb index 55142b5..400a426 100644 --- a/Gemma/Gemma_with_Langfun_and_LlamaCpp_Python_Bindings.ipynb +++ b/Gemma/Gemma_with_Langfun_and_LlamaCpp_Python_Bindings.ipynb @@ -833,21 +833,6 @@ "In this example, you'll demonstrate the model's capability in performing CoT reasoning for a simple puzzle.\n" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "pQW9oUO0ch8O" - }, - "outputs": [], - "source": [ - "question = (\n", - " 'Janetโ€™s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. '\n", - " 'She sells the remainder at the farmers\\' market daily for $2 per fresh duck egg. '\n", - " 'How much in dollars does she make every day at the farmers\\' market? '\n", - " 'Think step-by-step, solve the puzzle and generate Solution')" - ] - }, { "cell_type": "code", "execution_count": null, diff --git a/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb b/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb index 8f70741..1e23494 100644 --- a/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb +++ b/Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb @@ -133,7 +133,7 @@ "### Configure Your Credentials\n", "\n", "\n", - "Next, we'll securely store your Hugging Face token and Kaggle credentials using the Colab Secrets manager:\n", + "Next, we'll securely store your Hugging Face token using the Colab Secrets manager:\n", "\n", "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", "2. **Add Hugging Face Token**:\n", From 888e5436a1a274a10361fb9646173fbe5d3105ef Mon Sep 17 00:00:00 2001 From: Juyeong Ji Date: Mon, 21 Oct 2024 10:30:32 +0900 Subject: [PATCH 11/43] =?UTF-8?q?Colab=20=E3=82=92=E4=BD=BF=E7=94=A8?= =?UTF-8?q?=E3=81=97=E3=81=A6=E4=BD=9C=E6=88=90=E3=81=95=E3=82=8C=E3=81=BE?= =?UTF-8?q?=E3=81=97=E3=81=9F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Gemma/Custom_Vocabulary.ipynb | 2381 +++++++++++++++++++++++++++++++++ 1 file changed, 2381 insertions(+) create mode 100644 Gemma/Custom_Vocabulary.ipynb diff --git a/Gemma/Custom_Vocabulary.ipynb b/Gemma/Custom_Vocabulary.ipynb new file mode 100644 index 0000000..a31eeba --- /dev/null +++ b/Gemma/Custom_Vocabulary.ipynb @@ -0,0 +1,2381 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [], + "gpuType": "T4", + "toc_visible": true, + "authorship_tag": "ABX9TyNzoetBHJ/RyGzWlzpUM8Ag" + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + }, + "accelerator": "GPU" + }, + "cells": [ + { + "cell_type": "markdown", + "source": [ + "##### Copyright 2024 Google LLC." + ], + "metadata": { + "id": "hovLNQ-tZuwd" + } + }, + { + "cell_type": "code", + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ], + "metadata": { + "id": "yDENQhUBZxfL" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "# Gemma - How to use a custom vocabulary\n", + "\n", + "This notebook demonstrates how to use a custom vocabulary in Gemma.\n", + "\n", + "Consider a document containing a length word, such as \"MyCustomWordInMyDocument\" with a high frequency of occurrence. It will be fragmented into several tokens, resulting in inefficiencies. Gemma Tokenizer offeres a potential solution in the form of \"\" tokens. However, this approach requires additional training, as the base model lacks knowledge of these tokens.\n", + "\n", + "In this demo, you will fine-tune the model with a simple prompt \"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\" which means \"Rare Japanese surnames\".\n", + "\n", + "```\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "```\n", + "\n", + "Based on your application's requirements, you must replace your custom vocabulary with unused tokens. Then, feed the converted training datasets to the model. During the tuning process, the base model will learn the patterns in your document and start generating unused tokens accordingly. Finally, before printing the output text, convert it back to your custom vocabulary.\n", + "\n", + "Due to the fixed size of the vocabulary, adding a significant number of new tokens is not feasible. To do so, you would need to resize the tokenizer and retrain the model, which is inefficient and not recommended unless absolutely necessary.\n", + "\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
\n" + ], + "metadata": { + "id": "eNmbq93EZzID" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:\n", + "\n", + "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **T4 GPU**.\n", + "\n", + "\n", + "### Gemma setup on Kaggle\n", + "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n", + "\n", + "* Get access to Gemma on kaggle.com.\n", + "* Select a Colab runtime with sufficient resources to run the Gemma 2B model.\n", + "* Generate and configure a Kaggle username and API key.\n", + "\n", + "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment." + ], + "metadata": { + "id": "nathXe5ebTQI" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Set environment variables\n", + "\n", + "Set environment variables for `KAGGLE_USERNAME` and `KAGGLE_KEY`." + ], + "metadata": { + "id": "2SNOwNf6OV28" + } + }, + { + "cell_type": "code", + "source": [ + "import os\n", + "from google.colab import userdata, drive\n", + "\n", + "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n", + "# vars as appropriate for your system.\n", + "os.environ[\"KAGGLE_USERNAME\"] = userdata.get(\"KAGGLE_USERNAME\")\n", + "os.environ[\"KAGGLE_KEY\"] = userdata.get(\"KAGGLE_KEY\")\n", + "\n", + "# Mounting gDrive for to store artifacts\n", + "drive.mount(\"/content/drive\")" + ], + "metadata": { + "id": "HuGBBTQlbX7w" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "### Install dependencies\n", + "\n", + "Install Keras and KerasNLP" + ], + "metadata": { + "id": "1gaYpiL-bgWZ" + } + }, + { + "cell_type": "code", + "source": [ + "!pip install -q -U keras-nlp datasets\n", + "!pip install -q -U keras\n", + "\n", + "# Set the backbend before importing Keras\n", + "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n", + "# Avoid memory fragmentation on JAX backend.\n", + "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n", + "\n", + "import keras_nlp\n", + "import keras\n", + "\n", + "# Run at half precision.\n", + "#keras.config.set_floatx(\"bfloat16\")\n", + "\n", + "# Training Configurations\n", + "token_limit = 128\n", + "num_data_limit = 99\n", + "lora_name = \"my_lora\"\n", + "lora_rank = 4\n", + "lr_value = 5e-4\n", + "train_epoch = 5\n", + "model_id = \"gemma2_instruct_2b_en\"" + ], + "metadata": { + "id": "CcMRCmR4OVdx" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "## Identify the presence of Surnames in the Tokenizer's vocabulary\n", + "\n", + "Regarding the base Gemma Tokenizer, see below that it includes frequently used names, such as \"ไฝ่—ค\" and \"้ˆดๆœจ\", within its vocabulary. However, for less common names, the tokenizer begins to split them into multiple tokens, like below:\n", + "\n", + "```\n", + "235585 -> ๅฐ\n", + "240763 -> ๅถ‹\n", + " 4758 -> ใ•ใ‚“\n", + "```\n", + "\n", + "It's getting worse if you go very rare surnames like below:\n", + "\n", + "```\n", + "235771 -> ๅŠ \n", + " 450 -> <0xE9>\n", + " 389 -> <0xAC>\n", + " 391 -> <0xAE>\n", + " 4758 -> ใ•ใ‚“\n", + "```\n", + "\n", + "In the case of \"ๅŠ ้ฌฎ\", tokenizer typically splits this character into its UTF-8 representation. This is because the tokenizer hasn't encountered this character frequently during the training process. If you fine-tune the model, it will significantly improve efficiency. For example \"ๅŠ ้ฌฎ\", 4 tokens can be reduced to 1 token." + ], + "metadata": { + "id": "0DSjJE4aOKc7" + } + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Wp2aaKJ8OE-y", + "outputId": "79d08d08-5d3d-44b9-fb1a-4c97975b4bf5" + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "[ 108 56985 32012 235465 78638 4758 235394 79900 4758 235394\n", + " 122224 4758 235394 98647 4758 235394 119177 4758 235394 169944\n", + " 4758 235394 113698 4758 235394 119392 4758 235394 117774 4758\n", + " 235394 140762 4758 108 235308 235276 235274 235893 235465 235585\n", + " 240763 4758 235394 239713 235493 4758 235394 235469 236872 4758\n", + " 235394 241122 235722 4758 235394 238303 235841 4758 235394 157967\n", + " 4758 235394 238803 238313 4758 235394 235861 235875 4758 235394\n", + " 236447 236063 4758 235394 240052 236872 4758 108 235304 235324\n", + " 235276 235276 235274 235893 235465 235661 236316 4758 235394 238881\n", + " 235842 4758 235394 235469 239319 4758 235394 237338 235502 4758\n", + " 235394 237550 236228 4758 235394 235648 235858 4758 235394 243141\n", + " 238022 4758 235394 235771 450 389 391 4758 235394 236497\n", + " 241335 4758 235394 235990 242446 4758 108]\n", + " 108 -> \n", + "\n", + " 56985 -> ใƒˆใƒƒใƒ—\n", + " 32012 -> ใƒ†ใƒณ\n", + "235465 -> ๏ผš\n", + " 78638 -> ไฝ่—ค\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + " 79900 -> ้ˆดๆœจ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "122224 -> ้ซ˜ๆฉ‹\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + " 98647 -> ็”ฐไธญ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "119177 -> ไผŠ่—ค\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "169944 -> ๆธก่พบ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "113698 -> ๅฑฑๆœฌ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "119392 -> ไธญๆ‘\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "117774 -> ๅฐๆž—\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "140762 -> ๅŠ ่—ค\n", + " 4758 -> ใ•ใ‚“\n", + " 108 -> \n", + "\n", + "235308 -> 5\n", + "235276 -> 0\n", + "235274 -> 1\n", + "235893 -> ไฝ\n", + "235465 -> ๏ผš\n", + "235585 -> ๅฐ\n", + "240763 -> ๅถ‹\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "239713 -> ็•‘\n", + "235493 -> ไธญ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "235469 -> ๅคง\n", + "236872 -> ไบ•\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "241122 -> ็ฃฏ\n", + "235722 -> ้ƒจ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "238303 -> ๆต…\n", + "235841 -> ่ฆ‹\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "157967 -> ็ง‹็”ฐ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "238803 -> ่Šณ\n", + "238313 -> ่ณ€\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "235861 -> ็›ธ\n", + "235875 -> ๅŽŸ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "236447 -> ็ดฐ\n", + "236063 -> ็”ฐ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "240052 -> ๅช\n", + "236872 -> ไบ•\n", + " 4758 -> ใ•ใ‚“\n", + " 108 -> \n", + "\n", + "235304 -> 3\n", + "235324 -> 7\n", + "235276 -> 0\n", + "235276 -> 0\n", + "235274 -> 1\n", + "235893 -> ไฝ\n", + "235465 -> ๏ผš\n", + "235661 -> ๆณ•\n", + "236316 -> ๅท\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "238881 -> ไน™\n", + "235842 -> ้–“\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "235469 -> ๅคง\n", + "239319 -> ่ˆŒ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "237338 -> ๅทป\n", + "235502 -> ไธŠ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "237550 -> ๆˆธ\n", + "236228 -> ไฝ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "235648 -> ๅ‰\n", + "235858 -> ๆ›ด\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "243141 -> ๆขถ\n", + "238022 -> ๆตœ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "235771 -> ๅŠ \n", + " 450 -> ๏ฟฝ\n", + " 389 -> ๏ฟฝ\n", + " 391 -> ๏ฟฝ\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "236497 -> ๅƒ\n", + "241335 -> ่‰˜\n", + " 4758 -> ใ•ใ‚“\n", + "235394 -> ใ€\n", + "235990 -> ่ฅฟ\n", + "242446 -> ่ƒค\n", + " 4758 -> ใ•ใ‚“\n", + " 108 -> \n", + "\n" + ] + } + ], + "source": [ + "tokenizer = keras_nlp.models.GemmaTokenizer.from_preset(model_id)\n", + "\n", + "def detoken(tokens):\n", + " print(tokens)\n", + " for x in tokens:\n", + " word = tokenizer.detokenize([x])\n", + " print(f\"{x:6} -> {word}\")\n", + "\n", + "detoken(tokenizer(\"\"\"\n", + "ใƒˆใƒƒใƒ—ใƒ†ใƒณ๏ผšไฝ่—คใ•ใ‚“ใ€้ˆดๆœจใ•ใ‚“ใ€้ซ˜ๆฉ‹ใ•ใ‚“ใ€็”ฐไธญใ•ใ‚“ใ€ไผŠ่—คใ•ใ‚“ใ€ๆธก่พบใ•ใ‚“ใ€ๅฑฑๆœฌใ•ใ‚“ใ€ไธญๆ‘ใ•ใ‚“ใ€ๅฐๆž—ใ•ใ‚“ใ€ๅŠ ่—คใ•ใ‚“\n", + "501ไฝ๏ผšๅฐๅถ‹ใ•ใ‚“ใ€็•‘ไธญใ•ใ‚“ใ€ๅคงไบ•ใ•ใ‚“ใ€็ฃฏ้ƒจใ•ใ‚“ใ€ๆต…่ฆ‹ใ•ใ‚“ใ€็ง‹็”ฐใ•ใ‚“ใ€่Šณ่ณ€ใ•ใ‚“ใ€็›ธๅŽŸใ•ใ‚“ใ€็ดฐ็”ฐใ•ใ‚“ใ€ๅชไบ•ใ•ใ‚“\n", + "37001ไฝ๏ผšๆณ•ๅทใ•ใ‚“ใ€ไน™้–“ใ•ใ‚“ใ€ๅคง่ˆŒใ•ใ‚“ใ€ๅทปไธŠใ•ใ‚“ใ€ๆˆธไฝใ•ใ‚“ใ€ๅ‰ๆ›ดใ•ใ‚“ใ€ๆขถๆตœใ•ใ‚“ใ€ๅŠ ้ฌฎใ•ใ‚“ใ€ๅƒ่‰˜ใ•ใ‚“ใ€่ฅฟ่ƒคใ•ใ‚“\n", + "\"\"\"))" + ] + }, + { + "cell_type": "code", + "source": [ + "print(tokenizer.id_to_token(450))\n", + "print(tokenizer.id_to_token(389))\n", + "print(tokenizer.id_to_token(391))" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "pMFgoID3FAcJ", + "outputId": "8e89279a-efc3-457e-df03-7206a2a068bf" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "<0xE9>\n", + "<0xAC>\n", + "<0xAE>\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "tokenizer.get_vocabulary()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "ggigihbgvkLc", + "outputId": "f53fcb4f-3f17-40c1-e956-dedbad1cf3c3" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "['',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '<2mass>',\n", + " '[@BOS@]',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '\\n',\n", + " '\\n\\n',\n", + " '\\n\\n\\n',\n", + " '\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " '\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n',\n", + " 'โ–โ–',\n", + " 'โ–โ–โ–',\n", + " 'โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " 'โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '
',\n", + " '
',\n", + " '',\n", + " '
',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '

',\n", + " '

',\n", + " '

',\n", + " '

',\n", + " '

',\n", + " '
',\n", + " '
',\n", + " '
',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '',\n", + " '<0x00>',\n", + " '<0x01>',\n", + " '<0x02>',\n", + " '<0x03>',\n", + " '<0x04>',\n", + " '<0x05>',\n", + " '<0x06>',\n", + " '<0x07>',\n", + " '<0x08>',\n", + " '<0x09>',\n", + " '<0x0A>',\n", + " '<0x0B>',\n", + " '<0x0C>',\n", + " '<0x0D>',\n", + " '<0x0E>',\n", + " '<0x0F>',\n", + " '<0x10>',\n", + " '<0x11>',\n", + " '<0x12>',\n", + " '<0x13>',\n", + " '<0x14>',\n", + " '<0x15>',\n", + " '<0x16>',\n", + " '<0x17>',\n", + " '<0x18>',\n", + " '<0x19>',\n", + " '<0x1A>',\n", + " '<0x1B>',\n", + " '<0x1C>',\n", + " '<0x1D>',\n", + " '<0x1E>',\n", + " '<0x1F>',\n", + " '<0x20>',\n", + " '<0x21>',\n", + " '<0x22>',\n", + " '<0x23>',\n", + " '<0x24>',\n", + " '<0x25>',\n", + " '<0x26>',\n", + " '<0x27>',\n", + " '<0x28>',\n", + " '<0x29>',\n", + " '<0x2A>',\n", + " '<0x2B>',\n", + " '<0x2C>',\n", + " '<0x2D>',\n", + " '<0x2E>',\n", + " '<0x2F>',\n", + " '<0x30>',\n", + " '<0x31>',\n", + " '<0x32>',\n", + " '<0x33>',\n", + " '<0x34>',\n", + " '<0x35>',\n", + " '<0x36>',\n", + " '<0x37>',\n", + " '<0x38>',\n", + " '<0x39>',\n", + " '<0x3A>',\n", + " '<0x3B>',\n", + " '<0x3C>',\n", + " '<0x3D>',\n", + " '<0x3E>',\n", + " '<0x3F>',\n", + " '<0x40>',\n", + " '<0x41>',\n", + " '<0x42>',\n", + " '<0x43>',\n", + " '<0x44>',\n", + " '<0x45>',\n", + " '<0x46>',\n", + " '<0x47>',\n", + " '<0x48>',\n", + " '<0x49>',\n", + " '<0x4A>',\n", + " '<0x4B>',\n", + " '<0x4C>',\n", + " '<0x4D>',\n", + " '<0x4E>',\n", + " '<0x4F>',\n", + " '<0x50>',\n", + " '<0x51>',\n", + " '<0x52>',\n", + " '<0x53>',\n", + " '<0x54>',\n", + " '<0x55>',\n", + " '<0x56>',\n", + " '<0x57>',\n", + " '<0x58>',\n", + " '<0x59>',\n", + " '<0x5A>',\n", + " '<0x5B>',\n", + " '<0x5C>',\n", + " '<0x5D>',\n", + " '<0x5E>',\n", + " '<0x5F>',\n", + " '<0x60>',\n", + " '<0x61>',\n", + " '<0x62>',\n", + " '<0x63>',\n", + " '<0x64>',\n", + " '<0x65>',\n", + " '<0x66>',\n", + " '<0x67>',\n", + " '<0x68>',\n", + " '<0x69>',\n", + " '<0x6A>',\n", + " '<0x6B>',\n", + " '<0x6C>',\n", + " '<0x6D>',\n", + " '<0x6E>',\n", + " '<0x6F>',\n", + " '<0x70>',\n", + " '<0x71>',\n", + " '<0x72>',\n", + " '<0x73>',\n", + " '<0x74>',\n", + " '<0x75>',\n", + " '<0x76>',\n", + " '<0x77>',\n", + " '<0x78>',\n", + " '<0x79>',\n", + " '<0x7A>',\n", + " '<0x7B>',\n", + " '<0x7C>',\n", + " '<0x7D>',\n", + " '<0x7E>',\n", + " '<0x7F>',\n", + " '<0x80>',\n", + " '<0x81>',\n", + " '<0x82>',\n", + " '<0x83>',\n", + " '<0x84>',\n", + " '<0x85>',\n", + " '<0x86>',\n", + " '<0x87>',\n", + " '<0x88>',\n", + " '<0x89>',\n", + " '<0x8A>',\n", + " '<0x8B>',\n", + " '<0x8C>',\n", + " '<0x8D>',\n", + " '<0x8E>',\n", + " '<0x8F>',\n", + " '<0x90>',\n", + " '<0x91>',\n", + " '<0x92>',\n", + " '<0x93>',\n", + " '<0x94>',\n", + " '<0x95>',\n", + " '<0x96>',\n", + " '<0x97>',\n", + " '<0x98>',\n", + " '<0x99>',\n", + " '<0x9A>',\n", + " '<0x9B>',\n", + " '<0x9C>',\n", + " '<0x9D>',\n", + " '<0x9E>',\n", + " '<0x9F>',\n", + " '<0xA0>',\n", + " '<0xA1>',\n", + " '<0xA2>',\n", + " '<0xA3>',\n", + " '<0xA4>',\n", + " '<0xA5>',\n", + " '<0xA6>',\n", + " '<0xA7>',\n", + " '<0xA8>',\n", + " '<0xA9>',\n", + " '<0xAA>',\n", + " '<0xAB>',\n", + " '<0xAC>',\n", + " '<0xAD>',\n", + " '<0xAE>',\n", + " '<0xAF>',\n", + " '<0xB0>',\n", + " '<0xB1>',\n", + " '<0xB2>',\n", + " '<0xB3>',\n", + " '<0xB4>',\n", + " '<0xB5>',\n", + " '<0xB6>',\n", + " '<0xB7>',\n", + " '<0xB8>',\n", + " '<0xB9>',\n", + " '<0xBA>',\n", + " '<0xBB>',\n", + " '<0xBC>',\n", + " '<0xBD>',\n", + " '<0xBE>',\n", + " '<0xBF>',\n", + " '<0xC0>',\n", + " '<0xC1>',\n", + " '<0xC2>',\n", + " '<0xC3>',\n", + " '<0xC4>',\n", + " '<0xC5>',\n", + " '<0xC6>',\n", + " '<0xC7>',\n", + " '<0xC8>',\n", + " '<0xC9>',\n", + " '<0xCA>',\n", + " '<0xCB>',\n", + " '<0xCC>',\n", + " '<0xCD>',\n", + " '<0xCE>',\n", + " '<0xCF>',\n", + " '<0xD0>',\n", + " '<0xD1>',\n", + " '<0xD2>',\n", + " '<0xD3>',\n", + " '<0xD4>',\n", + " '<0xD5>',\n", + " '<0xD6>',\n", + " '<0xD7>',\n", + " '<0xD8>',\n", + " '<0xD9>',\n", + " '<0xDA>',\n", + " '<0xDB>',\n", + " '<0xDC>',\n", + " '<0xDD>',\n", + " '<0xDE>',\n", + " '<0xDF>',\n", + " '<0xE0>',\n", + " '<0xE1>',\n", + " '<0xE2>',\n", + " '<0xE3>',\n", + " '<0xE4>',\n", + " '<0xE5>',\n", + " '<0xE6>',\n", + " '<0xE7>',\n", + " '<0xE8>',\n", + " '<0xE9>',\n", + " '<0xEA>',\n", + " '<0xEB>',\n", + " '<0xEC>',\n", + " '<0xED>',\n", + " '<0xEE>',\n", + " '<0xEF>',\n", + " '<0xF0>',\n", + " '<0xF1>',\n", + " '<0xF2>',\n", + " '<0xF3>',\n", + " '<0xF4>',\n", + " '<0xF5>',\n", + " '<0xF6>',\n", + " '<0xF7>',\n", + " '<0xF8>',\n", + " '<0xF9>',\n", + " '<0xFA>',\n", + " '<0xFB>',\n", + " '<0xFC>',\n", + " '<0xFD>',\n", + " '<0xFE>',\n", + " '<0xFF>',\n", + " 'in',\n", + " 'โ–t',\n", + " 'er',\n", + " 'โ–a',\n", + " 'on',\n", + " 're',\n", + " 'en',\n", + " 'he',\n", + " 'an',\n", + " 'at',\n", + " 'or',\n", + " 'es',\n", + " 'โ–s',\n", + " 'ar',\n", + " 'ti',\n", + " 'te',\n", + " 'th',\n", + " 'st',\n", + " 'nd',\n", + " 'al',\n", + " 'โ–o',\n", + " 'le',\n", + " 'de',\n", + " 'โ–i',\n", + " 'se',\n", + " 'โ–c',\n", + " 'โ–d',\n", + " 'it',\n", + " 'nt',\n", + " 'is',\n", + " 'โ–p',\n", + " 'me',\n", + " 'ri',\n", + " 'ra',\n", + " 'ou',\n", + " 'as',\n", + " 'ed',\n", + " 'ne',\n", + " 'to',\n", + " 'ng',\n", + " 'โ–w',\n", + " 'ro',\n", + " 'li',\n", + " 'ta',\n", + " 'โ–f',\n", + " 'โ–b',\n", + " 'โ–m',\n", + " 'ic',\n", + " 'el',\n", + " 'la',\n", + " 'et',\n", + " 've',\n", + " 'ur',\n", + " 'โ–e',\n", + " 'ha',\n", + " 'co',\n", + " 'll',\n", + " 'ch',\n", + " 'โ–h',\n", + " 'ce',\n", + " 'โ–l',\n", + " 'ma',\n", + " 'ea',\n", + " 'io',\n", + " 'om',\n", + " 'tr',\n", + " 'id',\n", + " 'il',\n", + " 'ge',\n", + " 'si',\n", + " 'di',\n", + " 'hi',\n", + " 'lo',\n", + " 'ec',\n", + " 'ie',\n", + " 'โ–r',\n", + " 'un',\n", + " 'ac',\n", + " 'ct',\n", + " 'โ–n',\n", + " 'us',\n", + " 'pe',\n", + " 'be',\n", + " 'na',\n", + " 'ca',\n", + " 'ns',\n", + " 'of',\n", + " 'ut',\n", + " 'ol',\n", + " 'ot',\n", + " 'am',\n", + " 'ss',\n", + " 'em',\n", + " 'ad',\n", + " 'po',\n", + " 'os',\n", + " 'pa',\n", + " 'โ–S',\n", + " 'im',\n", + " 'ho',\n", + " 'โ–the',\n", + " 'ing',\n", + " 'โ–in',\n", + " 'โ–of',\n", + " 'โ–to',\n", + " 'โ–and',\n", + " 'ent',\n", + " 'โ–th',\n", + " 'โ–de',\n", + " 'โ–re',\n", + " 'โ–g',\n", + " 'โ–T',\n", + " 'โ–C',\n", + " 'โ–A',\n", + " 'ion',\n", + " 'tion',\n", + " 'โ–=',\n", + " 'โ–I',\n", + " 'โ–(',\n", + " 'qu',\n", + " 'โ–v',\n", + " ');',\n", + " 'โ–M',\n", + " 'โ–P',\n", + " 'โ–y',\n", + " 'ver',\n", + " 'โ–B',\n", + " 'um',\n", + " 'ul',\n", + " 'ig',\n", + " 'โ–is',\n", + " 'โ–for',\n", + " 'ly',\n", + " 'โ–u',\n", + " 'ate',\n", + " 'โ–D',\n", + " 'ow',\n", + " 'ter',\n", + " 'โ–on',\n", + " 'โ–{',\n", + " 'โ–wi',\n", + " 'โ–be',\n", + " 'end',\n", + " 'ir',\n", + " 'ts',\n", + " 'ers',\n", + " 'ue',\n", + " 'hat',\n", + " 'ati',\n", + " 'ine',\n", + " 'ck',\n", + " 'res',\n", + " 'โ–R',\n", + " 'ay',\n", + " '=\"',\n", + " 'gh',\n", + " 'โ–L',\n", + " 'ons',\n", + " 'men',\n", + " 'โ–con',\n", + " 'โ–F',\n", + " '//',\n", + " 'int',\n", + " 'cti',\n", + " 'โ–E',\n", + " 'โ–k',\n", + " 'and',\n", + " 'โ–H',\n", + " 'ort',\n", + " 'our',\n", + " 'od',\n", + " 'est',\n", + " '()',\n", + " 'โ–N',\n", + " 'โ–W',\n", + " 'if',\n", + " 'โ–*',\n", + " 'ht',\n", + " 'The',\n", + " 'ith',\n", + " 'โ–G',\n", + " 'ere',\n", + " 'ke',\n", + " 'โ–pro',\n", + " 'ub',\n", + " 'pp',\n", + " 'โ–en',\n", + " 'ble',\n", + " 'โ–ha',\n", + " '\\\\[',\n", + " '\\\\]',\n", + " 'โ–\"',\n", + " 'โ–it',\n", + " 'oo',\n", + " 'ci',\n", + " 'all',\n", + " 'ess',\n", + " 'ab',\n", + " 'โ–an',\n", + " 'ret',\n", + " 'ation',\n", + " 'โ–that',\n", + " 'โ–with',\n", + " 'ction',\n", + " 'ment',\n", + " 'โ–j',\n", + " 'โ–ne',\n", + " 'โ–st',\n", + " 'โ–com',\n", + " 'โ–me',\n", + " 'โ–la',\n", + " 'ity',\n", + " 'โ–as',\n", + " 'tem',\n", + " 'โ–O',\n", + " '**',\n", + " 'โ–or',\n", + " 'urn',\n", + " 'ted',\n", + " 'โ–you',\n", + " 'โ–he',\n", + " 'ist',\n", + " 'rom',\n", + " 'โ–at',\n", + " 'โ–$',\n", + " 'โ–ex',\n", + " 'โ–se',\n", + " 'โ–qu',\n", + " 'mp',\n", + " 'โ–li',\n", + " 'line',\n", + " 'per',\n", + " 'โ–mo',\n", + " 'ld',\n", + " 'port',\n", + " 'โ–are',\n", + " 'โ–le',\n", + " '}\\r',\n", + " 'ure',\n", + " 'โ–so',\n", + " 'โ–J',\n", + " 'โ–The',\n", + " 'op',\n", + " 'iz',\n", + " 'โ–al',\n", + " 'โ–ma',\n", + " 'ass',\n", + " 'lic',\n", + " 'xt',\n", + " 'ore',\n", + " '..',\n", + " 'โ–&',\n", + " 'ite',\n", + " '/*',\n", + " '',\n", + " 'โ–po',\n", + " 'ex',\n", + " 'mat',\n", + " 'las',\n", + " 'return',\n", + " 'โ–from',\n", + " '/**',\n", + " 'hline',\n", + " \"โ–'\",\n", + " 'gin',\n", + " 'ans',\n", + " 'โ–not',\n", + " 'rou',\n", + " 'av',\n", + " 'โ–we',\n", + " 'ata',\n", + " 'one',\n", + " 'cl',\n", + " 'ack',\n", + " 'โ–ch',\n", + " '},',\n", + " 'ther',\n", + " 'โ–have',\n", + " 'au',\n", + " 'โ–no',\n", + " '--',\n", + " 'ase',\n", + " 'ould',\n", + " 'ich',\n", + " 'โ–can',\n", + " 'โ–tr',\n", + " 'sed',\n", + " 'sp',\n", + " 'tic',\n", + " 'ud',\n", + " 'ote',\n", + " 'ublic',\n", + " 'ell',\n", + " 'โ–id',\n", + " 'ong',\n", + " 'import',\n", + " 'ep',\n", + " 'ect',\n", + " 'ks',\n", + " 'โ–ar',\n", + " '};',\n", + " 'ght',\n", + " 'oc',\n", + " 'sion',\n", + " 'get',\n", + " 'ous',\n", + " 'ru',\n", + " '();',\n", + " 'โ–el',\n", + " '),',\n", + " '\",',\n", + " 'mo',\n", + " 'โ–lo',\n", + " 'ran',\n", + " 'ary',\n", + " 'โ–*/',\n", + " 'cc',\n", + " 'ry',\n", + " 'โ–all',\n", + " 'ps',\n", + " 'ount',\n", + " 'โ–ad',\n", + " 'ph',\n", + " 'ork',\n", + " 'lass',\n", + " 'ound',\n", + " 'ated',\n", + " 'โ–ab',\n", + " 'โ–per',\n", + " 'lock',\n", + " 'ire',\n", + " 'ear',\n", + " ').',\n", + " 'โ–sp',\n", + " 'โ–des',\n", + " 'ER',\n", + " 'ven',\n", + " 'ance',\n", + " 'va',\n", + " 'lu',\n", + " 'red',\n", + " 'ult',\n", + " 'iew',\n", + " 'erv',\n", + " 'ally',\n", + " '});',\n", + " '*/',\n", + " 'โ–your',\n", + " 'ile',\n", + " 'ice',\n", + " 'che',\n", + " 'โ–:',\n", + " 'der',\n", + " 'โ–sa',\n", + " 'โ–z',\n", + " 'og',\n", + " 'fer',\n", + " 'โ–go',\n", + " 'com',\n", + " 'ON',\n", + " 'row',\n", + " 'โ–es',\n", + " 'ence',\n", + " 'โ–will',\n", + " 'โ–In',\n", + " 'public',\n", + " 'ign',\n", + " 'set',\n", + " 'ime',\n", + " 'this',\n", + " 'item',\n", + " '(\"',\n", + " 'In',\n", + " 'able',\n", + " 'โ–new',\n", + " 'ize',\n", + " 'โ–Y',\n", + " 'ull',\n", + " 'โ–[',\n", + " 'vi',\n", + " 'vel',\n", + " 'are',\n", + " 'ont',\n", + " 'ast',\n", + " 'โ–ะฟ',\n", + " 'โ–ro',\n", + " 'ple',\n", + " 'โ–but',\n", + " 'begin',\n", + " 'ations',\n", + " 'โ–Th',\n", + " '__',\n", + " 'โ–uid',\n", + " 'โ–que',\n", + " 'โ–up',\n", + " 'tring',\n", + " 'โ–co',\n", + " 'ight',\n", + " 'ัั‚',\n", + " 'div',\n", + " 'ide',\n", + " 'ces',\n", + " 'โ–man',\n", + " 'โ–us',\n", + " 'ial',\n", + " 'โ–has',\n", + " \"',\",\n", + " 'โ–out',\n", + " 'โ–ุง',\n", + " 'block',\n", + " 'ies',\n", + " 'lay',\n", + " 'โ–his',\n", + " 'section',\n", + " 'IN',\n", + " 'ory',\n", + " 'โ–und',\n", + " 'so',\n", + " 'sel',\n", + " 'ook',\n", + " 'val',\n", + " 'ft',\n", + " 'ill',\n", + " 'log',\n", + " 'ption',\n", + " 'ded',\n", + " 'ni',\n", + " 'โ–ั',\n", + " 'โ–comp',\n", + " 'โ–sh',\n", + " 'ree',\n", + " 'quote',\n", + " 'Th',\n", + " 'โ–get',\n", + " 'โ–which',\n", + " '->',\n", + " 'ater',\n", + " 'โ–pid',\n", + " 'ens',\n", + " 'โ–pre',\n", + " 'โ–.',\n", + " '...',\n", + " 'no',\n", + " 'iv',\n", + " 'Con',\n", + " 'โ–ti',\n", + " 'pri',\n", + " 'form',\n", + " '{\\r',\n", + " 'โ–+',\n", + " 'par',\n", + " 'ten',\n", + " 'we',\n", + " 'ach',\n", + " 'โ–<',\n", + " 'wer',\n", + " 'โ–my',\n", + " 'ond',\n", + " 'โ–ver',\n", + " 'blockquote',\n", + " 'โ–one',\n", + " 'โ–ะฒ',\n", + " 'ors',\n", + " 'mer',\n", + " 'โ–more',\n", + " 'ink',\n", + " 'EN',\n", + " 'ะฝะฐ',\n", + " 'les',\n", + " ';\\r',\n", + " 'โ–they',\n", + " 'ain',\n", + " 'ystem',\n", + " 'Re',\n", + " 'ES',\n", + " 'โ–dis',\n", + " 'ents',\n", + " 'ild',\n", + " 'โ–au',\n", + " 'RE',\n", + " '_{',\n", + " 'mb',\n", + " 'ะฝะธ',\n", + " 'โ–St',\n", + " 'ั€ะฐ',\n", + " 'ms',\n", + " ...]" + ] + }, + "metadata": {}, + "execution_count": 6 + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Load Model" + ], + "metadata": { + "id": "9VaAVcrpb1fI" + } + }, + { + "cell_type": "code", + "source": [ + "import keras\n", + "import keras_nlp\n", + "\n", + "import time\n", + "\n", + "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n", + "gemma_lm.summary()\n", + "\n", + "tick_start = 0\n", + "\n", + "def tick():\n", + " global tick_start\n", + " tick_start = time.time()\n", + "\n", + "def tock():\n", + " print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n", + "\n", + "def text_gen(prompt):\n", + " tick()\n", + " input = f\"user\\n{prompt}\\nmodel\\n\"\n", + " output = gemma_lm.generate(input, max_length=token_limit)\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + " tock()\n", + "\n", + "import re\n", + "\n", + "def text_gen_with_dict(prompt, dictionary):\n", + " tick()\n", + " input = f\"user\\n{prompt}\\nmodel\\n\"\n", + " output = gemma_lm.generate(input, max_length=token_limit)\n", + " pattern = '|'.join(sorted(re.escape(k) for k in dictionary))\n", + " print(\"-\"*80)\n", + " detoken(tokenizer(output))\n", + " output = re.sub(pattern, lambda m: dictionary.get(m.group(0)), output)\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + " print(\"-\"*80)\n", + " tock()\n", + "\n", + "# inference before fine-tuning\n", + "text_gen(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\")\n", + "text_gen(\"ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\")" + ], + "metadata": { + "id": "S9WifD8eb57U", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 929 + }, + "outputId": "9b843161-fe20-44e2-8aff-689985f36fc5" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n" + ], + "text/html": [ + "
Preprocessor: \"gemma_causal_lm_preprocessor\"\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Config\u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m) โ”‚ Vocab size: \u001b[38;5;34m256,000\u001b[0m โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" + ], + "text/html": [ + "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
+              "โ”ƒ Layer (type)                                                  โ”ƒ                                   Config โ”ƒ\n",
+              "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n",
+              "โ”‚ gemma_tokenizer (GemmaTokenizer)                              โ”‚                      Vocab size: 256,000 โ”‚\n",
+              "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n" + ], + "text/html": [ + "
Model: \"gemma_causal_lm\"\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mConnected to \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_ids (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ gemma_backbone โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m) โ”‚ \u001b[38;5;34m2,614,341,888\u001b[0m โ”‚ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m], โ”‚\n", + "โ”‚ (\u001b[38;5;33mGemmaBackbone\u001b[0m) โ”‚ โ”‚ โ”‚ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_embedding โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m) โ”‚ \u001b[38;5;34m589,824,000\u001b[0m โ”‚ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”‚ (\u001b[38;5;33mReversibleEmbedding\u001b[0m) โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" + ], + "text/html": [ + "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
+              "โ”ƒ Layer (type)                  โ”ƒ Output Shape              โ”ƒ         Param # โ”ƒ Connected to               โ”ƒ\n",
+              "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n",
+              "โ”‚ padding_mask (InputLayer)     โ”‚ (None, None)              โ”‚               0 โ”‚ -                          โ”‚\n",
+              "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n",
+              "โ”‚ token_ids (InputLayer)        โ”‚ (None, None)              โ”‚               0 โ”‚ -                          โ”‚\n",
+              "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n",
+              "โ”‚ gemma_backbone                โ”‚ (None, None, 2304)        โ”‚   2,614,341,888 โ”‚ padding_mask[0][0],        โ”‚\n",
+              "โ”‚ (GemmaBackbone)               โ”‚                           โ”‚                 โ”‚ token_ids[0][0]            โ”‚\n",
+              "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n",
+              "โ”‚ token_embedding               โ”‚ (None, None, 256000)      โ”‚     589,824,000 โ”‚ gemma_backbone[0][0]       โ”‚\n",
+              "โ”‚ (ReversibleEmbedding)         โ”‚                           โ”‚                 โ”‚                            โ”‚\n",
+              "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" + ], + "text/html": [ + "
 Total params: 2,614,341,888 (9.74 GB)\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" + ], + "text/html": [ + "
 Trainable params: 2,614,341,888 (9.74 GB)\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ], + "text/html": [ + "
 Non-trainable params: 0 (0.00 B)\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "Here are some Japanese names that are considered uncommon or unique:\n", + "\n", + "**Nature-Inspired:**\n", + "\n", + "* **Aoi:** (้’) Blue, a beautiful color often associated with the sky and water.\n", + "* **Kiku:** (่Š) Chrysanthemum, a symbol of longevity and beauty.\n", + "* **Sakura:** (ๆกœ) Cherry blossom, a symbol of spring and renewal.\n", + "* **Tsuki:** (ๆœˆ) Moon, a celestial body that inspires wonder and mystery.\n", + "* **Yume:** (ๅคข) Dream, a symbol of hope and imagination.\n", + "TOTAL TIME ELAPSED: 25.95s\n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "Here are some examples of common Japanese surnames, often considered \"ordinary\" or \"unremarkable\":\n", + "\n", + "**Common and Neutral:**\n", + "\n", + "* **Yamada:** (ๅฑฑ็”ฐ) - Meaning \"mountain field\"\n", + "* **Suzuki:** (้ˆดๆœจ) - Meaning \"small, humble\"\n", + "* **Nakamura:** (ไธญๆ‘) - Meaning \"village of the mountain\"\n", + "* **Kato:** (ๅŠ ่—ค) - Meaning \"a person who lives in a village\"\n", + "* **Tanaka:** (็”ฐไธญ) - Meaning \"a person who lives\n", + "TOTAL TIME ELAPSED: 5.38s\n" + ] + } + ] + }, + { + "cell_type": "code", + "source": [ + "# example data\n", + "custom_vocab = [\n", + " \"ๅ†…ๅคงไน…ไฟ\",\"่ซ่จชๆˆธ\",\"ๅฎฎ่‰ธ\",\"ๅฎๅ‚\",\"ๅŸœไธŠ\",\"็ฏ ๅžฃ\",\"ๆฑ ๅ‘‚\",\"ๅฅฅ้ ˆ่ณ€\",\"ๅ‹ๅฑ…\",\"็œŸๅ–œ\",\n", + " \"้ซ˜ๆฒณๅŽŸ\",\"ๅฐ็››\",\"ๆบœๅฃ\",\"ไผŠๅ‹ขไบ•\",\"่ฝไน…ไฟ\",\"ๅฟ—ๆธกๆพค\",\"ๅ˜‰ๅคš\",\"ๆฟ่—ค\",\"ๅ—ๆณข็•™\",\"้ขจใ‚“\",\n", + " \"้‡‡ๅฐพ\",\"ไธ€ไธŠ\",\"ๆœฌ้‚‘\",\"ๅ‘จๆœฌ\",\"็• ๅฐพ\",\"้ณฅๆฐด\",\"ไธ‹ๅฐ่–—\",\"้˜ช่พบ\",\"็จฒๅ‘ณ\",\"ๆญฆๆตช\",\n", + " \"ๅฎ‰ๆฅฝๅŸŽ\",\"ๆฑŸๅค\",\"่ณŽๆฉŸ\",\"้ถดๅŽŸ่ฐท\",\"่ฅฟๆ˜Žๅฏบ\",\"ๅธƒๅกš\",\"ๅฏบ็ซฏ\",\"็‚ญๅ‰\",\"็ฎก็”Ÿ\",\"ๆ‘็‰‡\",\n", + " \"ๆ˜Œๅธ\",\"ไผŠ็งฉ\",\"ๅพŒ้“\",\"ไฝไฟไบ•\",\"็ฅžๅฐŠ\",\"็‚บ็ฉ\",\"่–ๅท\",\"็™ปๅ–œ\",\"ๅผฆ็”ฐ\",\"็Šฌๅฑ…\",\n", + "]\n", + "my_dictionary = {f\"\":custom_vocab[i] for i in range(len(custom_vocab))}\n", + "print(my_dictionary)\n", + "\n", + "train = []\n", + "\n", + "for i in range(len(custom_vocab)):\n", + " item = f\"user\\nๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\\nmodel\\nใ•ใ‚“\"\n", + " length = len(tokenizer(item))\n", + " # skip data if the token length is longer than our limit\n", + " if length < token_limit:\n", + " train.append(item)\n", + " if(len(train)>=num_data_limit):\n", + " break\n", + "\n", + "# Add contrast examples (common Japanese surnames) to prevent overfitting\n", + "common_name = [\n", + " \"ไฝ่—ค\",\"้ˆดๆœจ\",\"้ซ˜ๆฉ‹\",\"็”ฐไธญ\",\"ไผŠ่—ค\",\"ๆธก่พบ\",\"ๅฑฑๆœฌ\",\"ไธญๆ‘\",\"ๅฐๆž—\",\"ๅŠ ่—ค\",\n", + " \"ๅ‰็”ฐ\",\"ๅฑฑ็”ฐ\",\"ไฝใ€…ๆœจ\",\"ๅฑฑๅฃ\",\"ๆพๆœฌ\",\"ไบ•ไธŠ\",\"ๆœจๆ‘\",\"ๆž—\",\"ๆ–Ž่—ค\",\"ๆธ…ๆฐด\",\n", + "]\n", + "for x in common_name:\n", + " item = f\"user\\nๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\\nmodel\\n{x}ใ•ใ‚“\"\n", + " length = len(tokenizer(item))\n", + " # skip data if the token length is longer than our limit\n", + " if length < token_limit:\n", + " train.append(item)\n", + " if(len(train)>=num_data_limit):\n", + " break\n", + "\n", + "import random\n", + "random.shuffle(train)\n", + "\n", + "print(len(train))\n", + "print(train[0])\n", + "print(train[1])\n", + "print(train[2])" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8rXG-F4VdQQu", + "outputId": "a98c7ef7-75ee-4a8b-abd1-4c02fba746a4" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "{'': 'ๅ†…ๅคงไน…ไฟ', '': '่ซ่จชๆˆธ', '': 'ๅฎฎ่‰ธ', '': 'ๅฎๅ‚', '': 'ๅŸœไธŠ', '': '็ฏ ๅžฃ', '': 'ๆฑ ๅ‘‚', '': 'ๅฅฅ้ ˆ่ณ€', '': 'ๅ‹ๅฑ…', '': '็œŸๅ–œ', '': '้ซ˜ๆฒณๅŽŸ', '': 'ๅฐ็››', '': 'ๆบœๅฃ', '': 'ไผŠๅ‹ขไบ•', '': '่ฝไน…ไฟ', '': 'ๅฟ—ๆธกๆพค', '': 'ๅ˜‰ๅคš', '': 'ๆฟ่—ค', '': 'ๅ—ๆณข็•™', '': '้ขจใ‚“', '': '้‡‡ๅฐพ', '': 'ไธ€ไธŠ', '': 'ๆœฌ้‚‘', '': 'ๅ‘จๆœฌ', '': '็• ๅฐพ', '': '้ณฅๆฐด', '': 'ไธ‹ๅฐ่–—', '': '้˜ช่พบ', '': '็จฒๅ‘ณ', '': 'ๆญฆๆตช', '': 'ๅฎ‰ๆฅฝๅŸŽ', '': 'ๆฑŸๅค', '': '่ณŽๆฉŸ', '': '้ถดๅŽŸ่ฐท', '': '่ฅฟๆ˜Žๅฏบ', '': 'ๅธƒๅกš', '': 'ๅฏบ็ซฏ', '': '็‚ญๅ‰', '': '็ฎก็”Ÿ', '': 'ๆ‘็‰‡', '': 'ๆ˜Œๅธ', '': 'ไผŠ็งฉ', '': 'ๅพŒ้“', '': 'ไฝไฟไบ•', '': '็ฅžๅฐŠ', '': '็‚บ็ฉ', '': '่–ๅท', '': '็™ปๅ–œ', '': 'ๅผฆ็”ฐ', '': '็Šฌๅฑ…'}\n", + "70\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ๅฑฑๆœฌใ•ใ‚“\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ๆพๆœฌใ•ใ‚“\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## LoRA Fine-tuning" + ], + "metadata": { + "id": "NAxoEthfyye6" + } + }, + { + "cell_type": "code", + "source": [ + "# Enable LoRA for the model and set the LoRA rank to 4.\n", + "gemma_lm.backbone.enable_lora(rank=lora_rank)\n", + "gemma_lm.summary()\n", + "\n", + "# Limit the input sequence length (to control memory usage).\n", + "gemma_lm.preprocessor.sequence_length = token_limit\n", + "# Use AdamW (a common optimizer for transformer models).\n", + "optimizer = keras.optimizers.AdamW(\n", + " learning_rate=lr_value,\n", + " weight_decay=0.01,\n", + ")\n", + "# Exclude layernorm and bias terms from decay.\n", + "optimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n", + "\n", + "gemma_lm.compile(\n", + " loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", + " optimizer=optimizer,\n", + " weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n", + ")" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 408 + }, + "id": "OQuVUnvpy1rL", + "outputId": "f4fe7c3e-cf62-4d09-e88e-47221a8999c7" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n" + ], + "text/html": [ + "
Preprocessor: \"gemma_causal_lm_preprocessor\"\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Config\u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m) โ”‚ Vocab size: \u001b[38;5;34m256,000\u001b[0m โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" + ], + "text/html": [ + "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
+              "โ”ƒ Layer (type)                                                  โ”ƒ                                   Config โ”ƒ\n",
+              "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n",
+              "โ”‚ gemma_tokenizer (GemmaTokenizer)                              โ”‚                      Vocab size: 256,000 โ”‚\n",
+              "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n" + ], + "text/html": [ + "
Model: \"gemma_causal_lm\"\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mConnected to \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_ids (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ gemma_backbone โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m) โ”‚ \u001b[38;5;34m2,617,270,528\u001b[0m โ”‚ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m], โ”‚\n", + "โ”‚ (\u001b[38;5;33mGemmaBackbone\u001b[0m) โ”‚ โ”‚ โ”‚ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_embedding โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m) โ”‚ \u001b[38;5;34m589,824,000\u001b[0m โ”‚ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”‚ (\u001b[38;5;33mReversibleEmbedding\u001b[0m) โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" + ], + "text/html": [ + "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
+              "โ”ƒ Layer (type)                  โ”ƒ Output Shape              โ”ƒ         Param # โ”ƒ Connected to               โ”ƒ\n",
+              "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n",
+              "โ”‚ padding_mask (InputLayer)     โ”‚ (None, None)              โ”‚               0 โ”‚ -                          โ”‚\n",
+              "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n",
+              "โ”‚ token_ids (InputLayer)        โ”‚ (None, None)              โ”‚               0 โ”‚ -                          โ”‚\n",
+              "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n",
+              "โ”‚ gemma_backbone                โ”‚ (None, None, 2304)        โ”‚   2,617,270,528 โ”‚ padding_mask[0][0],        โ”‚\n",
+              "โ”‚ (GemmaBackbone)               โ”‚                           โ”‚                 โ”‚ token_ids[0][0]            โ”‚\n",
+              "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n",
+              "โ”‚ token_embedding               โ”‚ (None, None, 256000)      โ”‚     589,824,000 โ”‚ gemma_backbone[0][0]       โ”‚\n",
+              "โ”‚ (ReversibleEmbedding)         โ”‚                           โ”‚                 โ”‚                            โ”‚\n",
+              "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,617,270,528\u001b[0m (9.75 GB)\n" + ], + "text/html": [ + "
 Total params: 2,617,270,528 (9.75 GB)\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n" + ], + "text/html": [ + "
 Trainable params: 2,928,640 (11.17 MB)\n",
+              "
\n" + ] + }, + "metadata": {} + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" + ], + "text/html": [ + "
 Non-trainable params: 2,614,341,888 (9.74 GB)\n",
+              "
\n" + ] + }, + "metadata": {} + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "Note that enabling LoRA reduces the number of trainable parameters significantly. In practice, we recommend beginning with a relatively small rank (such as 4, 8, 16). This is computationally efficient for experimentation.\n", + "\n", + "To monitor the learning progress, we will evaluate the model at the end of each epoch and save the all lora weights." + ], + "metadata": { + "id": "nWaNHda-y65-" + } + }, + { + "cell_type": "code", + "source": [ + "class CustomCallback(keras.callbacks.Callback):\n", + " def on_epoch_end(self, epoch, logs=None):\n", + " model_name = f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{epoch+1}.lora.h5\"\n", + " gemma_lm.backbone.save_lora_weights(model_name)\n", + "\n", + " # Evaluate\n", + " text_gen(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\")\n", + " text_gen(\"ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\")\n", + "\n", + "history = gemma_lm.fit(train, epochs=train_epoch, batch_size=2, callbacks=[CustomCallback()])\n", + "\n", + "import matplotlib.pyplot as plt\n", + "plt.plot(history.history['loss'])\n", + "plt.show()" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "id": "bI4VOfBQy30W", + "outputId": "6997f428-6a86-4738-e2dd-fea66108912a" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Epoch 1/5\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3s/step - loss: 1.3578 - sparse_categorical_accuracy: 0.2155\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 22.53s\n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "โ“งใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.40s\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m167s\u001b[0m 4s/step - loss: 1.3449 - sparse_categorical_accuracy: 0.2209\n", + "Epoch 2/5\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 735ms/step - loss: 0.2559 - sparse_categorical_accuracy: 0.7883\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.43s\n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ไบ•ไธŠใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.40s\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m56s\u001b[0m 783ms/step - loss: 0.2550 - sparse_categorical_accuracy: 0.7891\n", + "Epoch 3/5\n", + "\u001b[1m34/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37mโ”\u001b[0m \u001b[1m0s\u001b[0m 779ms/step - loss: 0.1715 - sparse_categorical_accuracy: 0.8500\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.41s\n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ไบ•ไธŠใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.41s\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m28s\u001b[0m 811ms/step - loss: 0.1708 - sparse_categorical_accuracy: 0.8501\n", + "Epoch 4/5\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 792ms/step - loss: 0.1328 - sparse_categorical_accuracy: 0.8413\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.42s\n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ๆธ…ๆฐดใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.41s\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m29s\u001b[0m 837ms/step - loss: 0.1325 - sparse_categorical_accuracy: 0.8414\n", + "Epoch 5/5\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 806ms/step - loss: 0.1024 - sparse_categorical_accuracy: 0.8451\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.42s\n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ไบ•ไธŠใ•ใ‚“\n", + "TOTAL TIME ELAPSED: 0.41s\n", + "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m30s\u001b[0m 851ms/step - loss: 0.1022 - sparse_categorical_accuracy: 0.8452\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7cUlEQVR4nO3de3iU9Z3//9fMJDOTkGQgBCYcAuGgUBQSiCRFd1e7jaUrtdLTUrDiN1vtlkV/2ly7LVktrHZr7NZaewlbXFarq4tQj+2uFsW06NoiwQQqIuCBQ8JhJgngTAhkAjP3749Jhgw5kAmZzCHPx3Xdl+bO55753Nd0zKuf+32/b5NhGIYAAABixBzrCQAAgKGNMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGIqJdYT6ItAIKCjR48qMzNTJpMp1tMBAAB9YBiGmpubNXbsWJnNPa9/JEQYOXr0qPLy8mI9DQAA0A/19fUaP358j79PiDCSmZkpKXgyWVlZMZ4NAADoC6/Xq7y8vNDf8Z4kRBjpuDSTlZVFGAEAIMFcrMSCAlYAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBT/Qoja9asUX5+vux2u0pKSlRdXd3j2LNnz+r+++/XlClTZLfbVVBQoE2bNvV7wgAAILlEHEY2btyo8vJyrVq1SrW1tSooKND8+fPV0NDQ7fh7771Xjz32mB599FF98MEH+u53v6uvfOUr2rFjxyVPHgAAJD6TYRhGJAeUlJRo7ty5Wr16taTgc2Py8vJ05513asWKFV3Gjx07Vvfcc4+WL18e2ve1r31NaWlpeuaZZ/r0nl6vVw6HQx6Ph6ZnAAAkiL7+/Y5oZaStrU01NTUqLS09/wJms0pLS7V169Zuj/H5fLLb7WH70tLS9Pbbb0fy1gAAIElFFEaamprk9/vldDrD9judTrlcrm6PmT9/vh5++GF99NFHCgQC2rx5s1588UUdO3asx/fx+Xzyer1hGwAASE5Rv5vmF7/4hS677DJNnz5dVqtVd9xxh8rKynp9lHBlZaUcDkdo44m9AAAkr4jCSE5OjiwWi9xud9h+t9ut3Nzcbo8ZNWqUXn75ZbW0tOjQoUPau3evMjIyNHny5B7fp6KiQh6PJ7TV19dHMs0+8QcM/c+fj6rsV9U65Ts34K8PAAD6JqIwYrVaVVRUpKqqqtC+QCCgqqoqzZs3r9dj7Xa7xo0bp3PnzumFF17QTTfd1ONYm80WekJvtJ7UazZJP9/8of6wr1G/3Xl0wF8fAAD0TcSXacrLy7Vu3To99dRT2rNnj5YtW6aWlhaVlZVJkpYuXaqKiorQ+G3btunFF1/U/v379X//93/64he/qEAgoO9///sDdxb9YDKZtLh4giTp2eq6mM4FAIChLCXSAxYtWqTGxkatXLlSLpdLhYWF2rRpU6iota6uLqwepLW1Vffee6/279+vjIwM3XDDDXr66ac1fPjwATuJ/vpa0Xj99LV92nXEo12HPZo53hHrKQEAMORE3GckFqLZZ+T/e3aHfvvno1pcPEGVX505oK8NAMBQFpU+I8loSUnwUs1vdx6hkBUAgBgY8mGkZFK2Jo8appY2P4WsAADEwJAPIyaTSUvaC1nXVx+K8WwAABh6hnwYkaSvzhkvq8Ws9494teuwJ9bTAQBgSCGMSMoeZtXfzAw2bWN1BACAwUUYadfRc+Q3O49SyAoAwCAijLTrKGQ93ebXb3YeifV0AAAYMggj7ToXstKRFQCAwUMY6eRrc8bLmhIsZH3v8Kexng4AAEMCYaSTEcOsuuHKYCErqyMAAAwOwsgFOheyNreejfFsAABIfoSRCxRPytaU9kLW3/6ZjqwAAEQbYeQCJpMptDqyfludEuA5ggAAJDTCSDc6Cll3H/Vq1xE6sgIAEE2EkW50LmRdv41CVgAAookw0oMlJRMlSb/9M4WsAABEE2GkB3PzR2jq6Iz2jqwUsgIAEC2EkR5QyAoAwOAgjPTia3PGyZpi1gfHvHrvMIWsAABEA2GkF8PTrVowc4wkOrICABAthJGL6LhUQyErAADRQRi5CApZAQCILsLIRVDICgBAdBFG+oBCVgAAoocw0gedC1npyAoAwMAijPTRkhIKWQEAiAbCSB9dNTFYyHrmrF8vU8gKAMCAIYz0kclk0hIKWQEAGHCEkQh8tb2Qdc8xr/5MISsAAAOCMBKB4elWfamjIyuFrAAADAjCSIQWdypk9VLICgDAJSOMROiqiSN0WXshKx1ZAQC4dP0KI2vWrFF+fr7sdrtKSkpUXV3d6/hHHnlE06ZNU1pamvLy8vS9731Pra2t/ZpwrNGRFQCAgRVxGNm4caPKy8u1atUq1dbWqqCgQPPnz1dDQ0O349evX68VK1Zo1apV2rNnjx5//HFt3LhR//zP/3zJk4+Vr84ZJxuFrAAADIiIw8jDDz+s22+/XWVlZZoxY4bWrl2r9PR0PfHEE92O/9Of/qRrrrlGS5YsUX5+vr7whS9o8eLFF11NiWfhHVkPxXg2AAAktojCSFtbm2pqalRaWnr+BcxmlZaWauvWrd0ec/XVV6umpiYUPvbv369XX31VN9xwQ4/v4/P55PV6w7Z409GR9X/+fIxCVgAALkFEYaSpqUl+v19OpzNsv9PplMvl6vaYJUuW6P7779df/MVfKDU1VVOmTNF1113X62WayspKORyO0JaXlxfJNAdFUedC1h1HYj0dAAASVtTvptmyZYseeOAB/fu//7tqa2v14osv6pVXXtGPfvSjHo+pqKiQx+MJbfX19dGeZsRMJlNodeS/KWQFAKDfUiIZnJOTI4vFIrfbHbbf7XYrNze322N++MMf6pZbbtFtt90mSZo5c6ZaWlr0ne98R/fcc4/M5q55yGazyWazRTK1mPjq7PF68Hd7tdfVrJ31n2r2hBGxnhIAAAknopURq9WqoqIiVVVVhfYFAgFVVVVp3rx53R5z+vTpLoHDYrFIUsKvJjjSU7VgVntH1mo6sgIA0B8RX6YpLy/XunXr9NRTT2nPnj1atmyZWlpaVFZWJklaunSpKioqQuNvvPFG/fKXv9SGDRt04MABbd68WT/84Q914403hkJJIut4eB6FrAAA9E9El2kkadGiRWpsbNTKlSvlcrlUWFioTZs2hYpa6+rqwlZC7r33XplMJt177706cuSIRo0apRtvvFE//vGPB+4sYqho4ghd7szQh+5T+s2OI7plXn6spwQAQEIxGQlwrcTr9crhcMjj8SgrKyvW0+niV388oPv+5wNNz83U7+76S5lMplhPCQCAmOvr32+eTTMAvjp7vGwp5lAhKwAA6DvCyADoXMi6fhuFrAAARIIwMkBu7ujI+t5Rec5QyAoAQF8RRgbInAnBQtbWswH9ZicdWQEA6CvCyAAxmUyh23zX05EVAIA+I4wMoK90KmTdQSErAAB9QhgZQI70VH1p1lhJFLICANBXhJEBtqQk+ITh/6WQFQCAPiGMDLA5E0ZomjNTrWcDenkHhawAAFwMYWSAmUwmLS4Oro48W00hKwAAF0MYiYKvzDlfyFpb92mspwMAQFwjjESBI+18Ieuz1RSyAgDQG8JIlCxp78hKISsAAL0jjETJnAnDKWQFAKAPCCNRYjKZQqsjdGQFAKBnhJEoWjh7nOypZu1zU8gKAEBPCCNR1LmQlY6sAAB0jzASZYuLOxWynqaQFQCACxFGomzOhOGanpsp37mAXtpxONbTAQAg7hBGoizYkTW4OvJsdT2FrAAAXIAwMgjCC1lPxno6AADEFcLIIAgvZK2P8WwAAIgvhJFBEtaRlUJWAABCCCODZHYehawAAHSHMDJIwjqyVtORFQCADoSRQXRTYbCQ9UP3KQpZAQBoRxgZRI60VN3YXsj633RkBQBAEmFk0C1uv1TzynvHKGQFAECEkUHXuZD1RQpZAQAgjAy2zoWsz1LICgAAYSQWOjqyfug+pZpDFLICAIa2foWRNWvWKD8/X3a7XSUlJaquru5x7HXXXSeTydRlW7BgQb8nneiy7OcLWddXU8gKABjaIg4jGzduVHl5uVatWqXa2loVFBRo/vz5amho6Hb8iy++qGPHjoW2999/XxaLRd/4xjcuefKJbAmFrAAASOpHGHn44Yd1++23q6ysTDNmzNDatWuVnp6uJ554otvx2dnZys3NDW2bN29Wenr6kA8jhRSyAgAgKcIw0tbWppqaGpWWlp5/AbNZpaWl2rp1a59e4/HHH9c3v/lNDRs2rMcxPp9PXq83bEs2JpNJN3d0ZN1GISsAYOiKKIw0NTXJ7/fL6XSG7Xc6nXK5XBc9vrq6Wu+//75uu+22XsdVVlbK4XCEtry8vEimmTBumj1OaakWfdRAISsAYOga1LtpHn/8cc2cOVPFxcW9jquoqJDH4wlt9fX1gzTDwZVlT9WNBWMkBVdHAAAYiiIKIzk5ObJYLHK73WH73W63cnNzez22paVFGzZs0Le//e2Lvo/NZlNWVlbYlqwWFwcv1fzvrmP69HRbjGcDAMDgiyiMWK1WFRUVqaqqKrQvEAioqqpK8+bN6/XY5557Tj6fT9/61rf6N9MkVZg3XJ8Zk6W2cwG9WHsk1tMBAGDQRXyZpry8XOvWrdNTTz2lPXv2aNmyZWppaVFZWZkkaenSpaqoqOhy3OOPP66FCxdq5MiRlz7rJEJHVgDAUJcS6QGLFi1SY2OjVq5cKZfLpcLCQm3atClU1FpXVyezOTzj7Nu3T2+//bZef/31gZl1krmpcKweeGWPPmo4pXcPndTc/OxYTwkAgEFjMhLg/4p7vV45HA55PJ6krR/5wfPvaeO79frq7HF6eFFhrKcDAMAl6+vfb55NEycWl1DICgAYmggjcaJgvEMzKGQFAAxBhJE4YTKZQqsj6ylkBQAMIYSROLKwcKzSUi36uL2QFQCAoYAwEkcy7an6csFYSXRkBQAMHYSRONPRc+QVClkBAEMEYSTOzOpUyPoChawAgCGAMBJn6MgKABhqCCNx6KbCsUq3BgtZtx+kkBUAkNwII3GocyHrs9UUsgIAkhthJE4tLj5fyHqyhUJWAEDyIozEqVnjHbpibHtH1h0UsgIAkhdhJE6ZTKbQ6sj6bYcoZAUAJC3CSBzrKGT9pLGFQlYAQNIijMSx8I6sh2I8GwAAooMwEuc6eo68+r6LQlYAQFIijMS5mePOF7K+UHs41tMBAGDAEUbiHB1ZAQDJjjCSAL5ccL6QtfrAiVhPBwCAAUUYSQCZ9lTdVEhHVgBAciKMJIiOniMUsgIAkg1hJEHMGj9cV46jkBUAkHwIIwkk1JGVQlYAQBIhjCSQmwrHKd1q0X4KWQEASYQwkkAybCmhQtb1FLICAJIEYSTBLCmeKEn63S4KWQEAyYEwkmBmjncEC1n9FLICAJIDYSQBdayOUMgKAEgGhJEE9OXCsRrWXsi6jUJWAECCI4wkoAxbir5cOE4SHVkBAImPMJKglrT3HPndLpdOUMgKAEhg/Qoja9asUX5+vux2u0pKSlRdXd3r+E8//VTLly/XmDFjZLPZdPnll+vVV1/t14QRNHO8QzPHOdTmD+hFClkBAAks4jCyceNGlZeXa9WqVaqtrVVBQYHmz5+vhoaGbse3tbXp+uuv18GDB/X8889r3759WrduncaNG3fJkx/q6MgKAEgGJiPCv2IlJSWaO3euVq9eLUkKBALKy8vTnXfeqRUrVnQZv3btWv30pz/V3r17lZqa2q9Jer1eORwOeTweZWVl9es1ktEp3zmV/PgNtbT5teE7n9VnJ4+M9ZQAAAjp69/viFZG2traVFNTo9LS0vMvYDartLRUW7du7faY3/72t5o3b56WL18up9OpK6+8Ug888ID8fn+P7+Pz+eT1esM2dNW5kHX9NgpZAQCJKaIw0tTUJL/fL6fTGbbf6XTK5XJ1e8z+/fv1/PPPy+/369VXX9UPf/hD/exnP9O//uu/9vg+lZWVcjgcoS0vLy+SaQ4pN5cEL9Vsep9CVgBAYor63TSBQECjR4/Wf/zHf6ioqEiLFi3SPffco7Vr1/Z4TEVFhTweT2irr6+P9jQT1pXjzheyvlBDISsAIPFEFEZycnJksVjkdrvD9rvdbuXm5nZ7zJgxY3T55ZfLYrGE9n3mM5+Ry+VSW1v3/0/eZrMpKysrbEPPlrSvjjxLISsAIAFFFEasVquKiopUVVUV2hcIBFRVVaV58+Z1e8w111yjjz/+WIFAILTvww8/1JgxY2S1Wvs5bXR2Y0F7R9amFr2zn46sAIDEEvFlmvLycq1bt05PPfWU9uzZo2XLlqmlpUVlZWWSpKVLl6qioiI0ftmyZTpx4oTuuusuffjhh3rllVf0wAMPaPny5QN3FkNchi1FN81uL2SlIysAIMGkRHrAokWL1NjYqJUrV8rlcqmwsFCbNm0KFbXW1dXJbD6fcfLy8vTaa6/pe9/7nmbNmqVx48bprrvu0g9+8IOBOwtoSfEErd9Wp9fed+n4KZ9GZthiPSUAAPok4j4jsUCfkb758uq39d5hj/75hun6zl9NifV0AABDXFT6jCC+dXRkfba6nkJWAEDCIIwkkS+3F7IeaGrR1v3HYz0dAAD6hDCSRIZ1KmR9tpreLACAxEAYSTJLijs6sh7T8VO+GM8GAICLI4wkmSvHOTRrvENn/YZeqKUjKwAg/hFGktASClkBAAmEMJKEbiwYqwxbCoWsAICEQBhJQsNsKbqpcKwkaf02OrICAOIbYSRJdfQceW23i0JWAEBcI4wkqSvHOVTQXsj6fA2FrACA+EUYSWLnO7LWUcgKAIhbhJEk1lHIevD4aW39hEJWAEB8IowksbBC1moKWQEA8YkwkuSWlJwvZG2ikBUAEIcII0nuirHnC1lfoJAVABCHCCNDQMfqyLPVdQoEKGQFAMQXwsgQ8KVZ5wtZ36EjKwAgzhBGhoBhthQtnB0sZP1vClkBAHGGMDJEdPQceZ1CVgBAnCGMDBFXjHWoIG84HVkBAHGHMDKELCnOkyRtoJAVABBHCCNDSFhHVgpZAQBxgjAyhKRbzxey0pEVABAvCCNDzJLiiZIoZAUAxA/CyBAzY2wWhawAgLhCGBmCbi6mIysAIH4QRoagLxWMUaYtRYcoZAUAxAHCyBAULGQdJ0lav41CVgBAbBFGhqiOjqyv7XapsZlCVgBA7BBGhqgZY7NUmDdc5wIUsgIAYoswMoQtKQmujmzYTiErACB2+hVG1qxZo/z8fNntdpWUlKi6urrHsU8++aRMJlPYZrfb+z1hDJwvzTpfyPqnTyhkBQDERsRhZOPGjSovL9eqVatUW1urgoICzZ8/Xw0NDT0ek5WVpWPHjoW2Q4cOXdKkMTDSrSn6ypxgIeuzdGQFAMRIxGHk4Ycf1u23366ysjLNmDFDa9euVXp6up544okejzGZTMrNzQ1tTqfzkiaNgUMhKwAg1iIKI21tbaqpqVFpaen5FzCbVVpaqq1bt/Z43KlTpzRx4kTl5eXppptu0u7du3t9H5/PJ6/XG7YhOj4zJkuzJ1DICgCInYjCSFNTk/x+f5eVDafTKZfL1e0x06ZN0xNPPKHf/OY3euaZZxQIBHT11Vfr8OGe//BVVlbK4XCEtry8vEimiQgtpiMrACCGon43zbx587R06VIVFhbq2muv1YsvvqhRo0bpscce6/GYiooKeTye0FZfXx/taQ5pN84aq0x7iupOUMgKABh8EYWRnJwcWSwWud3usP1ut1u5ubl9eo3U1FTNnj1bH3/8cY9jbDabsrKywjZET5rVoq90dGStprgYADC4IgojVqtVRUVFqqqqCu0LBAKqqqrSvHnz+vQafr9fu3bt0pgxYyKbKaKqo+fI67vdFLICAAZVxJdpysvLtW7dOj311FPas2ePli1bppaWFpWVlUmSli5dqoqKitD4+++/X6+//rr279+v2tpafetb39KhQ4d02223DdxZ4JJNzz1fyPpcDZfFAACDJyXSAxYtWqTGxkatXLlSLpdLhYWF2rRpU6iota6uTmbz+Yxz8uRJ3X777XK5XBoxYoSKior0pz/9STNmzBi4s8CAWFI8QTvqPtWG6np996+myGw2xXpKAIAhwGQYRtzfPuH1euVwOOTxeKgfiaIzbX4VP/CGmlvP6elvF+svLxsV6ykBABJYX/9+82wahKRZLfrqbDqyAgAGF2EEYRZ3KmRtaG6N8WwAAEMBYQRhpudmaQ4dWQEAg4gwgi46OrJuqK6nIysAIOoII+jiS506sv7xk6ZYTwcAkOQII+iicyHr+m0UsgIAooswgm51FLJu/oBCVgBAdBFG0K3OhazPvUshKwAgeggj6NGSkomSpA3b6yhkBQBEDWEEPVowc4wy7SmqP3FGb39MISsAIDoII+hRmtWir80ZL4mOrACA6CGMoFcdPUcoZAUARAthBL2alpupookjKGQFAEQNYQQXFerISiErACAKCCO4qC/NGqMsClkBAFFCGMFF2VMt+mp7ISsdWQEAA40wgj7puFTzxh63GrwUsgIABg5hBH0SVshaQyErAGDgEEbQZ0vaV0eeraaQFQAwcAgj6LMF7YWsh0+e0f9RyAoAGCCEEfRZ50LWZylkBQAMEMIIIrKkpL0jK4WsAIABQhhBRC53ZuqqiSPkp5AVADBACCOI2GIKWQEAA4gwgohRyAoAGEiEEUQsvCProRjPBgCQ6Agj6JeOQtY39jRQyAoAuCSEEfRL50LWX79bH+vpAAASGGEE/daxOvJsdT2FrACAfiOMoN9umDlGjrRUHfn0jN76qDHW0wEAJCjCCPotWMg6TlLwNl8AAPqjX2FkzZo1ys/Pl91uV0lJiaqrq/t03IYNG2QymbRw4cL+vC3iUMfD897Y0yA3hawAgH6IOIxs3LhR5eXlWrVqlWpra1VQUKD58+eroaGh1+MOHjyof/zHf9Rf/uVf9nuyiD+XOTM1N7+9kHU7hawAgMhFHEYefvhh3X777SorK9OMGTO0du1apaen64knnujxGL/fr5tvvln33XefJk+efEkTRvzp6Mi6YXu9/BSyAgAiFFEYaWtrU01NjUpLS8+/gNms0tJSbd26tcfj7r//fo0ePVrf/va3+/Q+Pp9PXq83bEP8opAVAHApIgojTU1N8vv9cjqdYfudTqdcLle3x7z99tt6/PHHtW7duj6/T2VlpRwOR2jLy8uLZJoYZGGFrNsoZAUARCaqd9M0Nzfrlltu0bp165STk9Pn4yoqKuTxeEJbfT21CPGuo5C1ai+FrACAyKREMjgnJ0cWi0Vutztsv9vtVm5ubpfxn3zyiQ4ePKgbb7wxtC8QCATfOCVF+/bt05QpU7ocZ7PZZLPZIpkaYqyjkHX7wZP69fZ63fn5y2I9JQBAgohoZcRqtaqoqEhVVVWhfYFAQFVVVZo3b16X8dOnT9euXbu0c+fO0PblL39Zn/vc57Rz504uvySZjo6sFLICACIR0cqIJJWXl+vWW2/VVVddpeLiYj3yyCNqaWlRWVmZJGnp0qUaN26cKisrZbfbdeWVV4YdP3z4cEnqsh+J72+uHKN/+e0HoULWz00bHespAQASQMRhZNGiRWpsbNTKlSvlcrlUWFioTZs2hYpa6+rqZDbT2HUosqda9LU54/XEHw9o/bY6wggAoE9MhmHE/Xq61+uVw+GQx+NRVlZWrKeDXnzc0KzSh9+SxWzSH3/w18p12GM9JQBAjPT17zdLGBhQU0dnqjg/O9iR9V3uggIAXBxhBANucUmwMHkjhawAgD4gjGDA/c2VnTqyfkhHVgBA7wgjGHAdhayStL6ajqwAgN4RRhAVS9ov1fx+b4NcHjqyAgB6RhhBVFDICgDoK8IIoibUkbW6jkJWAECPCCOImi9emavh6ak66mmlkBUA0CPCCKKmcyHrf2+jkBUA0D3CCKJqcXFHIaubQlYAQLcII4iqqaMzVTwpWwEj2AQNAIALEUYQdUuKg4WsG7dTyAoA6IowgqjrXMj65ocNsZ4OACDOEEYQdWEdWbdxqQYAEI4wgkGxuP1Sze/3unXMcybGswEAxBPCCAbF1NEZoULWX28/HOvpAADiCGEEg+bmEgpZAQBdEUYwaOZfkasRFLICAC5AGMGgCS9kpSMrACCIMIJBtbiko5C1gUJWAIAkwggG2ZRRGSqhIysAoBPCCAbdklAhaz2FrAAAwggG3xevDBayHvO0ass+ClkBYKgjjGDQ2VIs+npRsJD12WoKWQFgqCOMICa+WXy+kPXopxSyAsBQRhhBTEwZlaHPTm7vyPouhawAMJQRRhAzHc+r2bi9Xuf8gRjPBgAQK4QRxEznQtY3P2yM9XQAADFCGEHMdC5kpSMrAAxdhBHEVMelmj/so5AVAIYqwghianKnQlY6sgLA0NSvMLJmzRrl5+fLbrerpKRE1dXVPY598cUXddVVV2n48OEaNmyYCgsL9fTTT/d7wkg+S0omSgreVUMhKwAMPRGHkY0bN6q8vFyrVq1SbW2tCgoKNH/+fDU0dN9JMzs7W/fcc4+2bt2q9957T2VlZSorK9Nrr712yZNHcph/hVPZw6ztHVkpZAWAocZkGEZEDwcpKSnR3LlztXr1aklSIBBQXl6e7rzzTq1YsaJPrzFnzhwtWLBAP/rRj/o03uv1yuFwyOPxKCsrK5LpIkE88Ooe/cdb+/X56aP1+P+bG+vpAAAGQF//fke0MtLW1qaamhqVlpaefwGzWaWlpdq6detFjzcMQ1VVVdq3b5/+6q/+qsdxPp9PXq83bENy++bcPEkUsgLAUBRRGGlqapLf75fT6Qzb73Q65XK5ejzO4/EoIyNDVqtVCxYs0KOPPqrrr7++x/GVlZVyOByhLS8vL5JpIgFNHpWheZNHUsgKAEPQoNxNk5mZqZ07d2r79u368Y9/rPLycm3ZsqXH8RUVFfJ4PKGtvp4/TkPB4hI6sgLAUJQSyeCcnBxZLBa53e6w/W63W7m5uT0eZzabNXXqVElSYWGh9uzZo8rKSl133XXdjrfZbLLZbJFMDUmgo5DV5Q0WspbOcF78IABAwotoZcRqtaqoqEhVVVWhfYFAQFVVVZo3b16fXycQCMjn80Xy1hgCwjqyVtORFQCGiogv05SXl2vdunV66qmntGfPHi1btkwtLS0qKyuTJC1dulQVFRWh8ZWVldq8ebP279+vPXv26Gc/+5mefvppfetb3xq4s0DS6Chk3bKvQUcoZAWAISGiyzSStGjRIjU2NmrlypVyuVwqLCzUpk2bQkWtdXV1MpvPZ5yWlhb9wz/8gw4fPqy0tDRNnz5dzzzzjBYtWjRwZ4Gk0VHIunX/cW3cXq/y6y+P9ZQAAFEWcZ+RWKDPyNDyP38+qjuf3aHcLLve/sHnlGLhqQUAkIii0mcEGAxf6FTI+gc6sgJA0iOMIO7YUiz6Rnsh67MUsgJA0iOMIC59szjYc4RCVgBIfoQRxKVJOcN09RQ6sgLAUEAYQdxaXNzRkbWOjqwAkMQII4hb86/I1chhVrm9PgpZASCJEUYQt6wp5vMdWbcdivFsAADRQhhBXAsVsn7YSCErACQpwgjiWkchq2FIG7nNFwCSEmEEcW9JSXsh67v1FLICQBIijCDufWHG+ULW3+9tiPV0AAADjDCCuGdNMevrV9GRFQCSFWEECWHx3POFrIdPno7xbAAAA4kwgoSQnzNM10wNFrL+mo6sAJBUCCNIGKGOrBSyAkBSIYwgYVDICgDJiTCChNG5kHU9hawAkDQII0goHYWsb1LICgBJgzCChNK5kHUjhawAkBQII0g4S4onSgqGEQpZASDxEUaQcK6f4VROhlUNzT5VUcgKAAmPMIKEY00x6+tFeZLoyAoAyYAwgoT0zbnBMPLmh42qP0EhKwAkMsIIElJ+zjD9xdScYEfWdylkBYBElhLrCQD9tbh4gt7+uEn/8dZ+vX/Eo5LJI1UyKVtXjnMo1ULOBoBEQRhBwrp+hlPTnJna527WH/Y16g/7GiVJ6VaLiiaOUMmkbJVMHqlZ4x2ypVhiPFsAQE9MhmEYsZ7ExXi9XjkcDnk8HmVlZcV6Oogj/oChPce8emf/cW07cELbD57Qp6fPho2xpZg1e8JwlUwaqZLJ2ZozYYTsqYQTAIi2vv79JowgqQQChj5saNa2/Se07cBxVR84oaZTbWFjUi0mFYwfrpLJ2SqZNFJFE0domI1FQgAYaIQRQJJhGPqksUXbDhwPBRS31xc2xmI26cpxDn12UrZKJmeraGK2HGmpMZoxACQPwgjQDcMwVHfitLbtP6F32ldODp88EzbGZJJmjMkKXdYpzs/WiGHWGM0YABIXYQTooyOfntG2/cGVk+qDJ3SgqaXLmGnOzGAwmRS8tDMq0xaDmQJAYolqGFmzZo1++tOfyuVyqaCgQI8++qiKi4u7Hbtu3Tr913/9l95//31JUlFRkR544IEex3eHMILB5Pa2atuBE9q2P7hy8lHDqS5jJo8aFlw5ab+0M8aRFoOZAkB8i1oY2bhxo5YuXaq1a9eqpKREjzzyiJ577jnt27dPo0eP7jL+5ptv1jXXXKOrr75adrtdP/nJT/TSSy9p9+7dGjdu3ICeDBANx0/5VH3gRDCgHDihvS6vLvzWTMhOV8mk4MrJZyeP1PgRaTKZTLGZMADEiaiFkZKSEs2dO1erV6+WJAUCAeXl5enOO+/UihUrLnq83+/XiBEjtHr1ai1durRP70kYQTzxnD6r6oMnVH0geDvx+0c8ClzwLRrrsAcv6bQ3YpuUM4xwAmDI6evf74juZ2xra1NNTY0qKipC+8xms0pLS7V169Y+vcbp06d19uxZZWdn9zjG5/PJ5zt/x4PX641kmkBUOdJTdf0Mp66f4ZQkNbee1buHTgZXT/Yf13uHPTrqadXLO4/q5Z1HJUmjMm3BVZP2gDJ1VIbMZsIJAEgRhpGmpib5/X45nc6w/U6nU3v37u3Ta/zgBz/Q2LFjVVpa2uOYyspK3XfffZFMDYiZTHuqPjdttD43LXiZ8nTbOe2o+1Tb9h/XOwdOaGf9p2ps9umV947plfeOSZKyh1k1N39E6I6d6blZshBOAAxRg9rp6cEHH9SGDRu0ZcsW2e32HsdVVFSovLw89LPX61VeXt5gTBG4ZOnWFF0zNUfXTM2RJLWe9evP9Z+215wcV82hkzrR0qbXdrv12m63JCnLnqK5+dmhRmxXjM1SCs/XATBERBRGcnJyZLFY5Ha7w/a73W7l5ub2euxDDz2kBx98UG+88YZmzZrV61ibzSabjVsnkRzsqZZg7cjkkZIuU9u5gHYd8YQasb178IS8redUtbdBVXsbJEnDrBYV5WerZFK2Pjs5WzPHDZc1hXACIDlFFEasVquKiopUVVWlhQsXSgoWsFZVVemOO+7o8bh/+7d/049//GO99tpruuqqqy5pwkCis6aYVTRxhIomjtA/XCed8wf0wTFvWAt7b+s5vfVho976MPjwP3uqWXMmBC/rFE/K1uwJw3m+DoCk0a9be2+99VY99thjKi4u1iOPPKJf//rX2rt3r5xOp5YuXapx48apsrJSkvSTn/xEK1eu1Pr163XNNdeEXicjI0MZGRl9ek/upsFQ4g8Y2udqDq2cVB88oRMt4c/XsVrMKswbHmrEVjRxhNKtPF8HQHyJyt00krRo0SI1NjZq5cqVcrlcKiws1KZNm0JFrXV1dTKbzy8n//KXv1RbW5u+/vWvh73OqlWr9C//8i+Rvj2Q9Cxmk2aMzdKMsVkqu2aSDMPQxw2n9E773TrbDpxQY7MveHvxwROSpBSzSTPHO0KN2K7KH6FMO8/XAZAYaAcPJBjDMHTw+OlQMNm2/7iOelrDxphN0hVjHaFGbMWTsjU8nefrABhcPJsGGELqT5w+38L+4AkdOn467PcmU/vzddr7nBRPylZOBkXiAKKLMAIMYcc8Z863sN9/XJ80dn3439TRGe0P/gu2sHdm9Xy7PQD0B2EEQEhjc8fzdYJ36+x1NXcZM3Fk8Pk6HY3Yxo9Ij8FMASQTwgiAHp1saWt/vk4woHxw1Nvl+TrjhqeFnkpcMmmkJo5M5/k6ACJCGAHQZ97Ws3r3YMdlnRPadcQj/wXpxJllU3H73Tolk7I1dXQG4QRArwgjAPqtxXdONR0P/ztwXH+u96jNHwgbM3KYNVRzUjxppKbnZvLwPwBhCCMABkzrWb9q6zqeTHxCtXUn5TsXHk4caanB5+u0X9qZMYbn6wBDHWEEQNT4zvm167BH2w6c0Dv7gw//O93mDxuTYUvRVfkj2ldPRmrWeIdSCSfAkEIYATBozvoD2n3UG2rEtv3ACTX7zoWNSUu1qGjiiNClnYI8nq8DJDvCCICY8QcM7TnmDWvE9unps2FjrClmzc4brpJJ2Zo+JksTstM1YWS6smhjDyQNwgiAuBEIGPqo4VTo4X/bDhxX06m2bscOT08NBpMLt5HpGuNIk4UiWSBhEEYAxC3DMLS/qUXb9p/QuwdP6MDxFtWfON1jQOmQYjZp/Ig05bUHlIkjg//s+JmHAwLxhTACIOG0+M6p7sRp1Z04rfr2fx46Hvz3+pOnddbf+3+usodZQ8FkQnaaJmYPC/48Ml25WXZWVYBB1te/3ymDOCcA6NUwW4o+MyZLnxnT9T9a/oAht7c1FE7qLthOtLSFtj/Xf9rl+FSLSeNHhF/6yWtfXcnLTleGjf8cArHCtw9AQrCYTRo7PE1jh6dp3pSRXX7f3HpW9SfOtIeTlvZ/nlH9idM63L6qcqCpRQeauj40UAo2cet8+SevU2jJzbLT0A2IIsIIgKSQaU/VjLGpmjG2+1WVY54z3V7+qTtxWidPn9XxljYdb2nTzm5WVawWs8Znp3VbVJs3Il3DWFUBLgnfIABJz2IOXqIZPyJdmtL1997Ws6rr4fLPkZNn1OYPaH9ji/Y3dr+qkpNh7ebyzzBNyE7X6EwbqyrARVDACgC9OOcP6JinVfUnTutQp5DSEVwu7J9yIWuKWXkj0tov/wwLu/wzITtdaVYavyF5UcAKAAMgxWJWXvtqx9Xd/N5z5mzYikrnyz9HPj2jtnMBfdLYok8aWyQ1djl+VKYtfEWl/fLPhOx0jcpgVQVDAysjABAl5/wBHf20NeyyT3CFpUWHjp9Wc+u5Xo+3pZjDgsqFvVVop494x8oIAMRYisUcXOUYmd7t7z2nzwZXU9rv/qm/oFbFdy6gjxpO6aOGU90eP7pjVWVk18LaURk2mUysqiAxsDICAHHorD+go5+eOb+qcvx02L9f+CDCC9lTzT1e/hk/glUVDA5WRgAggaVazJo4cpgmjhzW5XeGYejT9lWVsMs/7YHlmOeMWs8G9KH7lD50d7+qkptlD7v8M2FkmiZkB+8AysmwsqqCQcXKCAAkmbZzwVWVQ53v/Om0snLqIqsqaamWHi//jB+RJlsKqyroG1ZGAGCIsqaYlZ8zTPk53a+qnOy8qnK8pdPqyhkd9ZzRmbN+7XM3a5+7ucvxJlNwVSVUUNu+upLrsCs3yy5nlp3blRExVkYAACG+c34dOXmmy6Wfju10m/+ir5FlT1GuIxhMnFntIaU9rAQDi00jM2w8uHAIYGUEABAxW4pFk0dlaPKojC6/MwxDJ1radOiCSz/1J0/L7fXJ5WnVmbN+eVvPydvac72KFOyKOzrT1h5YbGGBJRRiHHYeYDhE8CkDAPrEZDJpZEZwVWPOhBFdfm8Yhryt5+T2tsrtbZXL0/5Pb6vcXl9oX9MpX/vzglp1zNPa63tm2FI0uj2shAcWWyiwjMqwKcVijtZpYxAQRgAAA8JkMsmRlipHWqoud2b2OO6cP6CmU21ydQos50NLcF+D16dm3zmd8p3TqcZzPT4XKPi+Uk6GLbSqkuuwyZnZ6dKQwy5npl1ZaSncJRSnCCMAgEGVYjEHC14ddimv53EtvnPBgOJplbu5VS7P+dUVd3Nwf0OzT+cChhqbfWps9mnXEU+Pr2dPNXe5DNRR05LrsGl0ZvBnawqrLIONMAIAiEvDbCmaMipDU7qpX+kQCBhqavGpob1mpWN1JbjS4pO7fZ/nzFm1ng3o4PHTOnj8dK/vO3KY9XwtS6fA0jnEjEhPZZVlAPUrjKxZs0Y//elP5XK5VFBQoEcffVTFxcXdjt29e7dWrlypmpoaHTp0SD//+c919913X8qcAQCQJJnNJo3OtGt0pl1XjnP0OK71rD+0quLyBi8DuTouDYVWWnxq8wd0vKVNx1va9MGxnt/XajGHalm61LF0WnWh023fRBxGNm7cqPLycq1du1YlJSV65JFHNH/+fO3bt0+jR4/uMv706dOaPHmyvvGNb+h73/vegEwaAIBI2FMtPXa07dDRg6XzZSBXWDFu8DLR8ZY2tfkDOnzyjA6fPNPr+zrSUjsFFlvY7c65DrtGZ9mUM4ynM0fcZ6SkpERz587V6tWrJUmBQEB5eXm68847tWLFil6Pzc/P19133x3xygh9RgAA8cJ3zq8Gr08N7XUsYZeGOt1B1Ho20KfXS+m4zbm90DZ0aeiCQtxhCXibc1T6jLS1tammpkYVFRWhfWazWaWlpdq6dWv/ZwsAQIKwpViU1955tiedb3M+f2motf0OIl8ovDSeChbgHvW06uhFbnPO7LjNuYc6ltwsu3IyrAl5m3NEYaSpqUl+v19OpzNsv9Pp1N69ewdsUj6fTz6fL/Sz1+sdsNcGACDaIrnNufGUL+wyUPgdRMH9p3zn1Ow7p+bGc/qkl9uczR23OYcFFltYYBmdZVeWPb5uc47LNZ/Kykrdd999sZ4GAABRlWIxa4wjTWMcab2OO+VrbybXvsoSKsLtVNfS0BxsJtfQ7FNDs09Sz7c5p6Va2gPL+aLbxcUTun2e0WCIKIzk5OTIYrHI7XaH7Xe73crNzR2wSVVUVKi8vDz0s9frVV5eLzejAwCQxDJsKcq4yG3O/oCh4y0+ubupY+l8B5HnzFmdOevXgaYWHWg6v8ryhStyEyOMWK1WFRUVqaqqSgsXLpQULGCtqqrSHXfcMWCTstlsstlsA/Z6AAAkO0un25xnqufbnM+0+bvpeuvTxJE918BEW8SXacrLy3XrrbfqqquuUnFxsR555BG1tLSorKxMkrR06VKNGzdOlZWVkoJFrx988EHo348cOaKdO3cqIyNDU6dOHcBTAQAAF5NmtSg/Z1jMVkG6E3EYWbRokRobG7Vy5Uq5XC4VFhZq06ZNoaLWuro6mc3nK3mPHj2q2bNnh35+6KGH9NBDD+naa6/Vli1bLv0MAABAQou4z0gs0GcEAIDE09e/34l3MzIAAEgqhBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBTET8oLxY6Hp/j9XpjPBMAANBXHX+3L/YYvIQII83NzZKkvLy8GM8EAABEqrm5WQ6Ho8ffJ8RTewOBgI4eParMzEyZTKYBe12v16u8vDzV19cn7dOAk/0cOb/El+znyPklvmQ/x2ien2EYam5u1tixY2U291wZkhArI2azWePHj4/a62dlZSXl/8A6S/Zz5PwSX7KfI+eX+JL9HKN1fr2tiHSggBUAAMQUYQQAAMTUkA4jNptNq1atks1mi/VUoibZz5HzS3zJfo6cX+JL9nOMh/NLiAJWAACQvIb0yggAAIg9wggAAIgpwggAAIgpwggAAIippA8ja9asUX5+vux2u0pKSlRdXd3r+Oeee07Tp0+X3W7XzJkz9eqrrw7STPsvknN88sknZTKZwja73T6Is43MW2+9pRtvvFFjx46VyWTSyy+/fNFjtmzZojlz5shms2nq1Kl68sknoz7P/or0/LZs2dLl8zOZTHK5XIMz4QhVVlZq7ty5yszM1OjRo7Vw4ULt27fvosclyvewP+eXaN/BX/7yl5o1a1aoIda8efP0u9/9rtdjEuXzkyI/v0T7/C704IMPymQy6e677+513GB/hkkdRjZu3Kjy8nKtWrVKtbW1Kigo0Pz589XQ0NDt+D/96U9avHixvv3tb2vHjh1auHChFi5cqPfff3+QZ953kZ6jFOyyd+zYsdB26NChQZxxZFpaWlRQUKA1a9b0afyBAwe0YMECfe5zn9POnTt1991367bbbtNrr70W5Zn2T6Tn12Hfvn1hn+Ho0aOjNMNL8+abb2r58uV65513tHnzZp09e1Zf+MIX1NLS0uMxifQ97M/5SYn1HRw/frwefPBB1dTU6N1339Vf//Vf66abbtLu3bu7HZ9In58U+flJifX5dbZ9+3Y99thjmjVrVq/jYvIZGkmsuLjYWL58eehnv99vjB071qisrOx2/N/+7d8aCxYsCNtXUlJi/P3f/31U53kpIj3HX/3qV4bD4Rik2Q0sScZLL73U65jvf//7xhVXXBG2b9GiRcb8+fOjOLOB0Zfz+8Mf/mBIMk6ePDkocxpoDQ0NhiTjzTff7HFMIn4PO/Tl/BL5O9hhxIgRxn/+5392+7tE/vw69HZ+ifr5NTc3G5dddpmxefNm49prrzXuuuuuHsfG4jNM2pWRtrY21dTUqLS0NLTPbDartLRUW7du7faYrVu3ho2XpPnz5/c4Ptb6c46SdOrUKU2cOFF5eXkX/X8AiSbRPsP+Kiws1JgxY3T99dfrj3/8Y6yn02cej0eSlJ2d3eOYRP4M+3J+UuJ+B/1+vzZs2KCWlhbNmzev2zGJ/Pn15fykxPz8li9frgULFnT5bLoTi88wacNIU1OT/H6/nE5n2H6n09nj9XWXyxXR+FjrzzlOmzZNTzzxhH7zm9/omWeeUSAQ0NVXX63Dhw8PxpSjrqfP0Ov16syZMzGa1cAZM2aM1q5dqxdeeEEvvPCC8vLydN1116m2tjbWU7uoQCCgu+++W9dcc42uvPLKHscl2vewQ1/PLxG/g7t27VJGRoZsNpu++93v6qWXXtKMGTO6HZuIn18k55eIn9+GDRtUW1urysrKPo2PxWeYEE/txcCZN29eWOK/+uqr9ZnPfEaPPfaYfvSjH8VwZuiLadOmadq0aaGfr776an3yySf6+c9/rqeffjqGM7u45cuX6/3339fbb78d66lERV/PLxG/g9OmTdPOnTvl8Xj0/PPP69Zbb9Wbb77Z4x/sRBPJ+SXa51dfX6+77rpLmzdvjutC26QNIzk5ObJYLHK73WH73W63cnNzuz0mNzc3ovGx1p9zvFBqaqpmz56tjz/+OBpTHHQ9fYZZWVlKS0uL0ayiq7i4OO7/wN9xxx363//9X7311lsaP358r2MT7XsoRXZ+F0qE76DVatXUqVMlSUVFRdq+fbt+8Ytf6LHHHusyNhE/v0jO70Lx/vnV1NSooaFBc+bMCe3z+/166623tHr1avl8PlkslrBjYvEZJu1lGqvVqqKiIlVVVYX2BQIBVVVV9XgtcN68eWHjJWnz5s29XjuMpf6c44X8fr927dqlMWPGRGuagyrRPsOBsHPnzrj9/AzD0B133KGXXnpJv//97zVp0qSLHpNIn2F/zu9CifgdDAQC8vl83f4ukT6/nvR2fheK98/v85//vHbt2qWdO3eGtquuuko333yzdu7c2SWISDH6DKNWGhsHNmzYYNhsNuPJJ580PvjgA+M73/mOMXz4cMPlchmGYRi33HKLsWLFitD4P/7xj0ZKSorx0EMPGXv27DFWrVplpKamGrt27YrVKVxUpOd43333Ga+99prxySefGDU1NcY3v/lNw263G7t3747VKfSqubnZ2LFjh7Fjxw5DkvHwww8bO3bsMA4dOmQYhmGsWLHCuOWWW0Lj9+/fb6Snpxv/9E//ZOzZs8dYs2aNYbFYjE2bNsXqFHoV6fn9/Oc/N15++WXjo48+Mnbt2mXcddddhtlsNt54441YnUKvli1bZjgcDmPLli3GsWPHQtvp06dDYxL5e9if80u07+CKFSuMN9980zhw4IDx3nvvGStWrDBMJpPx+uuvG4aR2J+fYUR+fon2+XXnwrtp4uEzTOowYhiG8eijjxoTJkwwrFarUVxcbLzzzjuh31177bXGrbfeGjb+17/+tXH55ZcbVqvVuOKKK4xXXnllkGccuUjO8e677w6NdTqdxg033GDU1tbGYNZ903Er64VbxzndeuutxrXXXtvlmMLCQsNqtRqTJ082fvWrXw36vPsq0vP7yU9+YkyZMsWw2+1Gdna2cd111xm///3vYzP5Puju3CSFfSaJ/D3sz/kl2nfw7/7u74yJEycaVqvVGDVqlPH5z38+9IfaMBL78zOMyM8v0T6/7lwYRuLhMzQZhmFEb90FAACgd0lbMwIAABIDYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMTU/w+ZuvZWqANXDwAAAABJRU5ErkJggg==\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "## Try a different sampler\n", + "\n", + "The top-K algorithm randomly picks the next token from the tokens of top K probability.\n", + "\n", + "**NOTE: Due to randomness of the sampler, you may encounter \\ tokens that surpass the number you trained with.**" + ], + "metadata": { + "id": "4QkVxDR1IHmx" + } + }, + { + "cell_type": "code", + "source": [ + "gemma_lm.compile(sampler=\"top_k\")\n", + "\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "1zwJmf9KzwW0", + "outputId": "dc5c39b2-0bc2-4888-c786-475724887bba" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 142742 39742 107 108 106 2516\n", + " 108 23 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + "142742 -> ็ใ—ใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 23 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ๅ˜‰ๅคšใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 22.22s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 142742 39742 107 108 106 2516\n", + " 108 59 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + "142742 -> ็ใ—ใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 59 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.54s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 142742 39742 107 108 106 2516\n", + " 108 75 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + "142742 -> ็ใ—ใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 75 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.51s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 142742 39742 107 108 106 2516\n", + " 108 30 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + "142742 -> ็ใ—ใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 30 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ๅ‘จๆœฌใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.53s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 142742 39742 107 108 106 2516\n", + " 108 33 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + "142742 -> ็ใ—ใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 33 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ไธ‹ๅฐ่–—ใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.51s\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "source": [ + "Try a slight different prompts" + ], + "metadata": { + "id": "mqIx09b2INH0" + } + }, + { + "cell_type": "code", + "source": [ + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ้ข็™ฝใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ่ฆ‹ใŸใ“ใจใชใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎใ™ใ”ใ„ๅๅญ—\", my_dictionary)" + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "oWSODjdj8H4V", + "outputId": "2c1d3ff6-d873-4f4c-df70-17f91421f294" + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 39742 107 108 106 2516 108 98647 4758\n", + " 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 98647 -> ็”ฐไธญ\n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎๅๅญ—\n", + "model\n", + "็”ฐไธญใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.69s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 77427 39742 107 108 106 2516 108 106\n", + " 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + " 77427 -> ้ข็™ฝใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ้ข็™ฝใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.50s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 46116 4302 3581 39742 107 108 106 2516\n", + " 108 51 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + " 46116 -> ่ฆ‹ใŸ\n", + " 4302 -> ใ“ใจ\n", + " 3581 -> ใชใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 51 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎ่ฆ‹ใŸใ“ใจใชใ„ๅๅญ—\n", + "model\n", + "็ฅžๅฐŠใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.58s\n", + "--------------------------------------------------------------------------------\n", + "[ 106 1645 108 47172 59956 39742 107 108 106 2516 108 16\n", + " 4758 107]\n", + " 106 -> \n", + " 1645 -> user\n", + " 108 -> \n", + "\n", + " 47172 -> ๆ—ฅๆœฌใฎ\n", + " 59956 -> ใ™ใ”ใ„\n", + " 39742 -> ๅๅญ—\n", + " 107 -> \n", + " 108 -> \n", + "\n", + " 106 -> \n", + " 2516 -> model\n", + " 108 -> \n", + "\n", + " 16 -> \n", + " 4758 -> ใ•ใ‚“\n", + " 107 -> \n", + "\n", + "Gemma output:\n", + "user\n", + "ๆ—ฅๆœฌใฎใ™ใ”ใ„ๅๅญ—\n", + "model\n", + "็œŸๅ–œใ•ใ‚“\n", + "--------------------------------------------------------------------------------\n", + "TOTAL TIME ELAPSED: 0.49s\n" + ] + } + ] + } + ] +} \ No newline at end of file From acf5f8b1716b80a2031480b87a17e3714c0bf893 Mon Sep 17 00:00:00 2001 From: Juyeong Ji Date: Mon, 21 Oct 2024 10:37:23 +0900 Subject: [PATCH 12/43] Update README & nbfmt --- Gemma/Custom_Vocabulary.ipynb | 739 ++++++++++++++++------------------ README.md | 1 + 2 files changed, 349 insertions(+), 391 deletions(-) diff --git a/Gemma/Custom_Vocabulary.ipynb b/Gemma/Custom_Vocabulary.ipynb index a31eeba..d1c6985 100644 --- a/Gemma/Custom_Vocabulary.ipynb +++ b/Gemma/Custom_Vocabulary.ipynb @@ -1,34 +1,22 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "provenance": [], - "gpuType": "T4", - "toc_visible": true, - "authorship_tag": "ABX9TyNzoetBHJ/RyGzWlzpUM8Ag" - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "language_info": { - "name": "python" - }, - "accelerator": "GPU" - }, "cells": [ { "cell_type": "markdown", - "source": [ - "##### Copyright 2024 Google LLC." - ], "metadata": { "id": "hovLNQ-tZuwd" - } + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] }, { "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "yDENQhUBZxfL" + }, + "outputs": [], "source": [ "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", @@ -41,15 +29,13 @@ "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." - ], - "metadata": { - "id": "yDENQhUBZxfL" - }, - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", + "metadata": { + "id": "eNmbq93EZzID" + }, "source": [ "# Gemma - How to use a custom vocabulary\n", "\n", @@ -75,13 +61,13 @@ " Run in Google Colab\n", " \n", "\n" - ], - "metadata": { - "id": "eNmbq93EZzID" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "nathXe5ebTQI" + }, "source": [ "## Setup\n", "\n", @@ -101,24 +87,26 @@ "* Generate and configure a Kaggle username and API key.\n", "\n", "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment." - ], - "metadata": { - "id": "nathXe5ebTQI" - } + ] }, { "cell_type": "markdown", + "metadata": { + "id": "2SNOwNf6OV28" + }, "source": [ "### Set environment variables\n", "\n", "Set environment variables for `KAGGLE_USERNAME` and `KAGGLE_KEY`." - ], - "metadata": { - "id": "2SNOwNf6OV28" - } + ] }, { "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HuGBBTQlbX7w" + }, + "outputs": [], "source": [ "import os\n", "from google.colab import userdata, drive\n", @@ -130,26 +118,26 @@ "\n", "# Mounting gDrive for to store artifacts\n", "drive.mount(\"/content/drive\")" - ], - "metadata": { - "id": "HuGBBTQlbX7w" - }, - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", + "metadata": { + "id": "1gaYpiL-bgWZ" + }, "source": [ "### Install dependencies\n", "\n", "Install Keras and KerasNLP" - ], - "metadata": { - "id": "1gaYpiL-bgWZ" - } + ] }, { "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "CcMRCmR4OVdx" + }, + "outputs": [], "source": [ "!pip install -q -U keras-nlp datasets\n", "!pip install -q -U keras\n", @@ -173,15 +161,13 @@ "lr_value = 5e-4\n", "train_epoch = 5\n", "model_id = \"gemma2_instruct_2b_en\"" - ], - "metadata": { - "id": "CcMRCmR4OVdx" - }, - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", + "metadata": { + "id": "0DSjJE4aOKc7" + }, "source": [ "## Identify the presence of Surnames in the Tokenizer's vocabulary\n", "\n", @@ -204,25 +190,18 @@ "```\n", "\n", "In the case of \"ๅŠ ้ฌฎ\", tokenizer typically splits this character into its UTF-8 representation. This is because the tokenizer hasn't encountered this character frequently during the training process. If you fine-tune the model, it will significantly improve efficiency. For example \"ๅŠ ้ฌฎ\", 4 tokens can be reduced to 1 token." - ], - "metadata": { - "id": "0DSjJE4aOKc7" - } + ] }, { "cell_type": "code", "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "Wp2aaKJ8OE-y", - "outputId": "79d08d08-5d3d-44b9-fb1a-4c97975b4bf5" + "id": "Wp2aaKJ8OE-y" }, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "[ 108 56985 32012 235465 78638 4758 235394 79900 4758 235394\n", " 122224 4758 235394 98647 4758 235394 119177 4758 235394 169944\n", @@ -389,47 +368,35 @@ }, { "cell_type": "code", - "source": [ - "print(tokenizer.id_to_token(450))\n", - "print(tokenizer.id_to_token(389))\n", - "print(tokenizer.id_to_token(391))" - ], + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "pMFgoID3FAcJ", - "outputId": "8e89279a-efc3-457e-df03-7206a2a068bf" + "id": "pMFgoID3FAcJ" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "<0xE9>\n", "<0xAC>\n", "<0xAE>\n" ] } + ], + "source": [ + "print(tokenizer.id_to_token(450))\n", + "print(tokenizer.id_to_token(389))\n", + "print(tokenizer.id_to_token(391))" ] }, { "cell_type": "code", - "source": [ - "tokenizer.get_vocabulary()" - ], + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "ggigihbgvkLc", - "outputId": "f53fcb4f-3f17-40c1-e956-dedbad1cf3c3" + "id": "ggigihbgvkLc" }, - "execution_count": null, "outputs": [ { - "output_type": "execute_result", "data": { "text/plain": [ "['',\n", @@ -1435,100 +1402,46 @@ " ...]" ] }, + "execution_count": 6, "metadata": {}, - "execution_count": 6 + "output_type": "execute_result" } + ], + "source": [ + "tokenizer.get_vocabulary()" ] }, { "cell_type": "markdown", - "source": [ - "## Load Model" - ], "metadata": { "id": "9VaAVcrpb1fI" - } + }, + "source": [ + "## Load Model" + ] }, { "cell_type": "code", - "source": [ - "import keras\n", - "import keras_nlp\n", - "\n", - "import time\n", - "\n", - "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n", - "gemma_lm.summary()\n", - "\n", - "tick_start = 0\n", - "\n", - "def tick():\n", - " global tick_start\n", - " tick_start = time.time()\n", - "\n", - "def tock():\n", - " print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n", - "\n", - "def text_gen(prompt):\n", - " tick()\n", - " input = f\"user\\n{prompt}\\nmodel\\n\"\n", - " output = gemma_lm.generate(input, max_length=token_limit)\n", - " print(\"\\nGemma output:\")\n", - " print(output)\n", - " tock()\n", - "\n", - "import re\n", - "\n", - "def text_gen_with_dict(prompt, dictionary):\n", - " tick()\n", - " input = f\"user\\n{prompt}\\nmodel\\n\"\n", - " output = gemma_lm.generate(input, max_length=token_limit)\n", - " pattern = '|'.join(sorted(re.escape(k) for k in dictionary))\n", - " print(\"-\"*80)\n", - " detoken(tokenizer(output))\n", - " output = re.sub(pattern, lambda m: dictionary.get(m.group(0)), output)\n", - " print(\"\\nGemma output:\")\n", - " print(output)\n", - " print(\"-\"*80)\n", - " tock()\n", - "\n", - "# inference before fine-tuning\n", - "text_gen(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\")\n", - "text_gen(\"ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\")" - ], + "execution_count": null, "metadata": { - "id": "S9WifD8eb57U", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 929 - }, - "outputId": "9b843161-fe20-44e2-8aff-689985f36fc5" + "id": "S9WifD8eb57U" }, - "execution_count": null, "outputs": [ { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n" - ], "text/html": [ "
Preprocessor: \"gemma_causal_lm_preprocessor\"\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", - "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Config\u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", - "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", - "โ”‚ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m) โ”‚ Vocab size: \u001b[38;5;34m256,000\u001b[0m โ”‚\n", - "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" - ], "text/html": [ "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
               "โ”ƒ Layer (type)                                                  โ”ƒ                                   Config โ”ƒ\n",
@@ -1536,41 +1449,33 @@
               "โ”‚ gemma_tokenizer (GemmaTokenizer)                              โ”‚                      Vocab size: 256,000 โ”‚\n",
               "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
               "
\n" + ], + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Config\u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m) โ”‚ Vocab size: \u001b[38;5;34m256,000\u001b[0m โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n" - ], "text/html": [ "
Model: \"gemma_causal_lm\"\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", - "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mConnected to \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", - "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", - "โ”‚ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", - "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", - "โ”‚ token_ids (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", - "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", - "โ”‚ gemma_backbone โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m) โ”‚ \u001b[38;5;34m2,614,341,888\u001b[0m โ”‚ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m], โ”‚\n", - "โ”‚ (\u001b[38;5;33mGemmaBackbone\u001b[0m) โ”‚ โ”‚ โ”‚ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", - "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", - "โ”‚ token_embedding โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m) โ”‚ \u001b[38;5;34m589,824,000\u001b[0m โ”‚ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", - "โ”‚ (\u001b[38;5;33mReversibleEmbedding\u001b[0m) โ”‚ โ”‚ โ”‚ โ”‚\n", - "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" - ], "text/html": [ "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
               "โ”ƒ Layer (type)                  โ”ƒ Output Shape              โ”ƒ         Param # โ”ƒ Connected to               โ”ƒ\n",
@@ -1586,52 +1491,68 @@
               "โ”‚ (ReversibleEmbedding)         โ”‚                           โ”‚                 โ”‚                            โ”‚\n",
               "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
               "
\n" + ], + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mConnected to \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_ids (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ gemma_backbone โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m) โ”‚ \u001b[38;5;34m2,614,341,888\u001b[0m โ”‚ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m], โ”‚\n", + "โ”‚ (\u001b[38;5;33mGemmaBackbone\u001b[0m) โ”‚ โ”‚ โ”‚ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_embedding โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m) โ”‚ \u001b[38;5;34m589,824,000\u001b[0m โ”‚ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”‚ (\u001b[38;5;33mReversibleEmbedding\u001b[0m) โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" - ], "text/html": [ "
 Total params: 2,614,341,888 (9.74 GB)\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" - ], "text/html": [ "
 Trainable params: 2,614,341,888 (9.74 GB)\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" - ], "text/html": [ "
 Non-trainable params: 0 (0.00 B)\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "\n", "Gemma output:\n", @@ -1665,17 +1586,88 @@ "TOTAL TIME ELAPSED: 5.38s\n" ] } - ] - }, - { - "cell_type": "code", + ], "source": [ - "# example data\n", - "custom_vocab = [\n", - " \"ๅ†…ๅคงไน…ไฟ\",\"่ซ่จชๆˆธ\",\"ๅฎฎ่‰ธ\",\"ๅฎๅ‚\",\"ๅŸœไธŠ\",\"็ฏ ๅžฃ\",\"ๆฑ ๅ‘‚\",\"ๅฅฅ้ ˆ่ณ€\",\"ๅ‹ๅฑ…\",\"็œŸๅ–œ\",\n", - " \"้ซ˜ๆฒณๅŽŸ\",\"ๅฐ็››\",\"ๆบœๅฃ\",\"ไผŠๅ‹ขไบ•\",\"่ฝไน…ไฟ\",\"ๅฟ—ๆธกๆพค\",\"ๅ˜‰ๅคš\",\"ๆฟ่—ค\",\"ๅ—ๆณข็•™\",\"้ขจใ‚“\",\n", - " \"้‡‡ๅฐพ\",\"ไธ€ไธŠ\",\"ๆœฌ้‚‘\",\"ๅ‘จๆœฌ\",\"็• ๅฐพ\",\"้ณฅๆฐด\",\"ไธ‹ๅฐ่–—\",\"้˜ช่พบ\",\"็จฒๅ‘ณ\",\"ๆญฆๆตช\",\n", - " \"ๅฎ‰ๆฅฝๅŸŽ\",\"ๆฑŸๅค\",\"่ณŽๆฉŸ\",\"้ถดๅŽŸ่ฐท\",\"่ฅฟๆ˜Žๅฏบ\",\"ๅธƒๅกš\",\"ๅฏบ็ซฏ\",\"็‚ญๅ‰\",\"็ฎก็”Ÿ\",\"ๆ‘็‰‡\",\n", + "import keras\n", + "import keras_nlp\n", + "\n", + "import time\n", + "\n", + "gemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(model_id)\n", + "gemma_lm.summary()\n", + "\n", + "tick_start = 0\n", + "\n", + "def tick():\n", + " global tick_start\n", + " tick_start = time.time()\n", + "\n", + "def tock():\n", + " print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n", + "\n", + "def text_gen(prompt):\n", + " tick()\n", + " input = f\"user\\n{prompt}\\nmodel\\n\"\n", + " output = gemma_lm.generate(input, max_length=token_limit)\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + " tock()\n", + "\n", + "import re\n", + "\n", + "def text_gen_with_dict(prompt, dictionary):\n", + " tick()\n", + " input = f\"user\\n{prompt}\\nmodel\\n\"\n", + " output = gemma_lm.generate(input, max_length=token_limit)\n", + " pattern = '|'.join(sorted(re.escape(k) for k in dictionary))\n", + " print(\"-\"*80)\n", + " detoken(tokenizer(output))\n", + " output = re.sub(pattern, lambda m: dictionary.get(m.group(0)), output)\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + " print(\"-\"*80)\n", + " tock()\n", + "\n", + "# inference before fine-tuning\n", + "text_gen(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\")\n", + "text_gen(\"ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8rXG-F4VdQQu" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'': 'ๅ†…ๅคงไน…ไฟ', '': '่ซ่จชๆˆธ', '': 'ๅฎฎ่‰ธ', '': 'ๅฎๅ‚', '': 'ๅŸœไธŠ', '': '็ฏ ๅžฃ', '': 'ๆฑ ๅ‘‚', '': 'ๅฅฅ้ ˆ่ณ€', '': 'ๅ‹ๅฑ…', '': '็œŸๅ–œ', '': '้ซ˜ๆฒณๅŽŸ', '': 'ๅฐ็››', '': 'ๆบœๅฃ', '': 'ไผŠๅ‹ขไบ•', '': '่ฝไน…ไฟ', '': 'ๅฟ—ๆธกๆพค', '': 'ๅ˜‰ๅคš', '': 'ๆฟ่—ค', '': 'ๅ—ๆณข็•™', '': '้ขจใ‚“', '': '้‡‡ๅฐพ', '': 'ไธ€ไธŠ', '': 'ๆœฌ้‚‘', '': 'ๅ‘จๆœฌ', '': '็• ๅฐพ', '': '้ณฅๆฐด', '': 'ไธ‹ๅฐ่–—', '': '้˜ช่พบ', '': '็จฒๅ‘ณ', '': 'ๆญฆๆตช', '': 'ๅฎ‰ๆฅฝๅŸŽ', '': 'ๆฑŸๅค', '': '่ณŽๆฉŸ', '': '้ถดๅŽŸ่ฐท', '': '่ฅฟๆ˜Žๅฏบ', '': 'ๅธƒๅกš', '': 'ๅฏบ็ซฏ', '': '็‚ญๅ‰', '': '็ฎก็”Ÿ', '': 'ๆ‘็‰‡', '': 'ๆ˜Œๅธ', '': 'ไผŠ็งฉ', '': 'ๅพŒ้“', '': 'ไฝไฟไบ•', '': '็ฅžๅฐŠ', '': '็‚บ็ฉ', '': '่–ๅท', '': '็™ปๅ–œ', '': 'ๅผฆ็”ฐ', '': '็Šฌๅฑ…'}\n", + "70\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ๅฑฑๆœฌใ•ใ‚“\n", + "user\n", + "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", + "model\n", + "ใ•ใ‚“\n", + "user\n", + "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", + "model\n", + "ๆพๆœฌใ•ใ‚“\n" + ] + } + ], + "source": [ + "# example data\n", + "custom_vocab = [\n", + " \"ๅ†…ๅคงไน…ไฟ\",\"่ซ่จชๆˆธ\",\"ๅฎฎ่‰ธ\",\"ๅฎๅ‚\",\"ๅŸœไธŠ\",\"็ฏ ๅžฃ\",\"ๆฑ ๅ‘‚\",\"ๅฅฅ้ ˆ่ณ€\",\"ๅ‹ๅฑ…\",\"็œŸๅ–œ\",\n", + " \"้ซ˜ๆฒณๅŽŸ\",\"ๅฐ็››\",\"ๆบœๅฃ\",\"ไผŠๅ‹ขไบ•\",\"่ฝไน…ไฟ\",\"ๅฟ—ๆธกๆพค\",\"ๅ˜‰ๅคš\",\"ๆฟ่—ค\",\"ๅ—ๆณข็•™\",\"้ขจใ‚“\",\n", + " \"้‡‡ๅฐพ\",\"ไธ€ไธŠ\",\"ๆœฌ้‚‘\",\"ๅ‘จๆœฌ\",\"็• ๅฐพ\",\"้ณฅๆฐด\",\"ไธ‹ๅฐ่–—\",\"้˜ช่พบ\",\"็จฒๅ‘ณ\",\"ๆญฆๆตช\",\n", + " \"ๅฎ‰ๆฅฝๅŸŽ\",\"ๆฑŸๅค\",\"่ณŽๆฉŸ\",\"้ถดๅŽŸ่ฐท\",\"่ฅฟๆ˜Žๅฏบ\",\"ๅธƒๅกš\",\"ๅฏบ็ซฏ\",\"็‚ญๅ‰\",\"็ฎก็”Ÿ\",\"ๆ‘็‰‡\",\n", " \"ๆ˜Œๅธ\",\"ไผŠ็งฉ\",\"ๅพŒ้“\",\"ไฝไฟไบ•\",\"็ฅžๅฐŠ\",\"็‚บ็ฉ\",\"่–ๅท\",\"็™ปๅ–œ\",\"ๅผฆ็”ฐ\",\"็Šฌๅฑ…\",\n", "]\n", "my_dictionary = {f\"\":custom_vocab[i] for i in range(len(custom_vocab))}\n", @@ -1713,103 +1705,39 @@ "print(train[0])\n", "print(train[1])\n", "print(train[2])" - ], - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "8rXG-F4VdQQu", - "outputId": "a98c7ef7-75ee-4a8b-abd1-4c02fba746a4" - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "{'': 'ๅ†…ๅคงไน…ไฟ', '': '่ซ่จชๆˆธ', '': 'ๅฎฎ่‰ธ', '': 'ๅฎๅ‚', '': 'ๅŸœไธŠ', '': '็ฏ ๅžฃ', '': 'ๆฑ ๅ‘‚', '': 'ๅฅฅ้ ˆ่ณ€', '': 'ๅ‹ๅฑ…', '': '็œŸๅ–œ', '': '้ซ˜ๆฒณๅŽŸ', '': 'ๅฐ็››', '': 'ๆบœๅฃ', '': 'ไผŠๅ‹ขไบ•', '': '่ฝไน…ไฟ', '': 'ๅฟ—ๆธกๆพค', '': 'ๅ˜‰ๅคš', '': 'ๆฟ่—ค', '': 'ๅ—ๆณข็•™', '': '้ขจใ‚“', '': '้‡‡ๅฐพ', '': 'ไธ€ไธŠ', '': 'ๆœฌ้‚‘', '': 'ๅ‘จๆœฌ', '': '็• ๅฐพ', '': '้ณฅๆฐด', '': 'ไธ‹ๅฐ่–—', '': '้˜ช่พบ', '': '็จฒๅ‘ณ', '': 'ๆญฆๆตช', '': 'ๅฎ‰ๆฅฝๅŸŽ', '': 'ๆฑŸๅค', '': '่ณŽๆฉŸ', '': '้ถดๅŽŸ่ฐท', '': '่ฅฟๆ˜Žๅฏบ', '': 'ๅธƒๅกš', '': 'ๅฏบ็ซฏ', '': '็‚ญๅ‰', '': '็ฎก็”Ÿ', '': 'ๆ‘็‰‡', '': 'ๆ˜Œๅธ', '': 'ไผŠ็งฉ', '': 'ๅพŒ้“', '': 'ไฝไฟไบ•', '': '็ฅžๅฐŠ', '': '็‚บ็ฉ', '': '่–ๅท', '': '็™ปๅ–œ', '': 'ๅผฆ็”ฐ', '': '็Šฌๅฑ…'}\n", - "70\n", - "user\n", - "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", - "model\n", - "ๅฑฑๆœฌใ•ใ‚“\n", - "user\n", - "ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\n", - "model\n", - "ใ•ใ‚“\n", - "user\n", - "ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\n", - "model\n", - "ๆพๆœฌใ•ใ‚“\n" - ] - } ] }, { "cell_type": "markdown", - "source": [ - "## LoRA Fine-tuning" - ], "metadata": { "id": "NAxoEthfyye6" - } + }, + "source": [ + "## LoRA Fine-tuning" + ] }, { "cell_type": "code", - "source": [ - "# Enable LoRA for the model and set the LoRA rank to 4.\n", - "gemma_lm.backbone.enable_lora(rank=lora_rank)\n", - "gemma_lm.summary()\n", - "\n", - "# Limit the input sequence length (to control memory usage).\n", - "gemma_lm.preprocessor.sequence_length = token_limit\n", - "# Use AdamW (a common optimizer for transformer models).\n", - "optimizer = keras.optimizers.AdamW(\n", - " learning_rate=lr_value,\n", - " weight_decay=0.01,\n", - ")\n", - "# Exclude layernorm and bias terms from decay.\n", - "optimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n", - "\n", - "gemma_lm.compile(\n", - " loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", - " optimizer=optimizer,\n", - " weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n", - ")" - ], + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 408 - }, - "id": "OQuVUnvpy1rL", - "outputId": "f4fe7c3e-cf62-4d09-e88e-47221a8999c7" + "id": "OQuVUnvpy1rL" }, - "execution_count": null, "outputs": [ { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n" - ], "text/html": [ "
Preprocessor: \"gemma_causal_lm_preprocessor\"\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1mPreprocessor: \"gemma_causal_lm_preprocessor\"\u001b[0m\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", - "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Config\u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", - "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", - "โ”‚ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m) โ”‚ Vocab size: \u001b[38;5;34m256,000\u001b[0m โ”‚\n", - "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" - ], "text/html": [ "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
               "โ”ƒ Layer (type)                                                  โ”ƒ                                   Config โ”ƒ\n",
@@ -1817,41 +1745,33 @@
               "โ”‚ gemma_tokenizer (GemmaTokenizer)                              โ”‚                      Vocab size: 256,000 โ”‚\n",
               "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
               "
\n" + ], + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Config\u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ gemma_tokenizer (\u001b[38;5;33mGemmaTokenizer\u001b[0m) โ”‚ Vocab size: \u001b[38;5;34m256,000\u001b[0m โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n" - ], "text/html": [ "
Model: \"gemma_causal_lm\"\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1mModel: \"gemma_causal_lm\"\u001b[0m\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", - "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mConnected to \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", - "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", - "โ”‚ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", - "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", - "โ”‚ token_ids (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", - "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", - "โ”‚ gemma_backbone โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m) โ”‚ \u001b[38;5;34m2,617,270,528\u001b[0m โ”‚ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m], โ”‚\n", - "โ”‚ (\u001b[38;5;33mGemmaBackbone\u001b[0m) โ”‚ โ”‚ โ”‚ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", - "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", - "โ”‚ token_embedding โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m) โ”‚ \u001b[38;5;34m589,824,000\u001b[0m โ”‚ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", - "โ”‚ (\u001b[38;5;33mReversibleEmbedding\u001b[0m) โ”‚ โ”‚ โ”‚ โ”‚\n", - "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" - ], "text/html": [ "
โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n",
               "โ”ƒ Layer (type)                  โ”ƒ Output Shape              โ”ƒ         Param # โ”ƒ Connected to               โ”ƒ\n",
@@ -1867,93 +1787,109 @@
               "โ”‚ (ReversibleEmbedding)         โ”‚                           โ”‚                 โ”‚                            โ”‚\n",
               "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n",
               "
\n" + ], + "text/plain": [ + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mConnected to \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ padding_mask (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_ids (\u001b[38;5;33mInputLayer\u001b[0m) โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m) โ”‚ \u001b[38;5;34m0\u001b[0m โ”‚ - โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ gemma_backbone โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m2304\u001b[0m) โ”‚ \u001b[38;5;34m2,617,270,528\u001b[0m โ”‚ padding_mask[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m], โ”‚\n", + "โ”‚ (\u001b[38;5;33mGemmaBackbone\u001b[0m) โ”‚ โ”‚ โ”‚ token_ids[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค\n", + "โ”‚ token_embedding โ”‚ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256000\u001b[0m) โ”‚ \u001b[38;5;34m589,824,000\u001b[0m โ”‚ gemma_backbone[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m] โ”‚\n", + "โ”‚ (\u001b[38;5;33mReversibleEmbedding\u001b[0m) โ”‚ โ”‚ โ”‚ โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,617,270,528\u001b[0m (9.75 GB)\n" - ], "text/html": [ "
 Total params: 2,617,270,528 (9.75 GB)\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m2,617,270,528\u001b[0m (9.75 GB)\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n" - ], "text/html": [ "
 Trainable params: 2,928,640 (11.17 MB)\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m2,928,640\u001b[0m (11.17 MB)\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" - ], "text/html": [ "
 Non-trainable params: 2,614,341,888 (9.74 GB)\n",
               "
\n" + ], + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m2,614,341,888\u001b[0m (9.74 GB)\n" ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" } + ], + "source": [ + "# Enable LoRA for the model and set the LoRA rank to 4.\n", + "gemma_lm.backbone.enable_lora(rank=lora_rank)\n", + "gemma_lm.summary()\n", + "\n", + "# Limit the input sequence length (to control memory usage).\n", + "gemma_lm.preprocessor.sequence_length = token_limit\n", + "# Use AdamW (a common optimizer for transformer models).\n", + "optimizer = keras.optimizers.AdamW(\n", + " learning_rate=lr_value,\n", + " weight_decay=0.01,\n", + ")\n", + "# Exclude layernorm and bias terms from decay.\n", + "optimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n", + "\n", + "gemma_lm.compile(\n", + " loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", + " optimizer=optimizer,\n", + " weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n", + ")" ] }, { "cell_type": "markdown", + "metadata": { + "id": "nWaNHda-y65-" + }, "source": [ "Note that enabling LoRA reduces the number of trainable parameters significantly. In practice, we recommend beginning with a relatively small rank (such as 4, 8, 16). This is computationally efficient for experimentation.\n", "\n", "To monitor the learning progress, we will evaluate the model at the end of each epoch and save the all lora weights." - ], - "metadata": { - "id": "nWaNHda-y65-" - } + ] }, { "cell_type": "code", - "source": [ - "class CustomCallback(keras.callbacks.Callback):\n", - " def on_epoch_end(self, epoch, logs=None):\n", - " model_name = f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{epoch+1}.lora.h5\"\n", - " gemma_lm.backbone.save_lora_weights(model_name)\n", - "\n", - " # Evaluate\n", - " text_gen(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\")\n", - " text_gen(\"ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\")\n", - "\n", - "history = gemma_lm.fit(train, epochs=train_epoch, batch_size=2, callbacks=[CustomCallback()])\n", - "\n", - "import matplotlib.pyplot as plt\n", - "plt.plot(history.history['loss'])\n", - "plt.show()" - ], + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "bI4VOfBQy30W", - "outputId": "6997f428-6a86-4738-e2dd-fea66108912a" + "id": "bI4VOfBQy30W" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "Epoch 1/5\n", "\u001b[1m35/35\u001b[0m \u001b[32mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 3s/step - loss: 1.3578 - sparse_categorical_accuracy: 0.2155\n", @@ -2038,53 +1974,56 @@ ] }, { - "output_type": "display_data", "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7cUlEQVR4nO3de3iU9Z3//9fMJDOTkGQgBCYcAuGgUBQSiCRFd1e7jaUrtdLTUrDiN1vtlkV/2ly7LVktrHZr7NZaewlbXFarq4tQj+2uFsW06NoiwQQqIuCBQ8JhJgngTAhkAjP3749Jhgw5kAmZzCHPx3Xdl+bO55753Nd0zKuf+32/b5NhGIYAAABixBzrCQAAgKGNMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGIqJdYT6ItAIKCjR48qMzNTJpMp1tMBAAB9YBiGmpubNXbsWJnNPa9/JEQYOXr0qPLy8mI9DQAA0A/19fUaP358j79PiDCSmZkpKXgyWVlZMZ4NAADoC6/Xq7y8vNDf8Z4kRBjpuDSTlZVFGAEAIMFcrMSCAlYAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBT/Qoja9asUX5+vux2u0pKSlRdXd3j2LNnz+r+++/XlClTZLfbVVBQoE2bNvV7wgAAILlEHEY2btyo8vJyrVq1SrW1tSooKND8+fPV0NDQ7fh7771Xjz32mB599FF98MEH+u53v6uvfOUr2rFjxyVPHgAAJD6TYRhGJAeUlJRo7ty5Wr16taTgc2Py8vJ05513asWKFV3Gjx07Vvfcc4+WL18e2ve1r31NaWlpeuaZZ/r0nl6vVw6HQx6Ph6ZnAAAkiL7+/Y5oZaStrU01NTUqLS09/wJms0pLS7V169Zuj/H5fLLb7WH70tLS9Pbbb0fy1gAAIElFFEaamprk9/vldDrD9judTrlcrm6PmT9/vh5++GF99NFHCgQC2rx5s1588UUdO3asx/fx+Xzyer1hGwAASE5Rv5vmF7/4hS677DJNnz5dVqtVd9xxh8rKynp9lHBlZaUcDkdo44m9AAAkr4jCSE5OjiwWi9xud9h+t9ut3Nzcbo8ZNWqUXn75ZbW0tOjQoUPau3evMjIyNHny5B7fp6KiQh6PJ7TV19dHMs0+8QcM/c+fj6rsV9U65Ts34K8PAAD6JqIwYrVaVVRUpKqqqtC+QCCgqqoqzZs3r9dj7Xa7xo0bp3PnzumFF17QTTfd1ONYm80WekJvtJ7UazZJP9/8of6wr1G/3Xl0wF8fAAD0TcSXacrLy7Vu3To99dRT2rNnj5YtW6aWlhaVlZVJkpYuXaqKiorQ+G3btunFF1/U/v379X//93/64he/qEAgoO9///sDdxb9YDKZtLh4giTp2eq6mM4FAIChLCXSAxYtWqTGxkatXLlSLpdLhYWF2rRpU6iota6uLqwepLW1Vffee6/279+vjIwM3XDDDXr66ac1fPjwATuJ/vpa0Xj99LV92nXEo12HPZo53hHrKQEAMORE3GckFqLZZ+T/e3aHfvvno1pcPEGVX505oK8NAMBQFpU+I8loSUnwUs1vdx6hkBUAgBgY8mGkZFK2Jo8appY2P4WsAADEwJAPIyaTSUvaC1nXVx+K8WwAABh6hnwYkaSvzhkvq8Ws9494teuwJ9bTAQBgSCGMSMoeZtXfzAw2bWN1BACAwUUYadfRc+Q3O49SyAoAwCAijLTrKGQ93ebXb3YeifV0AAAYMggj7ToXstKRFQCAwUMY6eRrc8bLmhIsZH3v8Kexng4AAEMCYaSTEcOsuuHKYCErqyMAAAwOwsgFOheyNreejfFsAABIfoSRCxRPytaU9kLW3/6ZjqwAAEQbYeQCJpMptDqyfludEuA5ggAAJDTCSDc6Cll3H/Vq1xE6sgIAEE2EkW50LmRdv41CVgAAookw0oMlJRMlSb/9M4WsAABEE2GkB3PzR2jq6Iz2jqwUsgIAEC2EkR5QyAoAwOAgjPTia3PGyZpi1gfHvHrvMIWsAABEA2GkF8PTrVowc4wkOrICABAthJGL6LhUQyErAADRQRi5CApZAQCILsLIRVDICgBAdBFG+oBCVgAAoocw0gedC1npyAoAwMAijPTRkhIKWQEAiAbCSB9dNTFYyHrmrF8vU8gKAMCAIYz0kclk0hIKWQEAGHCEkQh8tb2Qdc8xr/5MISsAAAOCMBKB4elWfamjIyuFrAAADAjCSIQWdypk9VLICgDAJSOMROiqiSN0WXshKx1ZAQC4dP0KI2vWrFF+fr7sdrtKSkpUXV3d6/hHHnlE06ZNU1pamvLy8vS9731Pra2t/ZpwrNGRFQCAgRVxGNm4caPKy8u1atUq1dbWqqCgQPPnz1dDQ0O349evX68VK1Zo1apV2rNnjx5//HFt3LhR//zP/3zJk4+Vr84ZJxuFrAAADIiIw8jDDz+s22+/XWVlZZoxY4bWrl2r9PR0PfHEE92O/9Of/qRrrrlGS5YsUX5+vr7whS9o8eLFF11NiWfhHVkPxXg2AAAktojCSFtbm2pqalRaWnr+BcxmlZaWauvWrd0ec/XVV6umpiYUPvbv369XX31VN9xwQ4/v4/P55PV6w7Z409GR9X/+fIxCVgAALkFEYaSpqUl+v19OpzNsv9PplMvl6vaYJUuW6P7779df/MVfKDU1VVOmTNF1113X62WayspKORyO0JaXlxfJNAdFUedC1h1HYj0dAAASVtTvptmyZYseeOAB/fu//7tqa2v14osv6pVXXtGPfvSjHo+pqKiQx+MJbfX19dGeZsRMJlNodeS/KWQFAKDfUiIZnJOTI4vFIrfbHbbf7XYrNze322N++MMf6pZbbtFtt90mSZo5c6ZaWlr0ne98R/fcc4/M5q55yGazyWazRTK1mPjq7PF68Hd7tdfVrJ31n2r2hBGxnhIAAAknopURq9WqoqIiVVVVhfYFAgFVVVVp3rx53R5z+vTpLoHDYrFIUsKvJjjSU7VgVntH1mo6sgIA0B8RX6YpLy/XunXr9NRTT2nPnj1atmyZWlpaVFZWJklaunSpKioqQuNvvPFG/fKXv9SGDRt04MABbd68WT/84Q914403hkJJIut4eB6FrAAA9E9El2kkadGiRWpsbNTKlSvlcrlUWFioTZs2hYpa6+rqwlZC7r33XplMJt177706cuSIRo0apRtvvFE//vGPB+4sYqho4ghd7szQh+5T+s2OI7plXn6spwQAQEIxGQlwrcTr9crhcMjj8SgrKyvW0+niV388oPv+5wNNz83U7+76S5lMplhPCQCAmOvr32+eTTMAvjp7vGwp5lAhKwAA6DvCyADoXMi6fhuFrAAARIIwMkBu7ujI+t5Rec5QyAoAQF8RRgbInAnBQtbWswH9ZicdWQEA6CvCyAAxmUyh23zX05EVAIA+I4wMoK90KmTdQSErAAB9QhgZQI70VH1p1lhJFLICANBXhJEBtqQk+ITh/6WQFQCAPiGMDLA5E0ZomjNTrWcDenkHhawAAFwMYWSAmUwmLS4Oro48W00hKwAAF0MYiYKvzDlfyFpb92mspwMAQFwjjESBI+18Ieuz1RSyAgDQG8JIlCxp78hKISsAAL0jjETJnAnDKWQFAKAPCCNRYjKZQqsjdGQFAKBnhJEoWjh7nOypZu1zU8gKAEBPCCNR1LmQlY6sAAB0jzASZYuLOxWynqaQFQCACxFGomzOhOGanpsp37mAXtpxONbTAQAg7hBGoizYkTW4OvJsdT2FrAAAXIAwMgjCC1lPxno6AADEFcLIIAgvZK2P8WwAAIgvhJFBEtaRlUJWAABCCCODZHYehawAAHSHMDJIwjqyVtORFQCADoSRQXRTYbCQ9UP3KQpZAQBoRxgZRI60VN3YXsj633RkBQBAEmFk0C1uv1TzynvHKGQFAECEkUHXuZD1RQpZAQAgjAy2zoWsz1LICgAAYSQWOjqyfug+pZpDFLICAIa2foWRNWvWKD8/X3a7XSUlJaquru5x7HXXXSeTydRlW7BgQb8nneiy7OcLWddXU8gKABjaIg4jGzduVHl5uVatWqXa2loVFBRo/vz5amho6Hb8iy++qGPHjoW2999/XxaLRd/4xjcuefKJbAmFrAAASOpHGHn44Yd1++23q6ysTDNmzNDatWuVnp6uJ554otvx2dnZys3NDW2bN29Wenr6kA8jhRSyAgAgKcIw0tbWppqaGpWWlp5/AbNZpaWl2rp1a59e4/HHH9c3v/lNDRs2rMcxPp9PXq83bEs2JpNJN3d0ZN1GISsAYOiKKIw0NTXJ7/fL6XSG7Xc6nXK5XBc9vrq6Wu+//75uu+22XsdVVlbK4XCEtry8vEimmTBumj1OaakWfdRAISsAYOga1LtpHn/8cc2cOVPFxcW9jquoqJDH4wlt9fX1gzTDwZVlT9WNBWMkBVdHAAAYiiIKIzk5ObJYLHK73WH73W63cnNzez22paVFGzZs0Le//e2Lvo/NZlNWVlbYlqwWFwcv1fzvrmP69HRbjGcDAMDgiyiMWK1WFRUVqaqqKrQvEAioqqpK8+bN6/XY5557Tj6fT9/61rf6N9MkVZg3XJ8Zk6W2cwG9WHsk1tMBAGDQRXyZpry8XOvWrdNTTz2lPXv2aNmyZWppaVFZWZkkaenSpaqoqOhy3OOPP66FCxdq5MiRlz7rJEJHVgDAUJcS6QGLFi1SY2OjVq5cKZfLpcLCQm3atClU1FpXVyezOTzj7Nu3T2+//bZef/31gZl1krmpcKweeGWPPmo4pXcPndTc/OxYTwkAgEFjMhLg/4p7vV45HA55PJ6krR/5wfPvaeO79frq7HF6eFFhrKcDAMAl6+vfb55NEycWl1DICgAYmggjcaJgvEMzKGQFAAxBhJE4YTKZQqsj6ylkBQAMIYSROLKwcKzSUi36uL2QFQCAoYAwEkcy7an6csFYSXRkBQAMHYSRONPRc+QVClkBAEMEYSTOzOpUyPoChawAgCGAMBJn6MgKABhqCCNx6KbCsUq3BgtZtx+kkBUAkNwII3GocyHrs9UUsgIAkhthJE4tLj5fyHqyhUJWAEDyIozEqVnjHbpibHtH1h0UsgIAkhdhJE6ZTKbQ6sj6bYcoZAUAJC3CSBzrKGT9pLGFQlYAQNIijMSx8I6sh2I8GwAAooMwEuc6eo68+r6LQlYAQFIijMS5mePOF7K+UHs41tMBAGDAEUbiHB1ZAQDJjjCSAL5ccL6QtfrAiVhPBwCAAUUYSQCZ9lTdVEhHVgBAciKMJIiOniMUsgIAkg1hJEHMGj9cV46jkBUAkHwIIwkk1JGVQlYAQBIhjCSQmwrHKd1q0X4KWQEASYQwkkAybCmhQtb1FLICAJIEYSTBLCmeKEn63S4KWQEAyYEwkmBmjncEC1n9FLICAJIDYSQBdayOUMgKAEgGhJEE9OXCsRrWXsi6jUJWAECCI4wkoAxbir5cOE4SHVkBAImPMJKglrT3HPndLpdOUMgKAEhg/Qoja9asUX5+vux2u0pKSlRdXd3r+E8//VTLly/XmDFjZLPZdPnll+vVV1/t14QRNHO8QzPHOdTmD+hFClkBAAks4jCyceNGlZeXa9WqVaqtrVVBQYHmz5+vhoaGbse3tbXp+uuv18GDB/X8889r3759WrduncaNG3fJkx/q6MgKAEgGJiPCv2IlJSWaO3euVq9eLUkKBALKy8vTnXfeqRUrVnQZv3btWv30pz/V3r17lZqa2q9Jer1eORwOeTweZWVl9es1ktEp3zmV/PgNtbT5teE7n9VnJ4+M9ZQAAAjp69/viFZG2traVFNTo9LS0vMvYDartLRUW7du7faY3/72t5o3b56WL18up9OpK6+8Ug888ID8fn+P7+Pz+eT1esM2dNW5kHX9NgpZAQCJKaIw0tTUJL/fL6fTGbbf6XTK5XJ1e8z+/fv1/PPPy+/369VXX9UPf/hD/exnP9O//uu/9vg+lZWVcjgcoS0vLy+SaQ4pN5cEL9Vsep9CVgBAYor63TSBQECjR4/Wf/zHf6ioqEiLFi3SPffco7Vr1/Z4TEVFhTweT2irr6+P9jQT1pXjzheyvlBDISsAIPFEFEZycnJksVjkdrvD9rvdbuXm5nZ7zJgxY3T55ZfLYrGE9n3mM5+Ry+VSW1v3/0/eZrMpKysrbEPPlrSvjjxLISsAIAFFFEasVquKiopUVVUV2hcIBFRVVaV58+Z1e8w111yjjz/+WIFAILTvww8/1JgxY2S1Wvs5bXR2Y0F7R9amFr2zn46sAIDEEvFlmvLycq1bt05PPfWU9uzZo2XLlqmlpUVlZWWSpKVLl6qioiI0ftmyZTpx4oTuuusuffjhh3rllVf0wAMPaPny5QN3FkNchi1FN81uL2SlIysAIMGkRHrAokWL1NjYqJUrV8rlcqmwsFCbNm0KFbXW1dXJbD6fcfLy8vTaa6/pe9/7nmbNmqVx48bprrvu0g9+8IOBOwtoSfEErd9Wp9fed+n4KZ9GZthiPSUAAPok4j4jsUCfkb758uq39d5hj/75hun6zl9NifV0AABDXFT6jCC+dXRkfba6nkJWAEDCIIwkkS+3F7IeaGrR1v3HYz0dAAD6hDCSRIZ1KmR9tpreLACAxEAYSTJLijs6sh7T8VO+GM8GAICLI4wkmSvHOTRrvENn/YZeqKUjKwAg/hFGktASClkBAAmEMJKEbiwYqwxbCoWsAICEQBhJQsNsKbqpcKwkaf02OrICAOIbYSRJdfQceW23i0JWAEBcI4wkqSvHOVTQXsj6fA2FrACA+EUYSWLnO7LWUcgKAIhbhJEk1lHIevD4aW39hEJWAEB8IowksbBC1moKWQEA8YkwkuSWlJwvZG2ikBUAEIcII0nuirHnC1lfoJAVABCHCCNDQMfqyLPVdQoEKGQFAMQXwsgQ8KVZ5wtZ36EjKwAgzhBGhoBhthQtnB0sZP1vClkBAHGGMDJEdPQceZ1CVgBAnCGMDBFXjHWoIG84HVkBAHGHMDKELCnOkyRtoJAVABBHCCNDSFhHVgpZAQBxgjAyhKRbzxey0pEVABAvCCNDzJLiiZIoZAUAxA/CyBAzY2wWhawAgLhCGBmCbi6mIysAIH4QRoagLxWMUaYtRYcoZAUAxAHCyBAULGQdJ0lav41CVgBAbBFGhqiOjqyv7XapsZlCVgBA7BBGhqgZY7NUmDdc5wIUsgIAYoswMoQtKQmujmzYTiErACB2+hVG1qxZo/z8fNntdpWUlKi6urrHsU8++aRMJlPYZrfb+z1hDJwvzTpfyPqnTyhkBQDERsRhZOPGjSovL9eqVatUW1urgoICzZ8/Xw0NDT0ek5WVpWPHjoW2Q4cOXdKkMTDSrSn6ypxgIeuzdGQFAMRIxGHk4Ycf1u23366ysjLNmDFDa9euVXp6up544okejzGZTMrNzQ1tTqfzkiaNgUMhKwAg1iIKI21tbaqpqVFpaen5FzCbVVpaqq1bt/Z43KlTpzRx4kTl5eXppptu0u7du3t9H5/PJ6/XG7YhOj4zJkuzJ1DICgCInYjCSFNTk/x+f5eVDafTKZfL1e0x06ZN0xNPPKHf/OY3euaZZxQIBHT11Vfr8OGe//BVVlbK4XCEtry8vEimiQgtpiMrACCGon43zbx587R06VIVFhbq2muv1YsvvqhRo0bpscce6/GYiooKeTye0FZfXx/taQ5pN84aq0x7iupOUMgKABh8EYWRnJwcWSwWud3usP1ut1u5ubl9eo3U1FTNnj1bH3/8cY9jbDabsrKywjZET5rVoq90dGStprgYADC4IgojVqtVRUVFqqqqCu0LBAKqqqrSvHnz+vQafr9fu3bt0pgxYyKbKaKqo+fI67vdFLICAAZVxJdpysvLtW7dOj311FPas2ePli1bppaWFpWVlUmSli5dqoqKitD4+++/X6+//rr279+v2tpafetb39KhQ4d02223DdxZ4JJNzz1fyPpcDZfFAACDJyXSAxYtWqTGxkatXLlSLpdLhYWF2rRpU6iota6uTmbz+Yxz8uRJ3X777XK5XBoxYoSKior0pz/9STNmzBi4s8CAWFI8QTvqPtWG6np996+myGw2xXpKAIAhwGQYRtzfPuH1euVwOOTxeKgfiaIzbX4VP/CGmlvP6elvF+svLxsV6ykBABJYX/9+82wahKRZLfrqbDqyAgAGF2EEYRZ3KmRtaG6N8WwAAEMBYQRhpudmaQ4dWQEAg4gwgi46OrJuqK6nIysAIOoII+jiS506sv7xk6ZYTwcAkOQII+iicyHr+m0UsgIAooswgm51FLJu/oBCVgBAdBFG0K3OhazPvUshKwAgeggj6NGSkomSpA3b6yhkBQBEDWEEPVowc4wy7SmqP3FGb39MISsAIDoII+hRmtWir80ZL4mOrACA6CGMoFcdPUcoZAUARAthBL2alpupookjKGQFAEQNYQQXFerISiErACAKCCO4qC/NGqMsClkBAFFCGMFF2VMt+mp7ISsdWQEAA40wgj7puFTzxh63GrwUsgIABg5hBH0SVshaQyErAGDgEEbQZ0vaV0eeraaQFQAwcAgj6LMF7YWsh0+e0f9RyAoAGCCEEfRZ50LWZylkBQAMEMIIIrKkpL0jK4WsAIABQhhBRC53ZuqqiSPkp5AVADBACCOI2GIKWQEAA4gwgohRyAoAGEiEEUQsvCProRjPBgCQ6Agj6JeOQtY39jRQyAoAuCSEEfRL50LWX79bH+vpAAASGGEE/daxOvJsdT2FrACAfiOMoN9umDlGjrRUHfn0jN76qDHW0wEAJCjCCPotWMg6TlLwNl8AAPqjX2FkzZo1ys/Pl91uV0lJiaqrq/t03IYNG2QymbRw4cL+vC3iUMfD897Y0yA3hawAgH6IOIxs3LhR5eXlWrVqlWpra1VQUKD58+eroaGh1+MOHjyof/zHf9Rf/uVf9nuyiD+XOTM1N7+9kHU7hawAgMhFHEYefvhh3X777SorK9OMGTO0du1apaen64knnujxGL/fr5tvvln33XefJk+efEkTRvzp6Mi6YXu9/BSyAgAiFFEYaWtrU01NjUpLS8+/gNms0tJSbd26tcfj7r//fo0ePVrf/va3+/Q+Pp9PXq83bEP8opAVAHApIgojTU1N8vv9cjqdYfudTqdcLle3x7z99tt6/PHHtW7duj6/T2VlpRwOR2jLy8uLZJoYZGGFrNsoZAUARCaqd9M0Nzfrlltu0bp165STk9Pn4yoqKuTxeEJbfT21CPGuo5C1ai+FrACAyKREMjgnJ0cWi0Vutztsv9vtVm5ubpfxn3zyiQ4ePKgbb7wxtC8QCATfOCVF+/bt05QpU7ocZ7PZZLPZIpkaYqyjkHX7wZP69fZ63fn5y2I9JQBAgohoZcRqtaqoqEhVVVWhfYFAQFVVVZo3b16X8dOnT9euXbu0c+fO0PblL39Zn/vc57Rz504uvySZjo6sFLICACIR0cqIJJWXl+vWW2/VVVddpeLiYj3yyCNqaWlRWVmZJGnp0qUaN26cKisrZbfbdeWVV4YdP3z4cEnqsh+J72+uHKN/+e0HoULWz00bHespAQASQMRhZNGiRWpsbNTKlSvlcrlUWFioTZs2hYpa6+rqZDbT2HUosqda9LU54/XEHw9o/bY6wggAoE9MhmHE/Xq61+uVw+GQx+NRVlZWrKeDXnzc0KzSh9+SxWzSH3/w18p12GM9JQBAjPT17zdLGBhQU0dnqjg/O9iR9V3uggIAXBxhBANucUmwMHkjhawAgD4gjGDA/c2VnTqyfkhHVgBA7wgjGHAdhayStL6ajqwAgN4RRhAVS9ov1fx+b4NcHjqyAgB6RhhBVFDICgDoK8IIoibUkbW6jkJWAECPCCOImi9emavh6ak66mmlkBUA0CPCCKKmcyHrf2+jkBUA0D3CCKJqcXFHIaubQlYAQLcII4iqqaMzVTwpWwEj2AQNAIALEUYQdUuKg4WsG7dTyAoA6IowgqjrXMj65ocNsZ4OACDOEEYQdWEdWbdxqQYAEI4wgkGxuP1Sze/3unXMcybGswEAxBPCCAbF1NEZoULWX28/HOvpAADiCGEEg+bmEgpZAQBdEUYwaOZfkasRFLICAC5AGMGgCS9kpSMrACCIMIJBtbiko5C1gUJWAIAkwggG2ZRRGSqhIysAoBPCCAbdklAhaz2FrAAAwggG3xevDBayHvO0ass+ClkBYKgjjGDQ2VIs+npRsJD12WoKWQFgqCOMICa+WXy+kPXopxSyAsBQRhhBTEwZlaHPTm7vyPouhawAMJQRRhAzHc+r2bi9Xuf8gRjPBgAQK4QRxEznQtY3P2yM9XQAADFCGEHMdC5kpSMrAAxdhBHEVMelmj/so5AVAIYqwghianKnQlY6sgLA0NSvMLJmzRrl5+fLbrerpKRE1dXVPY598cUXddVVV2n48OEaNmyYCgsL9fTTT/d7wkg+S0omSgreVUMhKwAMPRGHkY0bN6q8vFyrVq1SbW2tCgoKNH/+fDU0dN9JMzs7W/fcc4+2bt2q9957T2VlZSorK9Nrr712yZNHcph/hVPZw6ztHVkpZAWAocZkGEZEDwcpKSnR3LlztXr1aklSIBBQXl6e7rzzTq1YsaJPrzFnzhwtWLBAP/rRj/o03uv1yuFwyOPxKCsrK5LpIkE88Ooe/cdb+/X56aP1+P+bG+vpAAAGQF//fke0MtLW1qaamhqVlpaefwGzWaWlpdq6detFjzcMQ1VVVdq3b5/+6q/+qsdxPp9PXq83bENy++bcPEkUsgLAUBRRGGlqapLf75fT6Qzb73Q65XK5ejzO4/EoIyNDVqtVCxYs0KOPPqrrr7++x/GVlZVyOByhLS8vL5JpIgFNHpWheZNHUsgKAEPQoNxNk5mZqZ07d2r79u368Y9/rPLycm3ZsqXH8RUVFfJ4PKGtvp4/TkPB4hI6sgLAUJQSyeCcnBxZLBa53e6w/W63W7m5uT0eZzabNXXqVElSYWGh9uzZo8rKSl133XXdjrfZbLLZbJFMDUmgo5DV5Q0WspbOcF78IABAwotoZcRqtaqoqEhVVVWhfYFAQFVVVZo3b16fXycQCMjn80Xy1hgCwjqyVtORFQCGiogv05SXl2vdunV66qmntGfPHi1btkwtLS0qKyuTJC1dulQVFRWh8ZWVldq8ebP279+vPXv26Gc/+5mefvppfetb3xq4s0DS6Chk3bKvQUcoZAWAISGiyzSStGjRIjU2NmrlypVyuVwqLCzUpk2bQkWtdXV1MpvPZ5yWlhb9wz/8gw4fPqy0tDRNnz5dzzzzjBYtWjRwZ4Gk0VHIunX/cW3cXq/y6y+P9ZQAAFEWcZ+RWKDPyNDyP38+qjuf3aHcLLve/sHnlGLhqQUAkIii0mcEGAxf6FTI+gc6sgJA0iOMIO7YUiz6Rnsh67MUsgJA0iOMIC59szjYc4RCVgBIfoQRxKVJOcN09RQ6sgLAUEAYQdxaXNzRkbWOjqwAkMQII4hb86/I1chhVrm9PgpZASCJEUYQt6wp5vMdWbcdivFsAADRQhhBXAsVsn7YSCErACQpwgjiWkchq2FIG7nNFwCSEmEEcW9JSXsh67v1FLICQBIijCDufWHG+ULW3+9tiPV0AAADjDCCuGdNMevrV9GRFQCSFWEECWHx3POFrIdPno7xbAAAA4kwgoSQnzNM10wNFrL+mo6sAJBUCCNIGKGOrBSyAkBSIYwgYVDICgDJiTCChNG5kHU9hawAkDQII0goHYWsb1LICgBJgzCChNK5kHUjhawAkBQII0g4S4onSgqGEQpZASDxEUaQcK6f4VROhlUNzT5VUcgKAAmPMIKEY00x6+tFeZLoyAoAyYAwgoT0zbnBMPLmh42qP0EhKwAkMsIIElJ+zjD9xdScYEfWdylkBYBElhLrCQD9tbh4gt7+uEn/8dZ+vX/Eo5LJI1UyKVtXjnMo1ULOBoBEQRhBwrp+hlPTnJna527WH/Y16g/7GiVJ6VaLiiaOUMmkbJVMHqlZ4x2ypVhiPFsAQE9MhmEYsZ7ExXi9XjkcDnk8HmVlZcV6Oogj/oChPce8emf/cW07cELbD57Qp6fPho2xpZg1e8JwlUwaqZLJ2ZozYYTsqYQTAIi2vv79JowgqQQChj5saNa2/Se07cBxVR84oaZTbWFjUi0mFYwfrpLJ2SqZNFJFE0domI1FQgAYaIQRQJJhGPqksUXbDhwPBRS31xc2xmI26cpxDn12UrZKJmeraGK2HGmpMZoxACQPwgjQDcMwVHfitLbtP6F32ldODp88EzbGZJJmjMkKXdYpzs/WiGHWGM0YABIXYQTooyOfntG2/cGVk+qDJ3SgqaXLmGnOzGAwmRS8tDMq0xaDmQJAYolqGFmzZo1++tOfyuVyqaCgQI8++qiKi4u7Hbtu3Tr913/9l95//31JUlFRkR544IEex3eHMILB5Pa2atuBE9q2P7hy8lHDqS5jJo8aFlw5ab+0M8aRFoOZAkB8i1oY2bhxo5YuXaq1a9eqpKREjzzyiJ577jnt27dPo0eP7jL+5ptv1jXXXKOrr75adrtdP/nJT/TSSy9p9+7dGjdu3ICeDBANx0/5VH3gRDCgHDihvS6vLvzWTMhOV8mk4MrJZyeP1PgRaTKZTLGZMADEiaiFkZKSEs2dO1erV6+WJAUCAeXl5enOO+/UihUrLnq83+/XiBEjtHr1ai1durRP70kYQTzxnD6r6oMnVH0geDvx+0c8ClzwLRrrsAcv6bQ3YpuUM4xwAmDI6evf74juZ2xra1NNTY0qKipC+8xms0pLS7V169Y+vcbp06d19uxZZWdn9zjG5/PJ5zt/x4PX641kmkBUOdJTdf0Mp66f4ZQkNbee1buHTgZXT/Yf13uHPTrqadXLO4/q5Z1HJUmjMm3BVZP2gDJ1VIbMZsIJAEgRhpGmpib5/X45nc6w/U6nU3v37u3Ta/zgBz/Q2LFjVVpa2uOYyspK3XfffZFMDYiZTHuqPjdttD43LXiZ8nTbOe2o+1Tb9h/XOwdOaGf9p2ps9umV947plfeOSZKyh1k1N39E6I6d6blZshBOAAxRg9rp6cEHH9SGDRu0ZcsW2e32HsdVVFSovLw89LPX61VeXt5gTBG4ZOnWFF0zNUfXTM2RJLWe9evP9Z+215wcV82hkzrR0qbXdrv12m63JCnLnqK5+dmhRmxXjM1SCs/XATBERBRGcnJyZLFY5Ha7w/a73W7l5ub2euxDDz2kBx98UG+88YZmzZrV61ibzSabjVsnkRzsqZZg7cjkkZIuU9u5gHYd8YQasb178IS8redUtbdBVXsbJEnDrBYV5WerZFK2Pjs5WzPHDZc1hXACIDlFFEasVquKiopUVVWlhQsXSgoWsFZVVemOO+7o8bh/+7d/049//GO99tpruuqqqy5pwkCis6aYVTRxhIomjtA/XCed8wf0wTFvWAt7b+s5vfVho976MPjwP3uqWXMmBC/rFE/K1uwJw3m+DoCk0a9be2+99VY99thjKi4u1iOPPKJf//rX2rt3r5xOp5YuXapx48apsrJSkvSTn/xEK1eu1Pr163XNNdeEXicjI0MZGRl9ek/upsFQ4g8Y2udqDq2cVB88oRMt4c/XsVrMKswbHmrEVjRxhNKtPF8HQHyJyt00krRo0SI1NjZq5cqVcrlcKiws1KZNm0JFrXV1dTKbzy8n//KXv1RbW5u+/vWvh73OqlWr9C//8i+Rvj2Q9Cxmk2aMzdKMsVkqu2aSDMPQxw2n9E773TrbDpxQY7MveHvxwROSpBSzSTPHO0KN2K7KH6FMO8/XAZAYaAcPJBjDMHTw+OlQMNm2/7iOelrDxphN0hVjHaFGbMWTsjU8nefrABhcPJsGGELqT5w+38L+4AkdOn467PcmU/vzddr7nBRPylZOBkXiAKKLMAIMYcc8Z863sN9/XJ80dn3439TRGe0P/gu2sHdm9Xy7PQD0B2EEQEhjc8fzdYJ36+x1NXcZM3Fk8Pk6HY3Yxo9Ij8FMASQTwgiAHp1saWt/vk4woHxw1Nvl+TrjhqeFnkpcMmmkJo5M5/k6ACJCGAHQZ97Ws3r3YMdlnRPadcQj/wXpxJllU3H73Tolk7I1dXQG4QRArwgjAPqtxXdONR0P/ztwXH+u96jNHwgbM3KYNVRzUjxppKbnZvLwPwBhCCMABkzrWb9q6zqeTHxCtXUn5TsXHk4caanB5+u0X9qZMYbn6wBDHWEEQNT4zvm167BH2w6c0Dv7gw//O93mDxuTYUvRVfkj2ldPRmrWeIdSCSfAkEIYATBozvoD2n3UG2rEtv3ACTX7zoWNSUu1qGjiiNClnYI8nq8DJDvCCICY8QcM7TnmDWvE9unps2FjrClmzc4brpJJ2Zo+JksTstM1YWS6smhjDyQNwgiAuBEIGPqo4VTo4X/bDhxX06m2bscOT08NBpMLt5HpGuNIk4UiWSBhEEYAxC3DMLS/qUXb9p/QuwdP6MDxFtWfON1jQOmQYjZp/Ig05bUHlIkjg//s+JmHAwLxhTACIOG0+M6p7sRp1Z04rfr2fx46Hvz3+pOnddbf+3+usodZQ8FkQnaaJmYPC/48Ml25WXZWVYBB1te/3ymDOCcA6NUwW4o+MyZLnxnT9T9a/oAht7c1FE7qLthOtLSFtj/Xf9rl+FSLSeNHhF/6yWtfXcnLTleGjf8cArHCtw9AQrCYTRo7PE1jh6dp3pSRXX7f3HpW9SfOtIeTlvZ/nlH9idM63L6qcqCpRQeauj40UAo2cet8+SevU2jJzbLT0A2IIsIIgKSQaU/VjLGpmjG2+1WVY54z3V7+qTtxWidPn9XxljYdb2nTzm5WVawWs8Znp3VbVJs3Il3DWFUBLgnfIABJz2IOXqIZPyJdmtL1997Ws6rr4fLPkZNn1OYPaH9ji/Y3dr+qkpNh7ebyzzBNyE7X6EwbqyrARVDACgC9OOcP6JinVfUnTutQp5DSEVwu7J9yIWuKWXkj0tov/wwLu/wzITtdaVYavyF5UcAKAAMgxWJWXvtqx9Xd/N5z5mzYikrnyz9HPj2jtnMBfdLYok8aWyQ1djl+VKYtfEWl/fLPhOx0jcpgVQVDAysjABAl5/wBHf20NeyyT3CFpUWHjp9Wc+u5Xo+3pZjDgsqFvVVop494x8oIAMRYisUcXOUYmd7t7z2nzwZXU9rv/qm/oFbFdy6gjxpO6aOGU90eP7pjVWVk18LaURk2mUysqiAxsDICAHHorD+go5+eOb+qcvx02L9f+CDCC9lTzT1e/hk/glUVDA5WRgAggaVazJo4cpgmjhzW5XeGYejT9lWVsMs/7YHlmOeMWs8G9KH7lD50d7+qkptlD7v8M2FkmiZkB+8AysmwsqqCQcXKCAAkmbZzwVWVQ53v/Om0snLqIqsqaamWHi//jB+RJlsKqyroG1ZGAGCIsqaYlZ8zTPk53a+qnOy8qnK8pdPqyhkd9ZzRmbN+7XM3a5+7ucvxJlNwVSVUUNu+upLrsCs3yy5nlp3blRExVkYAACG+c34dOXmmy6Wfju10m/+ir5FlT1GuIxhMnFntIaU9rAQDi00jM2w8uHAIYGUEABAxW4pFk0dlaPKojC6/MwxDJ1radOiCSz/1J0/L7fXJ5WnVmbN+eVvPydvac72KFOyKOzrT1h5YbGGBJRRiHHYeYDhE8CkDAPrEZDJpZEZwVWPOhBFdfm8Yhryt5+T2tsrtbZXL0/5Pb6vcXl9oX9MpX/vzglp1zNPa63tm2FI0uj2shAcWWyiwjMqwKcVijtZpYxAQRgAAA8JkMsmRlipHWqoud2b2OO6cP6CmU21ydQos50NLcF+D16dm3zmd8p3TqcZzPT4XKPi+Uk6GLbSqkuuwyZnZ6dKQwy5npl1ZaSncJRSnCCMAgEGVYjEHC14ddimv53EtvnPBgOJplbu5VS7P+dUVd3Nwf0OzT+cChhqbfWps9mnXEU+Pr2dPNXe5DNRR05LrsGl0ZvBnawqrLIONMAIAiEvDbCmaMipDU7qpX+kQCBhqavGpob1mpWN1JbjS4pO7fZ/nzFm1ng3o4PHTOnj8dK/vO3KY9XwtS6fA0jnEjEhPZZVlAPUrjKxZs0Y//elP5XK5VFBQoEcffVTFxcXdjt29e7dWrlypmpoaHTp0SD//+c919913X8qcAQCQJJnNJo3OtGt0pl1XjnP0OK71rD+0quLyBi8DuTouDYVWWnxq8wd0vKVNx1va9MGxnt/XajGHalm61LF0WnWh023fRBxGNm7cqPLycq1du1YlJSV65JFHNH/+fO3bt0+jR4/uMv706dOaPHmyvvGNb+h73/vegEwaAIBI2FMtPXa07dDRg6XzZSBXWDFu8DLR8ZY2tfkDOnzyjA6fPNPr+zrSUjsFFlvY7c65DrtGZ9mUM4ynM0fcZ6SkpERz587V6tWrJUmBQEB5eXm68847tWLFil6Pzc/P19133x3xygh9RgAA8cJ3zq8Gr08N7XUsYZeGOt1B1Ho20KfXS+m4zbm90DZ0aeiCQtxhCXibc1T6jLS1tammpkYVFRWhfWazWaWlpdq6dWv/ZwsAQIKwpViU1955tiedb3M+f2motf0OIl8ovDSeChbgHvW06uhFbnPO7LjNuYc6ltwsu3IyrAl5m3NEYaSpqUl+v19OpzNsv9Pp1N69ewdsUj6fTz6fL/Sz1+sdsNcGACDaIrnNufGUL+wyUPgdRMH9p3zn1Ow7p+bGc/qkl9uczR23OYcFFltYYBmdZVeWPb5uc47LNZ/Kykrdd999sZ4GAABRlWIxa4wjTWMcab2OO+VrbybXvsoSKsLtVNfS0BxsJtfQ7FNDs09Sz7c5p6Va2gPL+aLbxcUTun2e0WCIKIzk5OTIYrHI7XaH7Xe73crNzR2wSVVUVKi8vDz0s9frVV5eLzejAwCQxDJsKcq4yG3O/oCh4y0+ubupY+l8B5HnzFmdOevXgaYWHWg6v8ryhStyEyOMWK1WFRUVqaqqSgsXLpQULGCtqqrSHXfcMWCTstlsstlsA/Z6AAAkO0un25xnqufbnM+0+bvpeuvTxJE918BEW8SXacrLy3XrrbfqqquuUnFxsR555BG1tLSorKxMkrR06VKNGzdOlZWVkoJFrx988EHo348cOaKdO3cqIyNDU6dOHcBTAQAAF5NmtSg/Z1jMVkG6E3EYWbRokRobG7Vy5Uq5XC4VFhZq06ZNoaLWuro6mc3nK3mPHj2q2bNnh35+6KGH9NBDD+naa6/Vli1bLv0MAABAQou4z0gs0GcEAIDE09e/34l3MzIAAEgqhBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBTET8oLxY6Hp/j9XpjPBMAANBXHX+3L/YYvIQII83NzZKkvLy8GM8EAABEqrm5WQ6Ho8ffJ8RTewOBgI4eParMzEyZTKYBe12v16u8vDzV19cn7dOAk/0cOb/El+znyPklvmQ/x2ien2EYam5u1tixY2U291wZkhArI2azWePHj4/a62dlZSXl/8A6S/Zz5PwSX7KfI+eX+JL9HKN1fr2tiHSggBUAAMQUYQQAAMTUkA4jNptNq1atks1mi/VUoibZz5HzS3zJfo6cX+JL9nOMh/NLiAJWAACQvIb0yggAAIg9wggAAIgpwggAAIgpwggAAIippA8ja9asUX5+vux2u0pKSlRdXd3r+Oeee07Tp0+X3W7XzJkz9eqrrw7STPsvknN88sknZTKZwja73T6Is43MW2+9pRtvvFFjx46VyWTSyy+/fNFjtmzZojlz5shms2nq1Kl68sknoz7P/or0/LZs2dLl8zOZTHK5XIMz4QhVVlZq7ty5yszM1OjRo7Vw4ULt27fvosclyvewP+eXaN/BX/7yl5o1a1aoIda8efP0u9/9rtdjEuXzkyI/v0T7/C704IMPymQy6e677+513GB/hkkdRjZu3Kjy8nKtWrVKtbW1Kigo0Pz589XQ0NDt+D/96U9avHixvv3tb2vHjh1auHChFi5cqPfff3+QZ953kZ6jFOyyd+zYsdB26NChQZxxZFpaWlRQUKA1a9b0afyBAwe0YMECfe5zn9POnTt1991367bbbtNrr70W5Zn2T6Tn12Hfvn1hn+Ho0aOjNMNL8+abb2r58uV65513tHnzZp09e1Zf+MIX1NLS0uMxifQ97M/5SYn1HRw/frwefPBB1dTU6N1339Vf//Vf66abbtLu3bu7HZ9In58U+flJifX5dbZ9+3Y99thjmjVrVq/jYvIZGkmsuLjYWL58eehnv99vjB071qisrOx2/N/+7d8aCxYsCNtXUlJi/P3f/31U53kpIj3HX/3qV4bD4Rik2Q0sScZLL73U65jvf//7xhVXXBG2b9GiRcb8+fOjOLOB0Zfz+8Mf/mBIMk6ePDkocxpoDQ0NhiTjzTff7HFMIn4PO/Tl/BL5O9hhxIgRxn/+5392+7tE/vw69HZ+ifr5NTc3G5dddpmxefNm49prrzXuuuuuHsfG4jNM2pWRtrY21dTUqLS0NLTPbDartLRUW7du7faYrVu3ho2XpPnz5/c4Ptb6c46SdOrUKU2cOFF5eXkX/X8AiSbRPsP+Kiws1JgxY3T99dfrj3/8Y6yn02cej0eSlJ2d3eOYRP4M+3J+UuJ+B/1+vzZs2KCWlhbNmzev2zGJ/Pn15fykxPz8li9frgULFnT5bLoTi88wacNIU1OT/H6/nE5n2H6n09nj9XWXyxXR+FjrzzlOmzZNTzzxhH7zm9/omWeeUSAQ0NVXX63Dhw8PxpSjrqfP0Ov16syZMzGa1cAZM2aM1q5dqxdeeEEvvPCC8vLydN1116m2tjbWU7uoQCCgu+++W9dcc42uvPLKHscl2vewQ1/PLxG/g7t27VJGRoZsNpu++93v6qWXXtKMGTO6HZuIn18k55eIn9+GDRtUW1urysrKPo2PxWeYEE/txcCZN29eWOK/+uqr9ZnPfEaPPfaYfvSjH8VwZuiLadOmadq0aaGfr776an3yySf6+c9/rqeffjqGM7u45cuX6/3339fbb78d66lERV/PLxG/g9OmTdPOnTvl8Xj0/PPP69Zbb9Wbb77Z4x/sRBPJ+SXa51dfX6+77rpLmzdvjutC26QNIzk5ObJYLHK73WH73W63cnNzuz0mNzc3ovGx1p9zvFBqaqpmz56tjz/+OBpTHHQ9fYZZWVlKS0uL0ayiq7i4OO7/wN9xxx363//9X7311lsaP358r2MT7XsoRXZ+F0qE76DVatXUqVMlSUVFRdq+fbt+8Ytf6LHHHusyNhE/v0jO70Lx/vnV1NSooaFBc+bMCe3z+/166623tHr1avl8PlkslrBjYvEZJu1lGqvVqqKiIlVVVYX2BQIBVVVV9XgtcN68eWHjJWnz5s29XjuMpf6c44X8fr927dqlMWPGRGuagyrRPsOBsHPnzrj9/AzD0B133KGXXnpJv//97zVp0qSLHpNIn2F/zu9CifgdDAQC8vl83f4ukT6/nvR2fheK98/v85//vHbt2qWdO3eGtquuuko333yzdu7c2SWISDH6DKNWGhsHNmzYYNhsNuPJJ580PvjgA+M73/mOMXz4cMPlchmGYRi33HKLsWLFitD4P/7xj0ZKSorx0EMPGXv27DFWrVplpKamGrt27YrVKVxUpOd43333Ga+99prxySefGDU1NcY3v/lNw263G7t3747VKfSqubnZ2LFjh7Fjxw5DkvHwww8bO3bsMA4dOmQYhmGsWLHCuOWWW0Lj9+/fb6Snpxv/9E//ZOzZs8dYs2aNYbFYjE2bNsXqFHoV6fn9/Oc/N15++WXjo48+Mnbt2mXcddddhtlsNt54441YnUKvli1bZjgcDmPLli3GsWPHQtvp06dDYxL5e9if80u07+CKFSuMN9980zhw4IDx3nvvGStWrDBMJpPx+uuvG4aR2J+fYUR+fon2+XXnwrtp4uEzTOowYhiG8eijjxoTJkwwrFarUVxcbLzzzjuh31177bXGrbfeGjb+17/+tXH55ZcbVqvVuOKKK4xXXnllkGccuUjO8e677w6NdTqdxg033GDU1tbGYNZ903Er64VbxzndeuutxrXXXtvlmMLCQsNqtRqTJ082fvWrXw36vPsq0vP7yU9+YkyZMsWw2+1Gdna2cd111xm///3vYzP5Puju3CSFfSaJ/D3sz/kl2nfw7/7u74yJEycaVqvVGDVqlPH5z38+9IfaMBL78zOMyM8v0T6/7lwYRuLhMzQZhmFEb90FAACgd0lbMwIAABIDYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMTU/w+ZuvZWqANXDwAAAABJRU5ErkJggg==\n", "text/plain": [ "
" - ], - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7cUlEQVR4nO3de3iU9Z3//9fMJDOTkGQgBCYcAuGgUBQSiCRFd1e7jaUrtdLTUrDiN1vtlkV/2ly7LVktrHZr7NZaewlbXFarq4tQj+2uFsW06NoiwQQqIuCBQ8JhJgngTAhkAjP3749Jhgw5kAmZzCHPx3Xdl+bO55753Nd0zKuf+32/b5NhGIYAAABixBzrCQAAgKGNMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGKKMAIAAGIqJdYT6ItAIKCjR48qMzNTJpMp1tMBAAB9YBiGmpubNXbsWJnNPa9/JEQYOXr0qPLy8mI9DQAA0A/19fUaP358j79PiDCSmZkpKXgyWVlZMZ4NAADoC6/Xq7y8vNDf8Z4kRBjpuDSTlZVFGAEAIMFcrMSCAlYAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBT/Qoja9asUX5+vux2u0pKSlRdXd3j2LNnz+r+++/XlClTZLfbVVBQoE2bNvV7wgAAILlEHEY2btyo8vJyrVq1SrW1tSooKND8+fPV0NDQ7fh7771Xjz32mB599FF98MEH+u53v6uvfOUr2rFjxyVPHgAAJD6TYRhGJAeUlJRo7ty5Wr16taTgc2Py8vJ05513asWKFV3Gjx07Vvfcc4+WL18e2ve1r31NaWlpeuaZZ/r0nl6vVw6HQx6Ph6ZnAAAkiL7+/Y5oZaStrU01NTUqLS09/wJms0pLS7V169Zuj/H5fLLb7WH70tLS9Pbbb0fy1gAAIElFFEaamprk9/vldDrD9judTrlcrm6PmT9/vh5++GF99NFHCgQC2rx5s1588UUdO3asx/fx+Xzyer1hGwAASE5Rv5vmF7/4hS677DJNnz5dVqtVd9xxh8rKynp9lHBlZaUcDkdo44m9AAAkr4jCSE5OjiwWi9xud9h+t9ut3Nzcbo8ZNWqUXn75ZbW0tOjQoUPau3evMjIyNHny5B7fp6KiQh6PJ7TV19dHMs0+8QcM/c+fj6rsV9U65Ts34K8PAAD6JqIwYrVaVVRUpKqqqtC+QCCgqqoqzZs3r9dj7Xa7xo0bp3PnzumFF17QTTfd1ONYm80WekJvtJ7UazZJP9/8of6wr1G/3Xl0wF8fAAD0TcSXacrLy7Vu3To99dRT2rNnj5YtW6aWlhaVlZVJkpYuXaqKiorQ+G3btunFF1/U/v379X//93/64he/qEAgoO9///sDdxb9YDKZtLh4giTp2eq6mM4FAIChLCXSAxYtWqTGxkatXLlSLpdLhYWF2rRpU6iota6uLqwepLW1Vffee6/279+vjIwM3XDDDXr66ac1fPjwATuJ/vpa0Xj99LV92nXEo12HPZo53hHrKQEAMORE3GckFqLZZ+T/e3aHfvvno1pcPEGVX505oK8NAMBQFpU+I8loSUnwUs1vdx6hkBUAgBgY8mGkZFK2Jo8appY2P4WsAADEwJAPIyaTSUvaC1nXVx+K8WwAABh6hnwYkaSvzhkvq8Ws9494teuwJ9bTAQBgSCGMSMoeZtXfzAw2bWN1BACAwUUYadfRc+Q3O49SyAoAwCAijLTrKGQ93ebXb3YeifV0AAAYMggj7ToXstKRFQCAwUMY6eRrc8bLmhIsZH3v8Kexng4AAEMCYaSTEcOsuuHKYCErqyMAAAwOwsgFOheyNreejfFsAABIfoSRCxRPytaU9kLW3/6ZjqwAAEQbYeQCJpMptDqyfludEuA5ggAAJDTCSDc6Cll3H/Vq1xE6sgIAEE2EkW50LmRdv41CVgAAookw0oMlJRMlSb/9M4WsAABEE2GkB3PzR2jq6Iz2jqwUsgIAEC2EkR5QyAoAwOAgjPTia3PGyZpi1gfHvHrvMIWsAABEA2GkF8PTrVowc4wkOrICABAthJGL6LhUQyErAADRQRi5CApZAQCILsLIRVDICgBAdBFG+oBCVgAAoocw0gedC1npyAoAwMAijPTRkhIKWQEAiAbCSB9dNTFYyHrmrF8vU8gKAMCAIYz0kclk0hIKWQEAGHCEkQh8tb2Qdc8xr/5MISsAAAOCMBKB4elWfamjIyuFrAAADAjCSIQWdypk9VLICgDAJSOMROiqiSN0WXshKx1ZAQC4dP0KI2vWrFF+fr7sdrtKSkpUXV3d6/hHHnlE06ZNU1pamvLy8vS9731Pra2t/ZpwrNGRFQCAgRVxGNm4caPKy8u1atUq1dbWqqCgQPPnz1dDQ0O349evX68VK1Zo1apV2rNnjx5//HFt3LhR//zP/3zJk4+Vr84ZJxuFrAAADIiIw8jDDz+s22+/XWVlZZoxY4bWrl2r9PR0PfHEE92O/9Of/qRrrrlGS5YsUX5+vr7whS9o8eLFF11NiWfhHVkPxXg2AAAktojCSFtbm2pqalRaWnr+BcxmlZaWauvWrd0ec/XVV6umpiYUPvbv369XX31VN9xwQ4/v4/P55PV6w7Z409GR9X/+fIxCVgAALkFEYaSpqUl+v19OpzNsv9PplMvl6vaYJUuW6P7779df/MVfKDU1VVOmTNF1113X62WayspKORyO0JaXlxfJNAdFUedC1h1HYj0dAAASVtTvptmyZYseeOAB/fu//7tqa2v14osv6pVXXtGPfvSjHo+pqKiQx+MJbfX19dGeZsRMJlNodeS/KWQFAKDfUiIZnJOTI4vFIrfbHbbf7XYrNze322N++MMf6pZbbtFtt90mSZo5c6ZaWlr0ne98R/fcc4/M5q55yGazyWazRTK1mPjq7PF68Hd7tdfVrJ31n2r2hBGxnhIAAAknopURq9WqoqIiVVVVhfYFAgFVVVVp3rx53R5z+vTpLoHDYrFIUsKvJjjSU7VgVntH1mo6sgIA0B8RX6YpLy/XunXr9NRTT2nPnj1atmyZWlpaVFZWJklaunSpKioqQuNvvPFG/fKXv9SGDRt04MABbd68WT/84Q914403hkJJIut4eB6FrAAA9E9El2kkadGiRWpsbNTKlSvlcrlUWFioTZs2hYpa6+rqwlZC7r33XplMJt177706cuSIRo0apRtvvFE//vGPB+4sYqho4ghd7szQh+5T+s2OI7plXn6spwQAQEIxGQlwrcTr9crhcMjj8SgrKyvW0+niV388oPv+5wNNz83U7+76S5lMplhPCQCAmOvr32+eTTMAvjp7vGwp5lAhKwAA6DvCyADoXMi6fhuFrAAARIIwMkBu7ujI+t5Rec5QyAoAQF8RRgbInAnBQtbWswH9ZicdWQEA6CvCyAAxmUyh23zX05EVAIA+I4wMoK90KmTdQSErAAB9QhgZQI70VH1p1lhJFLICANBXhJEBtqQk+ITh/6WQFQCAPiGMDLA5E0ZomjNTrWcDenkHhawAAFwMYWSAmUwmLS4Oro48W00hKwAAF0MYiYKvzDlfyFpb92mspwMAQFwjjESBI+18Ieuz1RSyAgDQG8JIlCxp78hKISsAAL0jjETJnAnDKWQFAKAPCCNRYjKZQqsjdGQFAKBnhJEoWjh7nOypZu1zU8gKAEBPCCNR1LmQlY6sAAB0jzASZYuLOxWynqaQFQCACxFGomzOhOGanpsp37mAXtpxONbTAQAg7hBGoizYkTW4OvJsdT2FrAAAXIAwMgjCC1lPxno6AADEFcLIIAgvZK2P8WwAAIgvhJFBEtaRlUJWAABCCCODZHYehawAAHSHMDJIwjqyVtORFQCADoSRQXRTYbCQ9UP3KQpZAQBoRxgZRI60VN3YXsj633RkBQBAEmFk0C1uv1TzynvHKGQFAECEkUHXuZD1RQpZAQAgjAy2zoWsz1LICgAAYSQWOjqyfug+pZpDFLICAIa2foWRNWvWKD8/X3a7XSUlJaquru5x7HXXXSeTydRlW7BgQb8nneiy7OcLWddXU8gKABjaIg4jGzduVHl5uVatWqXa2loVFBRo/vz5amho6Hb8iy++qGPHjoW2999/XxaLRd/4xjcuefKJbAmFrAAASOpHGHn44Yd1++23q6ysTDNmzNDatWuVnp6uJ554otvx2dnZys3NDW2bN29Wenr6kA8jhRSyAgAgKcIw0tbWppqaGpWWlp5/AbNZpaWl2rp1a59e4/HHH9c3v/lNDRs2rMcxPp9PXq83bEs2JpNJN3d0ZN1GISsAYOiKKIw0NTXJ7/fL6XSG7Xc6nXK5XBc9vrq6Wu+//75uu+22XsdVVlbK4XCEtry8vEimmTBumj1OaakWfdRAISsAYOga1LtpHn/8cc2cOVPFxcW9jquoqJDH4wlt9fX1gzTDwZVlT9WNBWMkBVdHAAAYiiIKIzk5ObJYLHK73WH73W63cnNzez22paVFGzZs0Le//e2Lvo/NZlNWVlbYlqwWFwcv1fzvrmP69HRbjGcDAMDgiyiMWK1WFRUVqaqqKrQvEAioqqpK8+bN6/XY5557Tj6fT9/61rf6N9MkVZg3XJ8Zk6W2cwG9WHsk1tMBAGDQRXyZpry8XOvWrdNTTz2lPXv2aNmyZWppaVFZWZkkaenSpaqoqOhy3OOPP66FCxdq5MiRlz7rJEJHVgDAUJcS6QGLFi1SY2OjVq5cKZfLpcLCQm3atClU1FpXVyezOTzj7Nu3T2+//bZef/31gZl1krmpcKweeGWPPmo4pXcPndTc/OxYTwkAgEFjMhLg/4p7vV45HA55PJ6krR/5wfPvaeO79frq7HF6eFFhrKcDAMAl6+vfb55NEycWl1DICgAYmggjcaJgvEMzKGQFAAxBhJE4YTKZQqsj6ylkBQAMIYSROLKwcKzSUi36uL2QFQCAoYAwEkcy7an6csFYSXRkBQAMHYSRONPRc+QVClkBAEMEYSTOzOpUyPoChawAgCGAMBJn6MgKABhqCCNx6KbCsUq3BgtZtx+kkBUAkNwII3GocyHrs9UUsgIAkhthJE4tLj5fyHqyhUJWAEDyIozEqVnjHbpibHtH1h0UsgIAkhdhJE6ZTKbQ6sj6bYcoZAUAJC3CSBzrKGT9pLGFQlYAQNIijMSx8I6sh2I8GwAAooMwEuc6eo68+r6LQlYAQFIijMS5mePOF7K+UHs41tMBAGDAEUbiHB1ZAQDJjjCSAL5ccL6QtfrAiVhPBwCAAUUYSQCZ9lTdVEhHVgBAciKMJIiOniMUsgIAkg1hJEHMGj9cV46jkBUAkHwIIwkk1JGVQlYAQBIhjCSQmwrHKd1q0X4KWQEASYQwkkAybCmhQtb1FLICAJIEYSTBLCmeKEn63S4KWQEAyYEwkmBmjncEC1n9FLICAJIDYSQBdayOUMgKAEgGhJEE9OXCsRrWXsi6jUJWAECCI4wkoAxbir5cOE4SHVkBAImPMJKglrT3HPndLpdOUMgKAEhg/Qoja9asUX5+vux2u0pKSlRdXd3r+E8//VTLly/XmDFjZLPZdPnll+vVV1/t14QRNHO8QzPHOdTmD+hFClkBAAks4jCyceNGlZeXa9WqVaqtrVVBQYHmz5+vhoaGbse3tbXp+uuv18GDB/X8889r3759WrduncaNG3fJkx/q6MgKAEgGJiPCv2IlJSWaO3euVq9eLUkKBALKy8vTnXfeqRUrVnQZv3btWv30pz/V3r17lZqa2q9Jer1eORwOeTweZWVl9es1ktEp3zmV/PgNtbT5teE7n9VnJ4+M9ZQAAAjp69/viFZG2traVFNTo9LS0vMvYDartLRUW7du7faY3/72t5o3b56WL18up9OpK6+8Ug888ID8fn+P7+Pz+eT1esM2dNW5kHX9NgpZAQCJKaIw0tTUJL/fL6fTGbbf6XTK5XJ1e8z+/fv1/PPPy+/369VXX9UPf/hD/exnP9O//uu/9vg+lZWVcjgcoS0vLy+SaQ4pN5cEL9Vsep9CVgBAYor63TSBQECjR4/Wf/zHf6ioqEiLFi3SPffco7Vr1/Z4TEVFhTweT2irr6+P9jQT1pXjzheyvlBDISsAIPFEFEZycnJksVjkdrvD9rvdbuXm5nZ7zJgxY3T55ZfLYrGE9n3mM5+Ry+VSW1v3/0/eZrMpKysrbEPPlrSvjjxLISsAIAFFFEasVquKiopUVVUV2hcIBFRVVaV58+Z1e8w111yjjz/+WIFAILTvww8/1JgxY2S1Wvs5bXR2Y0F7R9amFr2zn46sAIDEEvFlmvLycq1bt05PPfWU9uzZo2XLlqmlpUVlZWWSpKVLl6qioiI0ftmyZTpx4oTuuusuffjhh3rllVf0wAMPaPny5QN3FkNchi1FN81uL2SlIysAIMGkRHrAokWL1NjYqJUrV8rlcqmwsFCbNm0KFbXW1dXJbD6fcfLy8vTaa6/pe9/7nmbNmqVx48bprrvu0g9+8IOBOwtoSfEErd9Wp9fed+n4KZ9GZthiPSUAAPok4j4jsUCfkb758uq39d5hj/75hun6zl9NifV0AABDXFT6jCC+dXRkfba6nkJWAEDCIIwkkS+3F7IeaGrR1v3HYz0dAAD6hDCSRIZ1KmR9tpreLACAxEAYSTJLijs6sh7T8VO+GM8GAICLI4wkmSvHOTRrvENn/YZeqKUjKwAg/hFGktASClkBAAmEMJKEbiwYqwxbCoWsAICEQBhJQsNsKbqpcKwkaf02OrICAOIbYSRJdfQceW23i0JWAEBcI4wkqSvHOVTQXsj6fA2FrACA+EUYSWLnO7LWUcgKAIhbhJEk1lHIevD4aW39hEJWAEB8IowksbBC1moKWQEA8YkwkuSWlJwvZG2ikBUAEIcII0nuirHnC1lfoJAVABCHCCNDQMfqyLPVdQoEKGQFAMQXwsgQ8KVZ5wtZ36EjKwAgzhBGhoBhthQtnB0sZP1vClkBAHGGMDJEdPQceZ1CVgBAnCGMDBFXjHWoIG84HVkBAHGHMDKELCnOkyRtoJAVABBHCCNDSFhHVgpZAQBxgjAyhKRbzxey0pEVABAvCCNDzJLiiZIoZAUAxA/CyBAzY2wWhawAgLhCGBmCbi6mIysAIH4QRoagLxWMUaYtRYcoZAUAxAHCyBAULGQdJ0lav41CVgBAbBFGhqiOjqyv7XapsZlCVgBA7BBGhqgZY7NUmDdc5wIUsgIAYoswMoQtKQmujmzYTiErACB2+hVG1qxZo/z8fNntdpWUlKi6urrHsU8++aRMJlPYZrfb+z1hDJwvzTpfyPqnTyhkBQDERsRhZOPGjSovL9eqVatUW1urgoICzZ8/Xw0NDT0ek5WVpWPHjoW2Q4cOXdKkMTDSrSn6ypxgIeuzdGQFAMRIxGHk4Ycf1u23366ysjLNmDFDa9euVXp6up544okejzGZTMrNzQ1tTqfzkiaNgUMhKwAg1iIKI21tbaqpqVFpaen5FzCbVVpaqq1bt/Z43KlTpzRx4kTl5eXppptu0u7du3t9H5/PJ6/XG7YhOj4zJkuzJ1DICgCInYjCSFNTk/x+f5eVDafTKZfL1e0x06ZN0xNPPKHf/OY3euaZZxQIBHT11Vfr8OGe//BVVlbK4XCEtry8vEimiQgtpiMrACCGon43zbx587R06VIVFhbq2muv1YsvvqhRo0bpscce6/GYiooKeTye0FZfXx/taQ5pN84aq0x7iupOUMgKABh8EYWRnJwcWSwWud3usP1ut1u5ubl9eo3U1FTNnj1bH3/8cY9jbDabsrKywjZET5rVoq90dGStprgYADC4IgojVqtVRUVFqqqqCu0LBAKqqqrSvHnz+vQafr9fu3bt0pgxYyKbKaKqo+fI67vdFLICAAZVxJdpysvLtW7dOj311FPas2ePli1bppaWFpWVlUmSli5dqoqKitD4+++/X6+//rr279+v2tpafetb39KhQ4d02223DdxZ4JJNzz1fyPpcDZfFAACDJyXSAxYtWqTGxkatXLlSLpdLhYWF2rRpU6iota6uTmbz+Yxz8uRJ3X777XK5XBoxYoSKior0pz/9STNmzBi4s8CAWFI8QTvqPtWG6np996+myGw2xXpKAIAhwGQYRtzfPuH1euVwOOTxeKgfiaIzbX4VP/CGmlvP6elvF+svLxsV6ykBABJYX/9+82wahKRZLfrqbDqyAgAGF2EEYRZ3KmRtaG6N8WwAAEMBYQRhpudmaQ4dWQEAg4gwgi46OrJuqK6nIysAIOoII+jiS506sv7xk6ZYTwcAkOQII+iicyHr+m0UsgIAooswgm51FLJu/oBCVgBAdBFG0K3OhazPvUshKwAgeggj6NGSkomSpA3b6yhkBQBEDWEEPVowc4wy7SmqP3FGb39MISsAIDoII+hRmtWir80ZL4mOrACA6CGMoFcdPUcoZAUARAthBL2alpupookjKGQFAEQNYQQXFerISiErACAKCCO4qC/NGqMsClkBAFFCGMFF2VMt+mp7ISsdWQEAA40wgj7puFTzxh63GrwUsgIABg5hBH0SVshaQyErAGDgEEbQZ0vaV0eeraaQFQAwcAgj6LMF7YWsh0+e0f9RyAoAGCCEEfRZ50LWZylkBQAMEMIIIrKkpL0jK4WsAIABQhhBRC53ZuqqiSPkp5AVADBACCOI2GIKWQEAA4gwgohRyAoAGEiEEUQsvCProRjPBgCQ6Agj6JeOQtY39jRQyAoAuCSEEfRL50LWX79bH+vpAAASGGEE/daxOvJsdT2FrACAfiOMoN9umDlGjrRUHfn0jN76qDHW0wEAJCjCCPotWMg6TlLwNl8AAPqjX2FkzZo1ys/Pl91uV0lJiaqrq/t03IYNG2QymbRw4cL+vC3iUMfD897Y0yA3hawAgH6IOIxs3LhR5eXlWrVqlWpra1VQUKD58+eroaGh1+MOHjyof/zHf9Rf/uVf9nuyiD+XOTM1N7+9kHU7hawAgMhFHEYefvhh3X777SorK9OMGTO0du1apaen64knnujxGL/fr5tvvln33XefJk+efEkTRvzp6Mi6YXu9/BSyAgAiFFEYaWtrU01NjUpLS8+/gNms0tJSbd26tcfj7r//fo0ePVrf/va3+/Q+Pp9PXq83bEP8opAVAHApIgojTU1N8vv9cjqdYfudTqdcLle3x7z99tt6/PHHtW7duj6/T2VlpRwOR2jLy8uLZJoYZGGFrNsoZAUARCaqd9M0Nzfrlltu0bp165STk9Pn4yoqKuTxeEJbfT21CPGuo5C1ai+FrACAyKREMjgnJ0cWi0Vutztsv9vtVm5ubpfxn3zyiQ4ePKgbb7wxtC8QCATfOCVF+/bt05QpU7ocZ7PZZLPZIpkaYqyjkHX7wZP69fZ63fn5y2I9JQBAgohoZcRqtaqoqEhVVVWhfYFAQFVVVZo3b16X8dOnT9euXbu0c+fO0PblL39Zn/vc57Rz504uvySZjo6sFLICACIR0cqIJJWXl+vWW2/VVVddpeLiYj3yyCNqaWlRWVmZJGnp0qUaN26cKisrZbfbdeWVV4YdP3z4cEnqsh+J72+uHKN/+e0HoULWz00bHespAQASQMRhZNGiRWpsbNTKlSvlcrlUWFioTZs2hYpa6+rqZDbT2HUosqda9LU54/XEHw9o/bY6wggAoE9MhmHE/Xq61+uVw+GQx+NRVlZWrKeDXnzc0KzSh9+SxWzSH3/w18p12GM9JQBAjPT17zdLGBhQU0dnqjg/O9iR9V3uggIAXBxhBANucUmwMHkjhawAgD4gjGDA/c2VnTqyfkhHVgBA7wgjGHAdhayStL6ajqwAgN4RRhAVS9ov1fx+b4NcHjqyAgB6RhhBVFDICgDoK8IIoibUkbW6jkJWAECPCCOImi9emavh6ak66mmlkBUA0CPCCKKmcyHrf2+jkBUA0D3CCKJqcXFHIaubQlYAQLcII4iqqaMzVTwpWwEj2AQNAIALEUYQdUuKg4WsG7dTyAoA6IowgqjrXMj65ocNsZ4OACDOEEYQdWEdWbdxqQYAEI4wgkGxuP1Sze/3unXMcybGswEAxBPCCAbF1NEZoULWX28/HOvpAADiCGEEg+bmEgpZAQBdEUYwaOZfkasRFLICAC5AGMGgCS9kpSMrACCIMIJBtbiko5C1gUJWAIAkwggG2ZRRGSqhIysAoBPCCAbdklAhaz2FrAAAwggG3xevDBayHvO0ass+ClkBYKgjjGDQ2VIs+npRsJD12WoKWQFgqCOMICa+WXy+kPXopxSyAsBQRhhBTEwZlaHPTm7vyPouhawAMJQRRhAzHc+r2bi9Xuf8gRjPBgAQK4QRxEznQtY3P2yM9XQAADFCGEHMdC5kpSMrAAxdhBHEVMelmj/so5AVAIYqwghianKnQlY6sgLA0NSvMLJmzRrl5+fLbrerpKRE1dXVPY598cUXddVVV2n48OEaNmyYCgsL9fTTT/d7wkg+S0omSgreVUMhKwAMPRGHkY0bN6q8vFyrVq1SbW2tCgoKNH/+fDU0dN9JMzs7W/fcc4+2bt2q9957T2VlZSorK9Nrr712yZNHcph/hVPZw6ztHVkpZAWAocZkGEZEDwcpKSnR3LlztXr1aklSIBBQXl6e7rzzTq1YsaJPrzFnzhwtWLBAP/rRj/o03uv1yuFwyOPxKCsrK5LpIkE88Ooe/cdb+/X56aP1+P+bG+vpAAAGQF//fke0MtLW1qaamhqVlpaefwGzWaWlpdq6detFjzcMQ1VVVdq3b5/+6q/+qsdxPp9PXq83bENy++bcPEkUsgLAUBRRGGlqapLf75fT6Qzb73Q65XK5ejzO4/EoIyNDVqtVCxYs0KOPPqrrr7++x/GVlZVyOByhLS8vL5JpIgFNHpWheZNHUsgKAEPQoNxNk5mZqZ07d2r79u368Y9/rPLycm3ZsqXH8RUVFfJ4PKGtvp4/TkPB4hI6sgLAUJQSyeCcnBxZLBa53e6w/W63W7m5uT0eZzabNXXqVElSYWGh9uzZo8rKSl133XXdjrfZbLLZbJFMDUmgo5DV5Q0WspbOcF78IABAwotoZcRqtaqoqEhVVVWhfYFAQFVVVZo3b16fXycQCMjn80Xy1hgCwjqyVtORFQCGiogv05SXl2vdunV66qmntGfPHi1btkwtLS0qKyuTJC1dulQVFRWh8ZWVldq8ebP279+vPXv26Gc/+5mefvppfetb3xq4s0DS6Chk3bKvQUcoZAWAISGiyzSStGjRIjU2NmrlypVyuVwqLCzUpk2bQkWtdXV1MpvPZ5yWlhb9wz/8gw4fPqy0tDRNnz5dzzzzjBYtWjRwZ4Gk0VHIunX/cW3cXq/y6y+P9ZQAAFEWcZ+RWKDPyNDyP38+qjuf3aHcLLve/sHnlGLhqQUAkIii0mcEGAxf6FTI+gc6sgJA0iOMIO7YUiz6Rnsh67MUsgJA0iOMIC59szjYc4RCVgBIfoQRxKVJOcN09RQ6sgLAUEAYQdxaXNzRkbWOjqwAkMQII4hb86/I1chhVrm9PgpZASCJEUYQt6wp5vMdWbcdivFsAADRQhhBXAsVsn7YSCErACQpwgjiWkchq2FIG7nNFwCSEmEEcW9JSXsh67v1FLICQBIijCDufWHG+ULW3+9tiPV0AAADjDCCuGdNMevrV9GRFQCSFWEECWHx3POFrIdPno7xbAAAA4kwgoSQnzNM10wNFrL+mo6sAJBUCCNIGKGOrBSyAkBSIYwgYVDICgDJiTCChNG5kHU9hawAkDQII0goHYWsb1LICgBJgzCChNK5kHUjhawAkBQII0g4S4onSgqGEQpZASDxEUaQcK6f4VROhlUNzT5VUcgKAAmPMIKEY00x6+tFeZLoyAoAyYAwgoT0zbnBMPLmh42qP0EhKwAkMsIIElJ+zjD9xdScYEfWdylkBYBElhLrCQD9tbh4gt7+uEn/8dZ+vX/Eo5LJI1UyKVtXjnMo1ULOBoBEQRhBwrp+hlPTnJna527WH/Y16g/7GiVJ6VaLiiaOUMmkbJVMHqlZ4x2ypVhiPFsAQE9MhmEYsZ7ExXi9XjkcDnk8HmVlZcV6Oogj/oChPce8emf/cW07cELbD57Qp6fPho2xpZg1e8JwlUwaqZLJ2ZozYYTsqYQTAIi2vv79JowgqQQChj5saNa2/Se07cBxVR84oaZTbWFjUi0mFYwfrpLJ2SqZNFJFE0domI1FQgAYaIQRQJJhGPqksUXbDhwPBRS31xc2xmI26cpxDn12UrZKJmeraGK2HGmpMZoxACQPwgjQDcMwVHfitLbtP6F32ldODp88EzbGZJJmjMkKXdYpzs/WiGHWGM0YABIXYQTooyOfntG2/cGVk+qDJ3SgqaXLmGnOzGAwmRS8tDMq0xaDmQJAYolqGFmzZo1++tOfyuVyqaCgQI8++qiKi4u7Hbtu3Tr913/9l95//31JUlFRkR544IEex3eHMILB5Pa2atuBE9q2P7hy8lHDqS5jJo8aFlw5ab+0M8aRFoOZAkB8i1oY2bhxo5YuXaq1a9eqpKREjzzyiJ577jnt27dPo0eP7jL+5ptv1jXXXKOrr75adrtdP/nJT/TSSy9p9+7dGjdu3ICeDBANx0/5VH3gRDCgHDihvS6vLvzWTMhOV8mk4MrJZyeP1PgRaTKZTLGZMADEiaiFkZKSEs2dO1erV6+WJAUCAeXl5enOO+/UihUrLnq83+/XiBEjtHr1ai1durRP70kYQTzxnD6r6oMnVH0geDvx+0c8ClzwLRrrsAcv6bQ3YpuUM4xwAmDI6evf74juZ2xra1NNTY0qKipC+8xms0pLS7V169Y+vcbp06d19uxZZWdn9zjG5/PJ5zt/x4PX641kmkBUOdJTdf0Mp66f4ZQkNbee1buHTgZXT/Yf13uHPTrqadXLO4/q5Z1HJUmjMm3BVZP2gDJ1VIbMZsIJAEgRhpGmpib5/X45nc6w/U6nU3v37u3Ta/zgBz/Q2LFjVVpa2uOYyspK3XfffZFMDYiZTHuqPjdttD43LXiZ8nTbOe2o+1Tb9h/XOwdOaGf9p2ps9umV947plfeOSZKyh1k1N39E6I6d6blZshBOAAxRg9rp6cEHH9SGDRu0ZcsW2e32HsdVVFSovLw89LPX61VeXt5gTBG4ZOnWFF0zNUfXTM2RJLWe9evP9Z+215wcV82hkzrR0qbXdrv12m63JCnLnqK5+dmhRmxXjM1SCs/XATBERBRGcnJyZLFY5Ha7w/a73W7l5ub2euxDDz2kBx98UG+88YZmzZrV61ibzSabjVsnkRzsqZZg7cjkkZIuU9u5gHYd8YQasb178IS8redUtbdBVXsbJEnDrBYV5WerZFK2Pjs5WzPHDZc1hXACIDlFFEasVquKiopUVVWlhQsXSgoWsFZVVemOO+7o8bh/+7d/049//GO99tpruuqqqy5pwkCis6aYVTRxhIomjtA/XCed8wf0wTFvWAt7b+s5vfVho976MPjwP3uqWXMmBC/rFE/K1uwJw3m+DoCk0a9be2+99VY99thjKi4u1iOPPKJf//rX2rt3r5xOp5YuXapx48apsrJSkvSTn/xEK1eu1Pr163XNNdeEXicjI0MZGRl9ek/upsFQ4g8Y2udqDq2cVB88oRMt4c/XsVrMKswbHmrEVjRxhNKtPF8HQHyJyt00krRo0SI1NjZq5cqVcrlcKiws1KZNm0JFrXV1dTKbzy8n//KXv1RbW5u+/vWvh73OqlWr9C//8i+Rvj2Q9Cxmk2aMzdKMsVkqu2aSDMPQxw2n9E773TrbDpxQY7MveHvxwROSpBSzSTPHO0KN2K7KH6FMO8/XAZAYaAcPJBjDMHTw+OlQMNm2/7iOelrDxphN0hVjHaFGbMWTsjU8nefrABhcPJsGGELqT5w+38L+4AkdOn467PcmU/vzddr7nBRPylZOBkXiAKKLMAIMYcc8Z863sN9/XJ80dn3439TRGe0P/gu2sHdm9Xy7PQD0B2EEQEhjc8fzdYJ36+x1NXcZM3Fk8Pk6HY3Yxo9Ij8FMASQTwgiAHp1saWt/vk4woHxw1Nvl+TrjhqeFnkpcMmmkJo5M5/k6ACJCGAHQZ97Ws3r3YMdlnRPadcQj/wXpxJllU3H73Tolk7I1dXQG4QRArwgjAPqtxXdONR0P/ztwXH+u96jNHwgbM3KYNVRzUjxppKbnZvLwPwBhCCMABkzrWb9q6zqeTHxCtXUn5TsXHk4caanB5+u0X9qZMYbn6wBDHWEEQNT4zvm167BH2w6c0Dv7gw//O93mDxuTYUvRVfkj2ldPRmrWeIdSCSfAkEIYATBozvoD2n3UG2rEtv3ACTX7zoWNSUu1qGjiiNClnYI8nq8DJDvCCICY8QcM7TnmDWvE9unps2FjrClmzc4brpJJ2Zo+JksTstM1YWS6smhjDyQNwgiAuBEIGPqo4VTo4X/bDhxX06m2bscOT08NBpMLt5HpGuNIk4UiWSBhEEYAxC3DMLS/qUXb9p/QuwdP6MDxFtWfON1jQOmQYjZp/Ig05bUHlIkjg//s+JmHAwLxhTACIOG0+M6p7sRp1Z04rfr2fx46Hvz3+pOnddbf+3+usodZQ8FkQnaaJmYPC/48Ml25WXZWVYBB1te/3ymDOCcA6NUwW4o+MyZLnxnT9T9a/oAht7c1FE7qLthOtLSFtj/Xf9rl+FSLSeNHhF/6yWtfXcnLTleGjf8cArHCtw9AQrCYTRo7PE1jh6dp3pSRXX7f3HpW9SfOtIeTlvZ/nlH9idM63L6qcqCpRQeauj40UAo2cet8+SevU2jJzbLT0A2IIsIIgKSQaU/VjLGpmjG2+1WVY54z3V7+qTtxWidPn9XxljYdb2nTzm5WVawWs8Znp3VbVJs3Il3DWFUBLgnfIABJz2IOXqIZPyJdmtL1997Ws6rr4fLPkZNn1OYPaH9ji/Y3dr+qkpNh7ebyzzBNyE7X6EwbqyrARVDACgC9OOcP6JinVfUnTutQp5DSEVwu7J9yIWuKWXkj0tov/wwLu/wzITtdaVYavyF5UcAKAAMgxWJWXvtqx9Xd/N5z5mzYikrnyz9HPj2jtnMBfdLYok8aWyQ1djl+VKYtfEWl/fLPhOx0jcpgVQVDAysjABAl5/wBHf20NeyyT3CFpUWHjp9Wc+u5Xo+3pZjDgsqFvVVop494x8oIAMRYisUcXOUYmd7t7z2nzwZXU9rv/qm/oFbFdy6gjxpO6aOGU90eP7pjVWVk18LaURk2mUysqiAxsDICAHHorD+go5+eOb+qcvx02L9f+CDCC9lTzT1e/hk/glUVDA5WRgAggaVazJo4cpgmjhzW5XeGYejT9lWVsMs/7YHlmOeMWs8G9KH7lD50d7+qkptlD7v8M2FkmiZkB+8AysmwsqqCQcXKCAAkmbZzwVWVQ53v/Om0snLqIqsqaamWHi//jB+RJlsKqyroG1ZGAGCIsqaYlZ8zTPk53a+qnOy8qnK8pdPqyhkd9ZzRmbN+7XM3a5+7ucvxJlNwVSVUUNu+upLrsCs3yy5nlp3blRExVkYAACG+c34dOXmmy6Wfju10m/+ir5FlT1GuIxhMnFntIaU9rAQDi00jM2w8uHAIYGUEABAxW4pFk0dlaPKojC6/MwxDJ1radOiCSz/1J0/L7fXJ5WnVmbN+eVvPydvac72KFOyKOzrT1h5YbGGBJRRiHHYeYDhE8CkDAPrEZDJpZEZwVWPOhBFdfm8Yhryt5+T2tsrtbZXL0/5Pb6vcXl9oX9MpX/vzglp1zNPa63tm2FI0uj2shAcWWyiwjMqwKcVijtZpYxAQRgAAA8JkMsmRlipHWqoud2b2OO6cP6CmU21ydQos50NLcF+D16dm3zmd8p3TqcZzPT4XKPi+Uk6GLbSqkuuwyZnZ6dKQwy5npl1ZaSncJRSnCCMAgEGVYjEHC14ddimv53EtvnPBgOJplbu5VS7P+dUVd3Nwf0OzT+cChhqbfWps9mnXEU+Pr2dPNXe5DNRR05LrsGl0ZvBnawqrLIONMAIAiEvDbCmaMipDU7qpX+kQCBhqavGpob1mpWN1JbjS4pO7fZ/nzFm1ng3o4PHTOnj8dK/vO3KY9XwtS6fA0jnEjEhPZZVlAPUrjKxZs0Y//elP5XK5VFBQoEcffVTFxcXdjt29e7dWrlypmpoaHTp0SD//+c919913X8qcAQCQJJnNJo3OtGt0pl1XjnP0OK71rD+0quLyBi8DuTouDYVWWnxq8wd0vKVNx1va9MGxnt/XajGHalm61LF0WnWh023fRBxGNm7cqPLycq1du1YlJSV65JFHNH/+fO3bt0+jR4/uMv706dOaPHmyvvGNb+h73/vegEwaAIBI2FMtPXa07dDRg6XzZSBXWDFu8DLR8ZY2tfkDOnzyjA6fPNPr+zrSUjsFFlvY7c65DrtGZ9mUM4ynM0fcZ6SkpERz587V6tWrJUmBQEB5eXm68847tWLFil6Pzc/P19133x3xygh9RgAA8cJ3zq8Gr08N7XUsYZeGOt1B1Ho20KfXS+m4zbm90DZ0aeiCQtxhCXibc1T6jLS1tammpkYVFRWhfWazWaWlpdq6dWv/ZwsAQIKwpViU1955tiedb3M+f2motf0OIl8ovDSeChbgHvW06uhFbnPO7LjNuYc6ltwsu3IyrAl5m3NEYaSpqUl+v19OpzNsv9Pp1N69ewdsUj6fTz6fL/Sz1+sdsNcGACDaIrnNufGUL+wyUPgdRMH9p3zn1Ow7p+bGc/qkl9uczR23OYcFFltYYBmdZVeWPb5uc47LNZ/Kykrdd999sZ4GAABRlWIxa4wjTWMcab2OO+VrbybXvsoSKsLtVNfS0BxsJtfQ7FNDs09Sz7c5p6Va2gPL+aLbxcUTun2e0WCIKIzk5OTIYrHI7XaH7Xe73crNzR2wSVVUVKi8vDz0s9frVV5eLzejAwCQxDJsKcq4yG3O/oCh4y0+ubupY+l8B5HnzFmdOevXgaYWHWg6v8ryhStyEyOMWK1WFRUVqaqqSgsXLpQULGCtqqrSHXfcMWCTstlsstlsA/Z6AAAkO0un25xnqufbnM+0+bvpeuvTxJE918BEW8SXacrLy3XrrbfqqquuUnFxsR555BG1tLSorKxMkrR06VKNGzdOlZWVkoJFrx988EHo348cOaKdO3cqIyNDU6dOHcBTAQAAF5NmtSg/Z1jMVkG6E3EYWbRokRobG7Vy5Uq5XC4VFhZq06ZNoaLWuro6mc3nK3mPHj2q2bNnh35+6KGH9NBDD+naa6/Vli1bLv0MAABAQou4z0gs0GcEAIDE09e/34l3MzIAAEgqhBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBThBEAABBTET8oLxY6Hp/j9XpjPBMAANBXHX+3L/YYvIQII83NzZKkvLy8GM8EAABEqrm5WQ6Ho8ffJ8RTewOBgI4eParMzEyZTKYBe12v16u8vDzV19cn7dOAk/0cOb/El+znyPklvmQ/x2ien2EYam5u1tixY2U291wZkhArI2azWePHj4/a62dlZSXl/8A6S/Zz5PwSX7KfI+eX+JL9HKN1fr2tiHSggBUAAMQUYQQAAMTUkA4jNptNq1atks1mi/VUoibZz5HzS3zJfo6cX+JL9nOMh/NLiAJWAACQvIb0yggAAIg9wggAAIgpwggAAIgpwggAAIippA8ja9asUX5+vux2u0pKSlRdXd3r+Oeee07Tp0+X3W7XzJkz9eqrrw7STPsvknN88sknZTKZwja73T6Is43MW2+9pRtvvFFjx46VyWTSyy+/fNFjtmzZojlz5shms2nq1Kl68sknoz7P/or0/LZs2dLl8zOZTHK5XIMz4QhVVlZq7ty5yszM1OjRo7Vw4ULt27fvosclyvewP+eXaN/BX/7yl5o1a1aoIda8efP0u9/9rtdjEuXzkyI/v0T7/C704IMPymQy6e677+513GB/hkkdRjZu3Kjy8nKtWrVKtbW1Kigo0Pz589XQ0NDt+D/96U9avHixvv3tb2vHjh1auHChFi5cqPfff3+QZ953kZ6jFOyyd+zYsdB26NChQZxxZFpaWlRQUKA1a9b0afyBAwe0YMECfe5zn9POnTt1991367bbbtNrr70W5Zn2T6Tn12Hfvn1hn+Ho0aOjNMNL8+abb2r58uV65513tHnzZp09e1Zf+MIX1NLS0uMxifQ97M/5SYn1HRw/frwefPBB1dTU6N1339Vf//Vf66abbtLu3bu7HZ9In58U+flJifX5dbZ9+3Y99thjmjVrVq/jYvIZGkmsuLjYWL58eehnv99vjB071qisrOx2/N/+7d8aCxYsCNtXUlJi/P3f/31U53kpIj3HX/3qV4bD4Rik2Q0sScZLL73U65jvf//7xhVXXBG2b9GiRcb8+fOjOLOB0Zfz+8Mf/mBIMk6ePDkocxpoDQ0NhiTjzTff7HFMIn4PO/Tl/BL5O9hhxIgRxn/+5392+7tE/vw69HZ+ifr5NTc3G5dddpmxefNm49prrzXuuuuuHsfG4jNM2pWRtrY21dTUqLS0NLTPbDartLRUW7du7faYrVu3ho2XpPnz5/c4Ptb6c46SdOrUKU2cOFF5eXkX/X8AiSbRPsP+Kiws1JgxY3T99dfrj3/8Y6yn02cej0eSlJ2d3eOYRP4M+3J+UuJ+B/1+vzZs2KCWlhbNmzev2zGJ/Pn15fykxPz8li9frgULFnT5bLoTi88wacNIU1OT/H6/nE5n2H6n09nj9XWXyxXR+FjrzzlOmzZNTzzxhH7zm9/omWeeUSAQ0NVXX63Dhw8PxpSjrqfP0Ov16syZMzGa1cAZM2aM1q5dqxdeeEEvvPCC8vLydN1116m2tjbWU7uoQCCgu+++W9dcc42uvPLKHscl2vewQ1/PLxG/g7t27VJGRoZsNpu++93v6qWXXtKMGTO6HZuIn18k55eIn9+GDRtUW1urysrKPo2PxWeYEE/txcCZN29eWOK/+uqr9ZnPfEaPPfaYfvSjH8VwZuiLadOmadq0aaGfr776an3yySf6+c9/rqeffjqGM7u45cuX6/3339fbb78d66lERV/PLxG/g9OmTdPOnTvl8Xj0/PPP69Zbb9Wbb77Z4x/sRBPJ+SXa51dfX6+77rpLmzdvjutC26QNIzk5ObJYLHK73WH73W63cnNzuz0mNzc3ovGx1p9zvFBqaqpmz56tjz/+OBpTHHQ9fYZZWVlKS0uL0ayiq7i4OO7/wN9xxx363//9X7311lsaP358r2MT7XsoRXZ+F0qE76DVatXUqVMlSUVFRdq+fbt+8Ytf6LHHHusyNhE/v0jO70Lx/vnV1NSooaFBc+bMCe3z+/166623tHr1avl8PlkslrBjYvEZJu1lGqvVqqKiIlVVVYX2BQIBVVVV9XgtcN68eWHjJWnz5s29XjuMpf6c44X8fr927dqlMWPGRGuagyrRPsOBsHPnzrj9/AzD0B133KGXXnpJv//97zVp0qSLHpNIn2F/zu9CifgdDAQC8vl83f4ukT6/nvR2fheK98/v85//vHbt2qWdO3eGtquuuko333yzdu7c2SWISDH6DKNWGhsHNmzYYNhsNuPJJ580PvjgA+M73/mOMXz4cMPlchmGYRi33HKLsWLFitD4P/7xj0ZKSorx0EMPGXv27DFWrVplpKamGrt27YrVKVxUpOd43333Ga+99prxySefGDU1NcY3v/lNw263G7t3747VKfSqubnZ2LFjh7Fjxw5DkvHwww8bO3bsMA4dOmQYhmGsWLHCuOWWW0Lj9+/fb6Snpxv/9E//ZOzZs8dYs2aNYbFYjE2bNsXqFHoV6fn9/Oc/N15++WXjo48+Mnbt2mXcddddhtlsNt54441YnUKvli1bZjgcDmPLli3GsWPHQtvp06dDYxL5e9if80u07+CKFSuMN9980zhw4IDx3nvvGStWrDBMJpPx+uuvG4aR2J+fYUR+fon2+XXnwrtp4uEzTOowYhiG8eijjxoTJkwwrFarUVxcbLzzzjuh31177bXGrbfeGjb+17/+tXH55ZcbVqvVuOKKK4xXXnllkGccuUjO8e677w6NdTqdxg033GDU1tbGYNZ903Er64VbxzndeuutxrXXXtvlmMLCQsNqtRqTJ082fvWrXw36vPsq0vP7yU9+YkyZMsWw2+1Gdna2cd111xm///3vYzP5Puju3CSFfSaJ/D3sz/kl2nfw7/7u74yJEycaVqvVGDVqlPH5z38+9IfaMBL78zOMyM8v0T6/7lwYRuLhMzQZhmFEb90FAACgd0lbMwIAABIDYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMQUYQQAAMTU/w+ZuvZWqANXDwAAAABJRU5ErkJggg==\n" + ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" } + ], + "source": [ + "class CustomCallback(keras.callbacks.Callback):\n", + " def on_epoch_end(self, epoch, logs=None):\n", + " model_name = f\"/content/drive/MyDrive/{lora_name}_{lora_rank}_epoch{epoch+1}.lora.h5\"\n", + " gemma_lm.backbone.save_lora_weights(model_name)\n", + "\n", + " # Evaluate\n", + " text_gen(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\")\n", + " text_gen(\"ๆ—ฅๆœฌใฎๅนณๅ‡กใชๅๅญ—\")\n", + "\n", + "history = gemma_lm.fit(train, epochs=train_epoch, batch_size=2, callbacks=[CustomCallback()])\n", + "\n", + "import matplotlib.pyplot as plt\n", + "plt.plot(history.history['loss'])\n", + "plt.show()" ] }, { "cell_type": "markdown", + "metadata": { + "id": "4QkVxDR1IHmx" + }, "source": [ "## Try a different sampler\n", "\n", "The top-K algorithm randomly picks the next token from the tokens of top K probability.\n", "\n", "**NOTE: Due to randomness of the sampler, you may encounter \\ tokens that surpass the number you trained with.**" - ], - "metadata": { - "id": "4QkVxDR1IHmx" - } + ] }, { "cell_type": "code", - "source": [ - "gemma_lm.compile(sampler=\"top_k\")\n", - "\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n" - ], + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "1zwJmf9KzwW0", - "outputId": "dc5c39b2-0bc2-4888-c786-475724887bba" + "id": "1zwJmf9KzwW0" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "--------------------------------------------------------------------------------\n", "[ 106 1645 108 47172 142742 39742 107 108 106 2516\n", @@ -2228,37 +2167,36 @@ "TOTAL TIME ELAPSED: 0.51s\n" ] } + ], + "source": [ + "gemma_lm.compile(sampler=\"top_k\")\n", + "\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ็ใ—ใ„ๅๅญ—\", my_dictionary)\n" ] }, { "cell_type": "markdown", - "source": [ - "Try a slight different prompts" - ], "metadata": { "id": "mqIx09b2INH0" - } + }, + "source": [ + "Try a slight different prompts" + ] }, { "cell_type": "code", - "source": [ - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ้ข็™ฝใ„ๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ่ฆ‹ใŸใ“ใจใชใ„ๅๅญ—\", my_dictionary)\n", - "text_gen_with_dict(\"ๆ—ฅๆœฌใฎใ™ใ”ใ„ๅๅญ—\", my_dictionary)" - ], + "execution_count": null, "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "oWSODjdj8H4V", - "outputId": "2c1d3ff6-d873-4f4c-df70-17f91421f294" + "id": "oWSODjdj8H4V" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "--------------------------------------------------------------------------------\n", "[ 106 1645 108 47172 39742 107 108 106 2516 108 98647 4758\n", @@ -2375,7 +2313,26 @@ "TOTAL TIME ELAPSED: 0.49s\n" ] } + ], + "source": [ + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ้ข็™ฝใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎ่ฆ‹ใŸใ“ใจใชใ„ๅๅญ—\", my_dictionary)\n", + "text_gen_with_dict(\"ๆ—ฅๆœฌใฎใ™ใ”ใ„ๅๅญ—\", my_dictionary)" ] } - ] -} \ No newline at end of file + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Custom_Vocabulary.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/README.md b/README.md index d8cfe18..5793255 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Finetune_with_Axolotl.ipynb](Gemma/Finetune_with_Axolotl.ipynb) | Finetune Gemma using [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl). | | [Finetune_with_XTuner.ipynb](Gemma/Finetune_with_XTuner.ipynb) | Finetune Gemma using [XTuner](https://github.com/InternLM/xtuner). | | [Finetune_with_LLaMA_Factory.ipynb](Gemma/Finetune_with_LLaMA_Factory.ipynb) | Finetune Gemma using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). | +| [Custom_Vocabulary.ipynb](Gemma/Custom_Vocabulary.ipynb) | Demonstrate how to use a custom vocabulary "<unused[0-98]>" tokens in Gemma. | | **Alignment** | | | [Aligning_DPO_Gemma_2b_it.ipynb](Gemma/Aligning_DPO_Gemma_2b_it.ipynb) | Demonstrate how to align a Gemma model using DPO (Direct Preference Optimization) with [Hugging Face TRL](https://huggingface.co/docs/trl/en/index). | | **Evaluation** | | From 882821ca6f0e8d11f491df4b45622ad156393fdb Mon Sep 17 00:00:00 2001 From: kkech Date: Mon, 21 Oct 2024 22:46:25 +0300 Subject: [PATCH 13/43] Create the transformers edition of the workshop "How to Fine tuning Gemma" --- ...ne_tuning_Gemma_Transformers_Edition.ipynb | 6102 +++++++++++++++++ 1 file changed, 6102 insertions(+) create mode 100644 Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb diff --git a/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb b/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb new file mode 100644 index 0000000..71fb523 --- /dev/null +++ b/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb @@ -0,0 +1,6102 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "i1PHqD-ZY4-c" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "YNDq8NbCY7oh" + }, + "source": [ + "# Workshop: How to Fine-tuning Gemma - Transformers Edition\n", + "\n", + "To illustrate fine-tuning the model for a specific task, You'll learn how to condition a Gemma model to answer in a specific language. Let's consider the example of generating a random Portuguese title based on a user's instruction such as \"Write a title\". To make this possible, you will curate a manageable dataset that can be manually processed. This approach is feasible because Gemma 2 has prior knowledge of general Portuguese language patterns, enabling it to adapt to this specific task effectively." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "u4EM3g9u2_KA" + }, + "source": [ + "## What is Fine-tuning\n", + "\n", + "In the first place, you have to understand what is fine-tuning. It's a specialized form of [transfer learning](https://en.wikipedia.org/wiki/Transfer_learning). It involves taking a pre-trained language model - one that has already been exposed to a vast corpus of text data and learned the general patterns and structures of language - and further training it on a smaller, more specific dataset. This additional training allows the model to adapt and refine its knowledge, making it better suited for a particular task or domain.\n", + "\n", + "Imagine you are a skilled gamer who excels at various genres, from action-adventures to strategy games. Fine-tuning is akin to taking you and having you focus intensely on mastering a specific game, like a complex real-time strategy (RTS) title. You already possess a strong foundation of gaming skills and knowledge, but the dedicated practice and study within the RTS genre sharpens your tactics, understanding of game mechanics, and overall proficiency within that particular realm.\n", + "\n", + "Similarly, pre-trained language models have a broad understanding of language, but fine-tuning helps them specialize. By exposing them to a curated dataset relevant to your desired application, you guide the model to learn the nuances and intricacies specific to that domain. It's like giving the model a crash course in the language of your chosen field, enabling it to perform tasks with greater accuracy and fluency.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3rzH5Ugf5RlJ" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model:\n", + "\n", + "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **T4 GPU**.\n", + "\n", + "\n", + "### Gemma setup on Kaggle\n", + "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n", + "\n", + "* Get access to Gemma on kaggle.com.\n", + "* Select a Colab runtime with sufficient resources to run the Gemma 2B model.\n", + "* Generate and configure a Kaggle username and API key.\n", + "\n", + "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "URMuBzkMVxpU" + }, + "source": [ + "### Set environemnt variables\n", + "\n", + "Set environment variables for ```HUGGING_FACE```." + ] + }, + { + "cell_type": "code", + "source": [ + "import os\n", + "from google.colab import userdata, drive\n", + "from huggingface_hub import login\n", + "\n", + "login(userdata.get(\"HUGGING_FACE\"))\n", + "\n", + "access_token = userdata.get(\"HUGGING_FACE\")\n", + "my_hf_username = userdata.get(\"HUGGING_FACE_UN\")\n", + "os.environ[\"HF_USER\"] = my_hf_username\n", + "os.environ[\"HF_TOKEN\"] = userdata.get(\"HUGGING_FACE\")" + ], + "metadata": { + "id": "TT7GexJnZZCj", + "outputId": "5f5cc93f-d9c8-4bb4-d828-b670ed352b35", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.\n", + "Token is valid (permission: read).\n", + "Your token has been saved to /root/.cache/huggingface/token\n", + "Login successful\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LXfDwRTQVns2" + }, + "source": [ + "### Install dependencies\n", + "\n", + "Install Transformers and Torch" + ] + }, + { + "cell_type": "code", + "source": [ + "!pip install transformers torch\n", + "# Set the backbend before importing Keras\n", + "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n", + "# Avoid memory fragmentation on JAX backend.\n", + "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n", + "\n", + "# Training Configurations\n", + "token_limit = 128\n", + "num_data_limit = 100\n", + "lora_name = \"my_lora\"\n", + "lora_rank = 4\n", + "lr_value = 1e-3\n", + "train_epoch = 5\n", + "model_id = \"google/gemma-2-2b-it\"" + ], + "metadata": { + "id": "WNn86PiiXTNf", + "outputId": "5ac7a598-0773-4596-a393-ba53cbfd2470", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.44.2)\n", + "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.4.1+cu121)\n", + "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1)\n", + "Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.24.7)\n", + "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4)\n", + "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.1)\n", + "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2)\n", + "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3)\n", + "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5)\n", + "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)\n", + "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.5)\n", + "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)\n", + "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch) (1.13.3)\n", + "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.4.1)\n", + "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n", + "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2024.6.1)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (3.0.1)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.0)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.10)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.2.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.8.30)\n", + "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kUl0t469YfQY" + }, + "source": [ + "## Load Model\n", + "\n", + "**Why Fine-tuning?**\n", + "\n", + "Before embarking on fine-tuning, it's crucial to evaluate if its benefits align with the specific requirements of your application. Fine-tuning involves meticulous data preparation and extensive training, making it an arduous process. Therefore, it's essential to assess whether the potential gains justify the significant effort required.\n", + "\n", + "**Try \"Prompt Engineering\" first.** before fine-tuning\n", + "\n", + "Would you like to enable Gemma's multilingual capabilities?\n", + "Please note that Gemma 2 already has some multilingual capabilities. Here's the example output from Gemma 2 2B instruction-tuned model.\n", + "\n", + "Do you wish to adjust the tone or writing style?\n", + "Gemma 2 might be familiar with the writing style you have in mind. Here's another output from the same model." + ] + }, + { + "cell_type": "code", + "source": [ + "from transformers import AutoModelForCausalLM, AutoTokenizer\n", + "import time\n", + "\n", + "# Load a pretrained model and tokenizer from Hugging Face\n", + "gemma_lm = AutoModelForCausalLM.from_pretrained(model_id, token=access_token)\n", + "tokenizer = AutoTokenizer.from_pretrained(model_id, token=access_token)\n", + "\n", + "# Summarize the model\n", + "print(gemma_lm)\n", + "\n", + "tick_start = 0\n", + "\n", + "def tick():\n", + " global tick_start\n", + " tick_start = time.time()\n", + "\n", + "def tock():\n", + " print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n", + "\n", + "def text_gen(prompt, token_limit=100): # You can set your token limit\n", + " tick()\n", + "\n", + " # Format input, same as your original code\n", + " input_text = f\"user\\n{prompt}\\nmodel\\n\"\n", + "\n", + " # Tokenize input\n", + " inputs = tokenizer(input_text, return_tensors=\"pt\")\n", + "\n", + " # Generate text using the model\n", + " output_tokens = gemma_lm.generate(\n", + " inputs[\"input_ids\"],\n", + " max_length=token_limit,\n", + " pad_token_id=tokenizer.eos_token_id # Prevent errors if the input length exceeds the model's limit\n", + " )\n", + "\n", + " # Decode the generated tokens back to text\n", + " output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)\n", + "\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + "\n", + " tock()" + ], + "metadata": { + "id": "ywcDWVhAXb_9", + "outputId": "d9af537b-6c7a-4018-ada6-7b5dc58328d8", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 880, + "referenced_widgets": [ + "5dcc5fb3ad9e41eeb877fdd04f09ba7d", + "eeda0cc4233f41c7a1bb077b335922af", + "7786f72663544d3b964f8706ee4c87db", + "16489924a7194b6780f51cb3fa808c5e", + "8a87e9b8ba9d41ce8b44f2fef3244a81", + "e84b2d9e1e9e46c28e788385d64bc7c6", + "60c77f65806f4ff688453439fe7c9d83", + "de9354062f994aaabb104890709fc34a", + "9ae9adefe60b4511a90cdb2b3ef6e03d", + "3a29897fd3b3454ca752f63f45007d7f", + "1ef2aa04eaba4620a49c748ca4380d99", + "f4188acdf47f4cce86c02cc0d2bf46c7", + "54c456ec68684e57aa57aa7715921bfb", + "4ad2c31035574f26bf6118ef0d1413a9", + "a12cf321b8ae4075a4c3b0677d7bdf45", + "f8234ec2555b4e5684816aed66b63651", + "d816729082e94c41988d467654ef7841", + "b2d33d479a634d5c807e2a1860f4bc8c", + "b542562fcd6c466c9aa7f12b1b881207", + "2e8af71a4d85433a8e68f5c3737d7fb9", + "cdf16eb2e8ca487b9e9c7de16381c5c8", + "15332ef69c9244ccaaf3030dc9cb113a", + "eee1dd11b43e435cb226ee6df594291d", + "cc3c5ae5ebb74b2dbc81333533cc78cf", + "2f79a5c2a23b4808a181904593072dc5", + "a1c592e1355844adaa9ea80718499e4c", + "9805d9bec3f14c41a7a87fced47bb6ca", + "aed3fca1af5648e3b681d0c44f9c4f86", + "5960f282615e4932af3d9b7f69de2efa", + "151d7e95158d4d8aa2faf2886c8e0604", + "b5e8f1d688ea4694ac83a1bf89e27ae0", + "02ddb4d4c134415fa89a98984ae45c5f", + "7ba48bd1bbb148ff9dc7a2c33b835a35", + "9dc263e2dbfa4d9d9a56d62a69a57a8f", + "6a3171fe96744527805d8dc9eb159006", + "0bd0296c84a248259b5c48ac4e6bb760", + "20bc2639f53d4701aeafdb30e60f47da", + "372294f99f90486391b77296724916cc", + "2d9816c05a3544c0afc5a41c74ad6179", + "52132d841f1f48dd8c1d87ee0d308764", + "2fd5a5bfe878469c8f6b710c672fda9c", + "3a0c665acdbb4c83918c0ac469114235", + "eebc1808719c45b3bc7ad525798ce120", + "5d5a094d9cb545788f75992cbd859257", + "ecb1c6f7effa46b296e3574cddcb88de", + "cf5ec90af2944a63a0f83d5520307b82", + "28c7d5e1e84848c29b3d533930ccc198", + "b2831f93a9d148f4871c435382c6302e", + "9a827d1a0a424d6381b3e1eaac796b1e", + "afda84f0f4a445ee9dee2ccf047ce5ce", + "fad3963f838c44d98af0fdebb787741c", + "2dbc25a62598422b9c0768e109f8fd92", + "f97ae90efc3f44498db9203ef90d232e", + "707955f6553648fbb75ea65f2df2eaf6", + "c2fd92147e044b96ace6530ae67ca181", + "64324fda5c814874aaa2e97fa78ab3f0", + "abc4f6598a1f4c57a74a55f854ddc66d", + "6bfe81a71ff54a1390ff1d90d7d78864", + "25c94e5770e84ecea0e0f68f5bd64905", + "d15f3939e8b143a5b2deeb82435f9d26", + "a006f1ee609b47dda37389b5478a803d", + "0254a7b1482545cba26b5708a8920b34", + "977e2dc71b224a05823aa8120fa472bd", + "f6bdb5a59a044f4bb7332068179adce1", + "ecb124db56ea4925b83e91206c53f89b", + "e29c06ee25f84198bbea49aa0b19bb65", + "43bccde2f2804b089e4e40081058855f", + "f96a8568f4dc409090a123ff63766e58", + "163aee402b454733b1e74d43834849fb", + "8ebf2a32eac047cf9367f615ec01f0a8", + "760de7e3cef94be8b82cf8fc99385844", + "22d61c177e4e4d09a62ac08785682151", + "7ad0cea5c87e40da9b1dd1966b222f5a", + "a1e5eef9348f4bfeb456e5f096301778", + "4d94e337748343bb8e26d7ad7d361906", + "bc1dcd6dcd16462999359b579146c856", + "7c79134c76334c8680e81b58b9f75819", + "488f3d23e6ee4370b0e3750c4610acc1", + "5b1b61f335594a4487667b9d60395218", + "350ac704ca744a859527627fa92f9aa7", + "5a25dc05639c4cb282b1a2c4bbfdb572", + "4aaae0e141b14af78d9207b5f99a4516", + "182f4c00d1e94a4081057f34462704b7", + "62bb5bafdf2e47628a3ff1b114bac45a", + "79fac9dcae554d6f885608bd29fb904e", + "3b43ab0605644de9a852218edf3516fb", + "67261f6d01b14c50a4d7413ae8a4d113", + "110e4a8116e847dc9e4cea9a9df9e2b5", + "9ac1990111d64f919644a292b4ea2d36", + "494574a90f844172b5dce66aba248564", + "6b30ac7eab3346e8978274b8bcb03371", + "bdee91edaed5469a98ed808735bb30b0", + "eb7cfbc69c4848b3b4274203a70b6f1e", + "adb76c55b90d4f5aadd92d51b526fc27", + "5dd424f1051645abb00a3a3aa7d93990", + "018cb6becc764afb8bc6d9e9f2196fb9", + "bdd09cce6f4d4cf5b555678537c029c9", + "41ffd80053b6402f8aba20e1413e30cd", + "015fabd8f4934029900eec53ef02bdb8", + "b91c257be15341179ce8a262b3962ef0", + "e207f4774be44704a804dfacd628ad5a", + "559c60d219f24907b2b3b8e7c78deed8", + "88189b7d8769412685e197c8c8b7ec5c", + "2c8f45c2c1624865a698eb8c5527b0e4", + "d55c5f99d58942c2ba0506090f9755c2", + "9b5c6869f25945fa9424593255af07fc", + "435dfab7d6c94fb7b8b1a9dbb81bdbe7", + "9c0fa484394142c3beee7e272f626b08", + "3ad6acbc511b4417b5ca4e56a8688828", + "2d693baa6ea04b58965141813bd0ceca", + "379696127cc841a08cb6ca5000180032", + "9d03f9f959254badae3c94b71e84f2cc", + "65d1d113a0a8446c83563d9aefd8b112", + "9f61fd282f104094bc821e8a4b9f27ce", + "fd0fc02ef4b545c0a459527aca8bc81e", + "01cf0405f63a4d38ad8d1fe31721806f", + "ae20add968524517a261a81be45cde48", + "dc76053f6ae847469aab55949ad182be", + "4d98195ba52f48a29a237777eeef35cf", + "ffeafe7d69484d64af7484311e32703d", + "b66e7b11b2404e3daeaed1272994adb5" + ] + } + }, + "execution_count": null, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "5dcc5fb3ad9e41eeb877fdd04f09ba7d", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "config.json: 0%| | 0.00/838 [00:00 \n", + "177383 -> olรก\n", + "235265 -> .\n", + " 6235 -> Pra\n", + " 3004 -> zer\n", + " 2190 -> em\n", + " 26809 -> conhe\n", + "235260 -> c\n", + "235442 -> รช\n", + "235290 -> -\n", + " 545 -> lo\n", + "235265 -> .\n", + " 687 -> O\n", + " 11030 -> tempo\n", + " 5365 -> estรก\n", + " 14693 -> muito\n", + " 12318 -> bom\n", + " 43897 -> hoje\n", + "235265 -> .\n", + "\n", + "[2, 235530, 235579, 45884, 235483, 235940, 27074, 20579, 89299, 30848, 197350, 99877, 235940, 133533, 118300, 161437, 3640, 236062, 84372, 236062, 197350, 6032, 235265]\n", + " 2 -> \n", + "235530 -> เคจ\n", + "235579 -> เคฎ\n", + " 45884 -> เคธเฅเคค\n", + "235483 -> เฅ‡\n", + "235940 -> เฅค\n", + " 27074 -> เค†เคช\n", + " 20579 -> เคธเฅ‡\n", + " 89299 -> เคฎเคฟเคฒ\n", + " 30848 -> เค•เคฐ\n", + "197350 -> เค…เคšเฅเค›เคพ\n", + " 99877 -> เคฒเค—เคพ\n", + "235940 -> เฅค\n", + "133533 -> เค†เคœ\n", + "118300 -> เคฎเฅŒ\n", + "161437 -> เคธเคฎ\n", + " 3640 -> เคธ\n", + "236062 -> เคš\n", + " 84372 -> เคฎเฅ\n", + "236062 -> เคš\n", + "197350 -> เค…เคšเฅเค›เคพ\n", + " 6032 -> เคนเฅˆ\n", + "235265 -> .\n" + ] + } + ], + "source": [ + "import jax.numpy as jnp\n", + "\n", + "# Function to detokenize (convert tokens back into words)\n", + "def detoken(tokens):\n", + " print(tokens['input_ids']) # Print the token IDs for debugging\n", + " input_ids = tokens['input_ids'] # Get input IDs from the tokenizer output\n", + "\n", + " for x in input_ids: # Iterate over the token list\n", + " # Use tokenizer.decode() to convert tokens back to words\n", + " word = tokenizer.decode([x]) # No need to convert to JAX array for decoding\n", + " print(f\"{x:6} -> {word}\")\n", + "\n", + "# Example text 1: Portuguese\n", + "detoken(tokenizer(\"olรก. Prazer em conhecรช-lo. O tempo estรก muito bom hoje.\", return_tensors=None))\n", + "print()\n", + "\n", + "# Example text 2: Hindi\n", + "detoken(tokenizer(\"เคจเคฎเคธเฅเคคเฅ‡เฅค เค†เคชเคธเฅ‡ เคฎเคฟเคฒเค•เคฐ เค…เคšเฅเค›เคพ เคฒเค—เคพเฅค เค†เคœ เคฎเฅŒเคธเคฎ เคธเคšเคฎเฅเคš เค…เคšเฅเค›เคพ เคนเฅˆ.\", return_tensors=None))\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9T7xe_jzslv4" + }, + "source": [ + "## Load Dataset\n", + "\n", + "How many datasets do you need? You can start with a relatively small dataset, approximately 10 to 20, those can have a significant impact on a model's behavior.\n", + "\n", + "To improve the output quality, a target of around 200 total examples is recommended. Nevertheless, the amount of data required for tuning really depends on how much you want to influence the model's behavior. Our recommendation is to commence with a limited amount of data and gradually incorporate additional data into the training process until the desired behavior is achieved." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ZiS-KU9osh_N", + "outputId": "dedf9024-ff52-4a36-e586-6fea9139a53f", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "15\n", + "user\n", + "Write a title\n", + "model\n", + "O Alquimista\n", + "\n", + "user\n", + "Write a title\n", + "model\n", + "Dom Casmurro\n", + "\n", + "user\n", + "Write a title\n", + "model\n", + "Memorial do Convento\n" + ] + } + ], + "source": [ + "# example titles\n", + "data = [\n", + " \"O Alquimista\", # by Paulo Coelho\n", + " \"Dom Casmurro\", # by Machado de Assis\n", + " \"Memorial do Convento\", # by Josรฉ Saramago\n", + " \"A Hora da Estrela\", # by Clarice Lispector\n", + " \"Vidas Secas\", # by Graciliano Ramos\n", + " \"O Cortiรงo\", # by Aluรญsio Azevedo\n", + " \"Grande Sertรฃo: Veredas\", # by Guimarรฃes Rosa\n", + " \"Capitรฃes da Areia\", # by Jorge Amado\n", + " \"A Sibila\", # by Agustina Bessa-Luรญs\n", + " \"Os Maias\", # by Eรงa de Queirรณs\n", + " \"O Crime do Padre Amaro\", # by Eรงa de Queirรณs\n", + " \"A Relรญquia\", # by Eรงa de Queirรณs\n", + " \"O Primo Basรญlio\", # by Eรงa de Queirรณs\n", + " \"A Ilustre Casa de Ramires\", # by Eรงa de Queirรณs\n", + " \"A Cidade e as Serras\" # by Eรงa de Queirรณs\n", + "]\n", + "\n", + "train = []\n", + "\n", + "for x in data:\n", + " item = f\"user\\nWrite a title\\nmodel\\n{x}\"\n", + " length = len(tokenizer(item))\n", + " # skip data if the token length is longer than our limit\n", + " if length < token_limit:\n", + " train.append(item)\n", + " if(len(train)>=num_data_limit):\n", + " break\n", + "\n", + "print(len(train))\n", + "print(train[0])\n", + "print()\n", + "print(train[1])\n", + "print()\n", + "print(train[2])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9s1o96HRtwV_" + }, + "source": [ + "See below example code, using HF datasets, if your datasets are much bigger." + ] + }, + { + "cell_type": "code", + "source": [ + "!pip install datasets" + ], + "metadata": { + "id": "ZS9zT92tiKHu" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "code", + "source": [ + "# from datasets import load_dataset\n", + "\n", + "# # Load the dataset\n", + "# ds = load_dataset(\"bebechien/korean_cake_boss\", split=\"train\")\n", + "# print(ds)\n", + "\n", + "# # Prepare the dataset for tokenization\n", + "# train = []\n", + "\n", + "# # Iterate through the dataset and format the prompts\n", + "# for x in ds:\n", + "# # Create the formatted input-output text\n", + "# item = f\"user\\n๋‹ค์Œ์— ๋Œ€ํ•œ ์ด๋ฉ”์ผ ๋‹ต์žฅ์„ ์ž‘์„ฑํ•ด์ค˜.\\n\\\"{x['input']}\\\"\\nmodel\\n{x['output']}\"\n", + "\n", + "# # Tokenize the item and get its length\n", + "# length = len(tokenizer(item)[\"input_ids\"])\n", + "# print(length)\n", + "# # Skip if the tokenized item is longer than the token limit\n", + "# if length < token_limit:\n", + "# train.append(item)\n", + "\n", + "# # Stop if we have reached the desired data limit\n", + "# if len(train) >= num_data_limit:\n", + "# break\n", + "\n", + "# # Output the results\n", + "# print(f\"Number of training examples: {len(train)}\")\n", + "# print(f\"First example: {train[0]}\")\n", + "# print(f\"Second example: {train[1]}\")\n", + "# print(f\"Third example: {train[2]}\")" + ], + "metadata": { + "id": "Vh9s8m_PgoAH", + "outputId": "ccdf0c19-c3b5-40d5-d7ae-dc7a2c59fe0b", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Dataset({\n", + " features: ['input', 'output'],\n", + " num_rows: 20\n", + "})\n", + "234\n", + "307\n", + "335\n", + "348\n", + "366\n", + "158\n", + "169\n", + "157\n", + "198\n", + "167\n", + "163\n", + "150\n", + "165\n", + "145\n", + "157\n", + "308\n", + "407\n", + "298\n", + "419\n", + "318\n", + "Number of training examples: 10\n", + "First example: user\n", + "๋‹ค์Œ์— ๋Œ€ํ•œ ์ด๋ฉ”์ผ ๋‹ต์žฅ์„ ์ž‘์„ฑํ•ด์ค˜.\n", + "\"์•ˆ๋…•ํ•˜์„ธ์š”, 10์›” 5์ผ์— ์žˆ์„ ๋”ธ ์•„์ด์˜ 5๋ฒˆ์งธ ์ƒ์ผ์„ ์œ„ํ•ด ์ผ€์ดํฌ๋ฅผ ์ฃผ๋ฌธํ•˜๊ณ  ์‹ถ์Šต๋‹ˆ๋‹ค. ์•„์ด๊ฐ€ ์ข‹์•„ํ•˜๋Š” ํ•‘ํฌ์ƒ‰ ๊ณต์ฃผ๋‹˜ ์ผ€์ดํฌ๊ฐ€ ๊ฐ€๋Šฅํ• ๊นŒ์š”?\"\n", + "model\n", + "๊ณ ๊ฐ๋‹˜, ์•ˆ๋…•ํ•˜์„ธ์š”.\n", + "\n", + "๋”ฐ๋‹˜์˜ 5๋ฒˆ์งธ ์ƒ์ผ์„ ์ง„์‹ฌ์œผ๋กœ ์ถ•ํ•˜๋“œ๋ฆฝ๋‹ˆ๋‹ค! ํ•‘ํฌ์ƒ‰ ๊ณต์ฃผ๋‹˜ ์ผ€์ดํฌ ์ฃผ๋ฌธ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ์›ํ•˜์‹œ๋Š” ๋””์ž์ธ์ด๋‚˜ ํŠน๋ณ„ํ•œ ์š”์ฒญ ์‚ฌํ•ญ์ด ์žˆ์œผ์‹œ๋ฉด ๋ง์”€ํ•ด์ฃผ์„ธ์š”.\n", + "\n", + "๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค.\n", + "\n", + "[๊ฐ€๊ฒŒ ์ด๋ฆ„] ๋“œ๋ฆผ\n", + "Second example: user\n", + "๋‹ค์Œ์— ๋Œ€ํ•œ ์ด๋ฉ”์ผ ๋‹ต์žฅ์„ ์ž‘์„ฑํ•ด์ค˜.\n", + "\"11์›” 10์ผ, ์ €ํฌ ๋ถ€๋ถ€์˜ ๊ฒฐํ˜ผ 10์ฃผ๋…„์„ ๊ธฐ๋…ํ•˜๊ธฐ ์œ„ํ•œ ์ผ€์ดํฌ๋ฅผ ์ฃผ๋ฌธํ•˜๋ ค๊ณ  ํ•ฉ๋‹ˆ๋‹ค. ๋‘˜์ด ํ•จ๊ป˜ ๋จน์„ ์ž‘์€ ์‚ฌ์ด์ฆˆ์˜ ํ•˜ํŠธ ๋ชจ์–‘ ์ผ€์ดํฌ๋ฅผ ์›ํ•ฉ๋‹ˆ๋‹ค.\"\n", + "model\n", + "๊ณ ๊ฐ๋‹˜, ์•ˆ๋…•ํ•˜์„ธ์š”.\r\n", + "\n", + "๊ฒฐํ˜ผ 10์ฃผ๋…„์„ ์ถ•ํ•˜๋“œ๋ฆฝ๋‹ˆ๋‹ค! ๋‘ ๋ถ„์˜ ํŠน๋ณ„ํ•œ ๋‚ ์„ ๋”์šฑ ๋น›๋‚ด๋“œ๋ฆด ํ•˜ํŠธ ๋ชจ์–‘ ์ผ€์ดํฌ ์ฃผ๋ฌธ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ์ผ€์ดํฌ ๋ง›๊ณผ ํฌ๊ธฐ, ๋””์ž์ธ ๋“ฑ ๋‹ค๋ฅธ ์š”์ฒญ ์‚ฌํ•ญ์ด ์žˆ์œผ์‹œ๋ฉด ๋ง์”€ํ•ด์ฃผ์„ธ์š”.\n", + "\n", + "๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค.\n", + "\n", + "[๊ฐ€๊ฒŒ ์ด๋ฆ„] ๋“œ๋ฆผ\n", + "Third example: user\n", + "๋‹ค์Œ์— ๋Œ€ํ•œ ์ด๋ฉ”์ผ ๋‹ต์žฅ์„ ์ž‘์„ฑํ•ด์ค˜.\n", + "\"3์›” 15์ผ์— ์žˆ์„ ๋Œ€ํ•™๊ต ์กธ์—…์‹์„ ์ถ•ํ•˜ํ•˜๊ธฐ ์œ„ํ•œ ์ผ€์ดํฌ๋ฅผ ์ฃผ๋ฌธํ•˜๊ณ  ์‹ถ์Šต๋‹ˆ๋‹ค. ํ•™๊ต ๋กœ๊ณ ๊ฐ€ ๋“ค์–ด๊ฐ„ ๋””์ž์ธ์ด ๊ฐ€๋Šฅํ•œ์ง€ ๊ถ๊ธˆํ•ฉ๋‹ˆ๋‹ค.\"\n", + "model\n", + "๊ณ ๊ฐ๋‹˜, ์•ˆ๋…•ํ•˜์„ธ์š”.\n", + "\n", + "์กธ์—…์„ ์ง„์‹ฌ์œผ๋กœ ์ถ•ํ•˜๋“œ๋ฆฝ๋‹ˆ๋‹ค! ํ•™๊ต ๋กœ๊ณ ๊ฐ€ ๋“ค์–ด๊ฐ„ ์ผ€์ดํฌ ์ฃผ๋ฌธ ๊ฐ€๋Šฅํ•ฉ๋‹ˆ๋‹ค. ๋กœ๊ณ  ํŒŒ์ผ์„ ๋ณด๋‚ด์ฃผ์‹œ๋ฉด ๋””์ž์ธ ์‹œ์•ˆ์„ ๋งŒ๋“ค์–ด ๋ณด์—ฌ๋“œ๋ฆฌ๊ฒ ์Šต๋‹ˆ๋‹ค. ๊ถ๊ธˆํ•œ ์ ์€ ์–ธ์ œ๋“  ๋ฌธ์˜ํ•ด์ฃผ์„ธ์š”.\n", + "\n", + "๊ฐ์‚ฌํ•ฉ๋‹ˆ๋‹ค.\n", + "\n", + "[๊ฐ€๊ฒŒ ์ด๋ฆ„] ๋“œ๋ฆผ\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5NTIrFbJ3dBv" + }, + "source": [ + "In the context of a small dataset, the primary concern is that the model may prioritize memorizing specific examples rather than generalizing well to new and unobserved data. This limitation highlights the importance of utilizing a larger dataset during fine-tuning, as it enhances the model's ability to capture broader patterns and relationships." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "th0WS33gayn9" + }, + "source": [ + "## LoRA Fine-tuning" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ugc2ub4nau1j" + }, + "source": [ + "![lora.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAnYAAAJfCAYAAADy95sHAAAAAXNSR0IArs4c6QAAIABJREFUeF7snQd4VMXXxt/0XiCU0HtTUKT33omAgPSOFKkqqGBB/ipSBEVApIgUqdKb9I70DtJ7QkggCek9u983NxJSyfa9e/e9z+OjJjNnzvmdSfLu3JkzNmq1Wg0+JEACJEACJEACJEACFk/AhsLO4nPIAEiABEiABEiABEhAIkBhx4lAAiRAAiRAAiRAAgohQGGnkEQyDBIgARIgARIgARKgsOMcIAESIAESIAESIAGFEKCwU0giGQYJkAAJkAAJkAAJUNhxDpAACZAACZAACZCAQghQ2CkkkQyDBEiABEiABEiABCjsOAdIgARIgARIgARIQCEEKOwUkkiGQQIkQAIkQAIkQAIUdpwDJEACJEACJEACJKAQAhR2CkkkwyABEiABEiABEiABCjvOARIgARIgARIgARJQCAEKO4UkkmGQAAmQAAmQAAmQAIUd5wAJkAAJkAAJkAAJKIQAhZ1CEskwSIAESIAESIAESIDCjnOABEiABEiABEiABBRCgMJOIYlkGCRAAiRAAiRAAiRAYcc5QAIkQAIkQAIkQAIKIUBhp5BEMgwSIAESIAESIAESoLDjHCABEiABEiABEiABhRCgsFNIIhkGCZAACZAACZAACVDYcQ6QAAmQAAmQAAmQgEIIUNgpJJEMgwRIgARIgARIgAQo7DgHSIAESIAESIAESEAhBCjsFJJIhkECJEACJEACJEACFHacAyRAAiRAAiRAAiSgEAIUdgpJJMMgARIgARIgARIgAQo7zgESIAESIAESIAESUAgBCjuFJJJhkAAJkAAJkAAJkACFHecACZAACZAACZAACSiEAIWdQhLJMEiABEiABEiABEiAwo5zgARIgARIgARIgAQUQoDCTiGJZBgkQAIkQAIkQAIkQGHHOUACJEACJEACJEACCiFAYaeQRDIMEiABEiABEiABEqCw4xwgARIgARIgARIgAYUQoLBTSCIZBgmQAAmQAAmQAAlQ2HEOkAAJkAAJkAAJkIBCCFDYKSSRDIMESIAESIAESIAEKOw4B0iABEiABEiABEhAIQQo7BSSSIZBAiRAAiRAAiRAAhR2nAMkQAIkQAIkQAIkoBACFHYKSSTDIAESIAESIAESIAEKO84BEiABEiABEiABElAIAQo7hSSSYZAACZAACZAACZAAhR3nAAmQAAmQAAmQAAkohACFnUISyTBIgARIgARIgARIgMKOc4AESIAESIAESIAEFEKAwk4hiWQYJEACJEACJEACJEBhxzlAAiRAAiRAAiRAAgohQGGnkEQyDBIgARIgARIgARKgsOMcIAESIAESIAESIAGFEKCwU0giGQYJkAAJkAAJkAAJUNhxDpAACZAACZAACZCAQghQ2CkkkQyDBEiABEiABEiABCjsOAdIgARIgARIgARIQCEEKOwUkkiGQQIkQAIkQAIkQAIUdpwDJEACJEACJEACJKAQAhR2CkkkwyABEiABEiABEiABCjvOARIgARIgARIgARJQCAEKO4UkkmGQAAmQAAmQAAmQAIUd5wAJkAAJkAAJkAAJKIQAhZ1CEqmUMC5fvoyLFy8iKSkJPj4+6NChA+zt7ZUSHuMgARIgARIgAaMSoLAzKl4a15ZA2bJlce/evbRu27dvh5+fn7Zm2J4ESIAESIAErJIAhZ1Vpl2+QZcqVQoPHz5Mc3DLli3o2LGj2R1+8uQJevXqBfFvPiRAAsoh4OjoiEWLFqFBgwbKCYqRWDUBCjurTr/8gpersNu8eTM6d+4sP2D0iARIQG8CH330EX7++We97dAACciBAIWdHLJAH9IIyFXYbdq0CV26dMGQHtUwZ3JbZowESEABBDb8fQN9P9mEsWPHYvbs2QqIiCGQAEBhx1kgKwJyF3aj+9fCnMntZMWMzpAACehGYOPf/6LryPUUdrrhYy+ZEqCwk2lirNUtCjtrzTzjJgHTE6CwMz1zjmh8AhR2xmfMEbQgQGGnBSw2JQES0IsAhZ1e+NhZpgQo7GSaGGt1i8LOWjPPuEnA9AQo7EzPnCManwCFnfEZcwQtCGgq7JKTk01auPjl4QnusdMimWxKAjInQGEn8wTRPZ0IUNjphI2djEUgJ2EXHR2NZcuWYdWqVbh+/ToiIyNRpkwZVK9eHTVq1EC/fv1QsGBBY7kFCjujoaVhEjAbAQo7s6HnwEYkQGFnRLg0rT2B7ISdi4sLhg4dikePHuVosGjRoti6dSuqVaum/aAa9KCw0wASm5CAhRGgsLOwhNFdjQhQ2GmEiY1MRSCzsGvTpg327t0LlUqVqwtCAO7btw/169fPta22DSjstCXG9iQgfwIUdvLPET3UngCFnfbM2MOIBDILu8xDeXl5oUKFCvD398fTp0+zeNKiRQtJ3Bn6obAzNFHaIwHzE6CwM38O6IHhCVDYGZ4pLepBICdh5+DggEmTJuHjjz+Gm5ubNMLu3bvRrVs3REVFZRjx4sWLqFq1qh5eZO1KYWdQnDRGArIgQGEnizTQCQMToLAzMFCa049AdsLO29tbOrzQtGnTLMZ37twJPz+/DF//5ptvMHnyZP0cydSbws6gOGmMBGRBgMJOFmmgEwYmQGFnYKA0px+BzMLO2dlZerXaoEGDbA2r1Wq8+eabuHHjRtr3hwwZgkWLFunnCIWdQfnRGAnIkQCFnRyzQp/0JUBhpy9B9jcogczCbu7cuRg1atRrxxgzZgxEu5ePWMHbvn27Qf3iip1BcdIYCciCAIWdLNJAJwxMgMLOwEBpTj8CmhYoTj/KDz/8gC+//DLtS6K23blz5/RzhCt2BuVHYyQgRwIUdnLMCn3SlwCFnb4E2d+gBHQRdn/88QcGDx6c5od4NXvt2jWD+sUVO4PipDESkAUBCjtZpIFOGJgAhZ2BgdKcfgR0EXarV69G7969Kez0Q8/eJGB1BCjsrC7lVhEwhZ1VpNlygqSws5xc0VMSsHQCFHaWnkH6nx0BCjvOC1kRoLCTVTroDAkomgCFnaLTa7XBUdhZberlGTiFnTzzQq9IQIkEKOyUmFXGRGHHOSArAhR2skoHnSEBRROgsFN0eq02OAo7q029PAOnsJNnXugVCSiRAIWdErPKmCjsOAdkRYDCTlbpoDMkoGgCFHaKTq/VBkdhZ7Wpl2fgFHbyzAu9IgElEqCwU2JWGROFHeeArAhQ2MkqHXSGBBRNgMJO0em12uAo7Kw29fIMXO7CbnivGpgzua084dErEiABrQhs2HUdvcZuxNixYzF79myt+rIxCciVAIWdXDNjpX7JVdht3rwZnTt3ttKsMGwSUDaBjz76CD///LOyg2R0VkOAws5qUm0ZgdasWROXL1+Gi4sLoqOjcfjwYTRs2PC1zu/duxdt2rRB3rx5ERoaiiZNmuDQoUMGDTgsLAwtW7aE+DcfEiAB5RBwc3PD3Llz0bRpU+UExUismgCFnVWnX37BJycnIzg4WBJ2Dg4O8PDw0MjJ8PBwqNVqxMbGwtfXF3Z2dhr1YyMSIAESIAESUBIBCjslZZOxkAAJkAAJkAAJWDUBCjurTj+DJwESIAESIAESUBIBCjslZZOxkAAJkAAJkAAJWDUBCjurTj+DJwESIAESIAESUBIBCjslZZOxkAAJkAAJkAAJWDUBCjurTj+DJwESIAESIAESUBIBCjslZZOxkAAJkAAJkAAJWDUBCjurTj+DJwESIAESIAESUBIBCjslZZOxkAAJkAAJkAAJWDUBCjurTj+DJwHlEvhz23ks3nBG6wA/G9wEfo0rad2PHUiABEhADgQo7OSQBfpAAiRgcAKLNpzG/DUntbb7zYct8F6Lylr3YwcSIAESkAMBCjs5ZIE+kAAJGJxAemHXvnFFuDg65DiGf3A4Tl/xl75PYWfwVNAgCZCACQlQ2JkQNociARIwHYH0wu7U6pFwdnqNsAsKx7sjl1HYmS49HIkESMBIBCjsjASWZkmABMxLgMLOvPw5OgmQgHkIUNiZhztHJQESMDIBCjsjA6Z5EiABWRKgsJNlWugUCZCAvgQo7PQlyP4kQAKWSIDCzhKzRp9JgARyJUBhlysiNiABElAgAQo7BSaVIZEACQAUdpwFJEAC1kiAws4as86YDU5ApVJj8cazcHG0Q1FfLxQv5I2iBb1eexLT4E7QYAYCFHacECRAAtZIgMLOGrPOmI1C4Ow1fwz73yYIkffyyeftimK+3pLQK1zQG8V8PaX/L+HrDU8PZ6P4QaOpBCjsOBNIgASskQCFnTVmnTEbjcDC9Wfw29oTGtn3cHWSVveKSf8I0eeFov/9u0Bed9jY2Ghkh42yJ0BhZ76ZkZScgrCIWIS+iMXzFzEICRf/xCHkRTT8g8IRF5eEkBcxaFC9JCYOaWY+RzkyCSiQAIWdApPKkMxHQKzWdf34T9wPCEtzonI5XwQEhSM8Kl5jx5wc7VCkgBB6XigmVvoK/Sf8CnqhcAFPONjbaWzLWhtS2Bk+85Ex8RnEWmh4rCTQJOEm/v0iFqHhMRrN9XcqFsKiyV3h4MC5bPhM0aI1E6Cws+bsM3ajEBArFT3Gr8KzsBjY29ti2ZRuqFzWFzGxCfAPisCjoHA8CYqQVi7E/4t/B4dGa+yLra0NfPN5SCt89auWRL+O1TXua00NKey0y/aNe8HSnE1dYUsVaEKwhYbFQAi45+ExSExK0c5oDq3dnB2wff5A5PVyNYg9GiEBEnhFgMKOs4EEjEDg4o0n+GDSBqT8/6GKIvk9sWZWL3i65bynLjEpGU+CI9OEnhB7AcER8H8ajifPI5GcrMrWyw5NKuHb0a2NEIHlm6Sw0y6HTQcuxIvIOO06ARAfNPJ6usAnjxvy5XFDfm83+Hi7Sv8tvpbf21Wy+8mMHZJtZyd7LJ/SDRVKFdB6LHYgARLInQCFXe6M2IIEdCKwZPNZzF35j9S3Sa3SmP15B53siNe7QSFRktB7/DQcNx88w4a9VyVbH/aoh2Hv19LJrtI7Udhpl2GxheDu49C0TmJVTYgzSaD9J9TypRNt+f/7eh5PV0nc5fQIUdf78zUIfBYpNZk5vj1a1C2nnXNsTQIkoDEBCjuNUbEhCWhHQK1WY8zUrTh2/qHUcVz/hujbQf/XpjfvP0OPT1dLNqeMbYP2jSpq55gFtBab7xMSkvXydPHGM1i+9bxk49Tqka8tPSNWSN8duSwtT+81r6zX2C7ODrCzs9XLhqk7D/92E05dfgx3F0fsXzLEIKV6RB6H/28Tzl9/IoUzrFsdfNi9jqlD43gkYFUEKOysKt0M1tQEIqPi0Xro74hLTIa9nS2WfNcVb1corJcbe0/ewWczd0o2/pzWA1XK+eplT46ddx69iS9/2W0w17QRdoYYdNHkLqhVpZghTJnMxtdz9mD7kRvS6tyBJUMNMu73Cw5gw77U1eWWdcthxrh2PO1tELI0QgI5E6Cw4+wgASMSuHQzEAO+/CtthBpvFsXv33bVa8T0r3gPLxsGbw8XvezJsTOFnemzMvvPY1i25bz0WvXcujGvfb2qiXdrd13GtN8PSU0rlMov7atzdnLQpCvbkAAJ6EGAwk4PeOxKAq8jIF4Dzll5XDpAIZ5mdcriu5Et4ebqpBe4/83fh80H/oWHqyOO/TlCL1ty7Zxe2Pk1qohSxX30crV/h+rSimlOjyjjsf6/fYu6DnTmymOcvuIvdbfEFbs/t1/ArGVHJf8PLBkCH283XVFAnLDtM2Ft2tz/dlQrdGj6hs722JEESEBzAhR2mrNiSxLQiIAQCV/P3YsjZ+9L7YWg+KhfA/Txq6ZR/9waDflmI8QtFxVL5cfamb1za26R308v7Db83Adli+eTfRxif5rYp2apwu7vYzfxxezU199/zeqN8iXz68xcHPj5+c9j+HPbBcmGna0NPh3UBD3avq2zTXYkARLQjACFnWac2IoENCJw/W4wxs/amXYCUNSbmzmuHSqXL6RRf00atRm2RDol26qe2LPUXpMuFteGws70KRMfFsSHBvH8+lUn1H+npN5ObDt0Hd8tOABxiEI8XVtVwYQPmr529VTvQWmABKycAIWdlU8Ahm84Amv+voRZy4+m1ZxrWL0kpoxuY9A7YZOSUlCr51yo1cCgzjUxpnd9wwUgI0sUdqZPxoOAMLw3doU08P9GtkTHZm8axInLtwLx8fQd0hVj4hH7TGd+2l6Re0MNAoxGSEBPAhR2egJkdxKIjUvE5Pn7sPfEHQmGeO00olc9DOpUw+AnANP/8f3mwxZ4r4V+ZTnkmj0KO9NnRmwhaNRvgTSw+MAgPjgY6gkKicbYaVtx68FzyWSRgp74ZUJHlNVz76Sh/KMdElASAQo7JWWTsZicwO2Hz/HpzJ149DRcGlsUbZ3+STtUe6OIUXw5ev4hxvywRbK9+H9dULOyZZXU0BQKhZ2mpAzbrlaPudK1YT3bVcXng5sY1Hh8QhK+mrMH+0/dleyKAsg/fNQWjWuWNug4NEYC1k6Aws7aZwDj15nApv3XpHIOL+/PrP1WMUz9qK1R779cvfMiZvxxRPJ598IP4JvPXWf/5dyRws482Wk//A/pCjtj7d8URbsX/HUaC/86JQVoYwOM6l0fg98z3OqgechxVBKQDwEKO/nkgp5YCAGx8jBl4UGpmOvLP05D36+DYe/X1rv2V24Ipi85DLGXz9HBDqfXjDL4q97cxjfV9ynsTEU64zj9JqzFlTtBqP5GESz57n2jOSGKbH8zZ49UuFs87RtWwKQRLeHkaG+0MWmYBKyFAIWdtWSacRqEwP2AUOnV6z3/MMleHk8XaZWuztvFDWI/NyOjpmzB8QsPUapIXmye0y+35hb7fQo786Tu4xnbcej0PZQo5I2t8wYY1QlxNd7YadsQHBotjVOlbEH8NKGDtJ2BDwmQgO4EKOx0Z8eeVkZAiI3vFuxH/H93mIpVjWmftDPpH6JOo5fjYeALNKxeCnO/6KjYDKQXdt3bvKV3HTtxyOR1BYpjYhPw97FbevE8dfUxDpxM3T9miQWKhd9TFh3E+j1XpP1v/6waqRcPTTqHhsdg3IwduHTrqdS8QF43zP68A94oW1CT7mxDAiSQDQEKO04LEsiFQEJiMsQrULGn7uUz8L0aGNWznkkvehdFX0Wpk+RkFXq1r4rPBhl2c7ucJgKvFDNPNsT+twXrTkqD53a/rqE8FCV8vl94AFsPXZdMOjnaYfLIVmjboIKhhqAdErAqAhR2VpVuBqstgcdPw6VXr7ceppZp8HB3wpQxbdGouv7FW7X15enzKLQdvkTqJk4sipOLSn0o7MyT2Q17r0oiSzw7fh2Ior5eJnNkxdbz0m0VokajeD7oUgsje9ZV7D5Sk4HlQFZHgMLO6lLOgDUlIDZ4f/vrPkT/f5068VQu54sfx7VHofwemprQqd09/1D8sfksJn/YEg4Odmk2zlz1x9DJqTcDzPuyIxpUK6WTfUvoJA6ohEcl6OXq7xtOY8O+q5KN3Faf/IPC8e7IZVLbYd3q4L3m+tUHzOvlDEcHyzsIcPjsfXw0bZvEYen33fBOpcIZciBWjW1tbfTKy+s6H7/wABN/3oWo2NSfuaa1y2DK6NZwdXE02pg0TAJKI0Bhp7SMMh69CYhXQ+IGibW7LqfZ6t3+Hem+Vwf7V0JL74GyMbDjyA1pxUTs48tcS2zjvqvS9Uzi2Tq3P0oUzmMMFxRjc9GG05i/RrPXiumFnZILP+eW3Gt3g9Dn87VSs5nj26NF3XJpXZ6/iMHw/23C+IGNUPftErmZ0vn7ogj32Klb8TgoQrIhihj/MrEjihTw1NkmO5KANRGgsLOmbDPWXAkEPovE57N24urdYKmth6sjJo9sieZ1Xv2By9WIDg2y28cnzMwY3x6t/vvj+sufx7F0yzmp9teZNaMzrObpMKTiu1DYaZ9icUNEm2G/Sx3Fna492r4t/beYnx98vV76uRArduP6N0Jvv3e0H0DDHlHR8dKdy6ev+Es9xOnzWZ/6Ga3wt4ZusRkJWAQBCjuLSBOdNAWBo+ce4Ks5uxEZk/oKsGKp/Jg53s/o+4wy7+NLH6u7iyNWzegprc6N/3GHVLVfvAretWCwKZBY9BgUdtqnT6xW1+wxV+o4pKvY41ZP+u8vZ+/Czkynht9r/ia+GNrMaKvYySkqzFx6JG3lXJxqFuN1Vug1etpniz1IIHsCFHacGVZPQPwBmbvqHyzfej6NRdeWVfDZ4MZG3yeV0z6+f+8HY/yMHZI/5Urkw59Te6D/F+ukQxy1qhTHosmdrT5vuQGgsMuNUPbfb9x/ASKi4yGE2zcjWmLJ5rOYu/IfqbFYLXZ3c0JUdOqHn2qVCmPmp35GvW1FHOgQN7yIn1PxiFXETwc2NumJdN1IshcJmIcAhZ15uHNUmRAIDo3BhJ924uLNQMkjF2cHfDWsOdo3qmhUD7PbxydKmHzcr2HaCsjMZUexcvsFyY9Ozd7EvhO3EROfhC4tK+Pr4S2M6p8SjFPY6ZbFLh+tkApwN6xeEu+3ehtjp22VTqoW9HHH6hk9kZwCfDRtK27cfyYNUCS/J2ZP7CB9ADHWc+7fAGnFOjwqXhpCFASfMa4dPN2cjTUk7ZKAxRKgsLPY1NFxfQmcvPwIX8zejReRcZKpMsV8MOvT9ihZJK++pl/bP/M+PvG6VezjS79RXRgQKxRiX9PL4q0vjY7t0wCijh6f1xOgsNNthgydvAlnrj6Wrq1zsLOVPky4ONpj6ZRuqFi6gGRUHO6ZNH8v9h6/Lf2/+EAkbmBpUrO0boNq0OtJcATGTN0GcWpcPOJ2jF8mdjD6z6sGrrEJCciKAIWdrNJBZ0xBQJRs+G3dSfy+8UxazayOTd/AxCFN4ezkYFQXxD6+L+fuTnuVVUHax9cexXy9sx03ODQK3cetSlupEI0yn1Y0qsMWbJzCTrfkZbefbvq4dmhdr3wWgwvXn8Fva0+kfX1kr3oY0qWWbgNr0Cs2LhETf9mNI2fvS63F4SZx+0v9d0xfV1IDd9mEBMxCgMLOLNg5qLkIiCuMJvy8G2evpZ62E1XuJw5pJr3qNOYjVt/mrf4Hy7Zk3Mf36aDGuV58fuLSI4z4bnOae3/N6o3yJfMb011F2Kaw0y2N4sDCyh0X0zoPfb82RvSom6OxA6fu4Ms5e9Ku2mvbsAImj2iZ67zWzTtArVZj7uoT+GPTWcmEOKUrtjD0fbearibZjwQURYDCTlHpZDCvIyD26Uz46W+EhMdKzcSrHLHx25h7g8Q4z8KipXEv3Ejdx+fsZC/tkdNmH9/8tSexaP1pqf+JlSNYsFWDqU5hpwGkbJqIGyB+WnFM+k79aiUx74uOud7+cPvhc+k1aVBIlNSvcpmC+GnCuyiQ1103JzToJW4n+d/8fUhMSpFai1V3sT82fVFvDcywCQkojgCFneJSyoAyExCf8H/fdBa/rT0J8RpWPK3rl4coRGvsivanLj/GxNm70u3jy4sfx7dH6aI+WiVK+P3hd5tw+2EIDi0dplVfa21MYadb5l9e51ahZH4s/6GbxtsTwiJiMW7Gdly8+VQaOH8eN/wyoQPeKFtQN0c06CUKKn88bTtE8WTxvFOxEGZ99q5RT+lq4BabkIBZCVDYmRU/Bzc2gfCoOOk10T8XHkpDiZsjPh3YCN3apBZeNdYjhNjC9aexaP2ptH187zauhC+HNdP4D2Vm38QfzimLD2LWeD9jua0ouxR2uqVTXF33+U9/Y/WMXlpfnydOe4ubU7Yeui4NLrY6TB7ZCm0bVNDNGQ16iRXxT6Ztx7V7qUXFffN54JcJ76JCqdSDHnxIwNoIUNhZW8atKN7LtwLx2ay/ERwaLUVdpKCnJIpenuwzFgohwMQq3cuq+eKP24TBTfGeAQqris3jxl5lNBYXU9ulsNONuCiYHRoem+WeWG2s/bntPH7+83jaCvkHXUSx47q5vtLVZoz0bcXNGN/8ug+7j9+Sviy2O0wZ09roN8bo6i/7kYAxCVDYGZMubZuNQOaipuIy8f+NbGn0ulcXrj+RVjtevhoS+/jEq1cedjD9VKCwMz3z9CP+c/GhtLc0KjZR+rL4GZwyurVRP5gs2XhGOljx8hnWrQ6Gd6ttNEFpXsIcnQSyJ0Bhx5mhSAInLj7EiO+3pMVm7AMHYh/f0s3nMG/NibRVilb1ykmnA7nCZp4pll7YdW/z1mtfgfsHR+DgqbuSo2LvpSFWV80TtbxGfRAQhrFTt+JxUITkmDioJPbdFS7gaTRHD525hy9+2Y24+CRpjJZ1y+G70a103gJhNEdpmASMRIDCzkhgadb8BH5dcwKLN5yRHGnToAKmfdzWKE5FRsVLtemOnU/dx2dvbytdkt6zXVWjjEejmhFIL+w065HaisJOG1q5t42Kjsf4WTvTtibk8XTBT5/54Z1KRXLvrGOLO49C8NHUbXjyPFKyIO59nj2hI3zzGe+Uro6ushsJGJwAhZ3BkdKgXAiIAwzD/rcprWbdF0OaGvzQxLXbTzF+1t9pZR7E9Uri1asxTwLKha/c/RCv5X5de1JrNyd92MLodQ21dsrCO4g6jj/+cRjrdl9J+/Dz1dBm6NS8stEiEzfKDPzyLzwMfCGNkc/bFVvn9oebq5PRxqRhEpADAQo7OWSBPhiNgChI3G3cKmkzuDgRu+KHbqhUxjDlF1btuIifVxxLu5y8cc3S+H5UK3i48/5KoyWUhi2awPq9VzD998NpPzO927+DT/o3hJ2drcHjEid0hbB7eVrW2LdiGDwAGiQBHQlQ2OkIjt0sh4AoTDx08kZp75vY27N2Zi+9DlHExCZg0vx9OHAydU+Wna0NxvRpgP4dq1sOFHpKAmYiIG59Gf/jTkREx0se1K1aAj9+0g7uboZdSftm3t60siviNgxxly0fErAGAhR21pBlxojF4rXcf6flmtQqjdmfd9CJyq0Hz6Q/SmKzvXgK5HXD9E/a61UaQidH2IkELJhAQFDrD2qzAAAgAElEQVQExk7binv+YVIUJQvnwS8TO6BE4TwGiSr97RniFowl379vtCvODOIwjZCAAQlQ2BkQJk3Jl4A4tTr6h604/l+h4k/6NUQ/LVfYxGukH/84knaFkVhp+GFsG4jN4HxIgAS0IyBqMk6YvQtHzz2QOnq6OWHGuPao83Zx7Qxlan38wgPpZ12tTv3gtWpGL+kWDD4kYC0EKOysJdOME+L0arfxq6SDDuL16ZLv3kfVioVzJSPKJny/YD92HkstfmpjAwzvXg9DutSULiDnQwIkoBsBsT1izqrjWLblvGRA/FyOH9hY5xPl9wNC0W/COkTHJUq3Xiz9rhsPMumWGvayYAIUdhacPLquPYErt59i0Ffrpc3b4oLydbN6v3bF7Z5/qPTq9cGT1FdGeb1cpbIptaoU035w9iABEsiWwPbDN/Dtb/uRlJwifb9Ly8qY8EFT6cCTpo/44NZnwpq0mnlTP25r1KvMNPWL7UjA1AQo7ExNnOOZncDKHRcwc+lRyQ/xOvXXLztlu/Im/tj8sPAA4hKTpbbV3yiC6Z+0Qz6+1jF7DumA8giID10fT98unWB/+fM281M/jbY6iA9qH367Oa20kbjCbFSvesqDxIhIQAMCFHYaQGIT5RH45McdaTcNfNijHoa9XystSHHv5LTfD2HzgX/Tvjaoc02M7FHXKGUZlEeXEZGAbgSCQ6Mwduo23HzwXDIgTrHPmdgBZYvne63BHxYdxF97UmvkiavLfvrUj9eI6ZYC9lIAAQo7BSSRIWhPIDomAT0/XS2dbhV75hZ+00V6vfoo8AU+nbkTtx+FSEa93J0xZWxrNKhWSvtB2IMESEBrAvEJSZg0by/2nrgj9XVzdsAPH7WFqBOZ3bN+zxVMWXRQ+lb5Evmw/IfucHF20HpcdiABpRCgsFNKJhmH1gRu3n+Gfl+sk065ir1zH3SpKZVEifnvjsm3yxfC9HHteQ2R1mTZgQT0J7Dgr9NYsC715hDx4Wt0r/oQK+fpnzNX/fHht5uQolJLr2xXTe9p1Hto9Y+KFkjA+AQo7IzPmCPImMCGvVfx/cIDWTzs4/cOxvZtoNXmbRmHSddIwCIJ7D95B1/P2ZO2z7V9wwqYNKKlVJPOPygcvT5fg6joBOl+5sWTuxj1/lmLBEinrZIAhZ1Vpp1Bpyfw1S+7sePoTelLHq6O+N+oVmhWuywhkQAJyICAKAo+dtr2tPuYq5QtiO/GtMYnM3bgfkDqafXJI1oY9d5ZGWCgCySgMQEKO41RsaFSCYg6db0/XwNHBzvMHO+Hor5eSg2VcZGARRIQdz6Pm7EDl249zXZ1XdS+40MCJJBKgMKOM4EEAASFRCOvlzMcHezJgwRIQIYEkpJSpG0TWw9dT/Ou3jslMHdiR55Wl2G+6JL5CFDYmY89RyYBEiABEtCSgLgH9uc/j0n3y/45tQfc3Zy0tMDmJKBsAhR2ys4voyMBEiABxREQ98EW8/VGicJ5FBcbAyIBfQlQ2OlLkP1JgARIgARIgARIQCYEKOxkkgi6QQIkQAIkQAIkQAL6EqCw05cg+5MACZAACZAACZCATAhQ2MkkEXSDBEiABEiABEiABPQlQGGnL0H2JwESIAESIAESIAGZEKCwk0ki6AYJkIB8CISFheHo0aNQq9XycYqekAAJKJpA3rx50bix/sW2jSLsLl++jIkTJyo6AQyOBEhAfgScnZ2xadMmvR1r3rw5Dh48qLcdGiABEiABbQicO3cO1atX16ZLlrZGEXYHDhxAixYt9HKMnUmABEhAWwKurq6IiYnRtluW9nPnzsWYMWOkr7et4IUiHg5626QBEiABEsiOwO/nQtK+HB0dDTc3N71AGVXYeReriHqDftDLQXYmARIgAU0I/P2/zjCUsBPjffDBB1iyZAne8nXGzgEV4OZgq4kbbEMCJEACGhNYczkEI7f5w8vLC2fOnEH58uU17ptTQ6MKu1J1O6DuoKl6O0kDJEACJJAbgfVj6sDBJsUgK3ZirMTERIhXssePH0fHN7yxtEup3Fzg90mABEhAYwJnAqLx7oq7SFHbYPfu3WjZsqXGfV/XkMLOIBhphARIwNwEDC3sRDzPnz9HjRo18PjxY0xo7IvPGhUyd5gcnwRIQAEEAiKT0OL3m3gWk4zZs2dj7NixBouKws5gKGmIBEjAnASMIexEPOIwWL169RAbG4vl75fEuxV5P6k588yxScDSCcQmqdB++R1cfhqLwYMH4/fffzdoSBR2BsVJYyRAAuYiYCxhJ+LZvHkzOnfuDFcHW+weWB6VC7qYK0yOSwIkYOEEBm58gK3Xw9GgQQOIw6aOjo4GjYjCzqA4aYwESMBcBIwp7ERM3377Lb755hsU9XLEgcHlkN/NsL+MzcWN45IACZiOwI/HgjD18FMUK1YM58+fR/78+Q0+OIWdwZHSIAmQgDkIGFvYiWLF3bt3x/r161GnmBu29C0LRzuelDVHrjkmCVgige03X6D/+ofS6f0TJ07g7bffNkoYFHZGwUqjJEACpiZgbGEn4hH77OrXr49Lly6hzzs+mONX3NRhcjwSIAELJHAtOA5tlt6G2F+3ceNGaWuHsR4KO2ORpV0SIAGTEjCFsBMB+fv7Sydlnz17hh9aF8HwWgVMGicHIwESsCwCz2MS0XzJHQREJOJ///sfJk2aZNQAKOyMipfGSYAETEXAVMJOxHPy5Ek0adIEyUmJWN+rNJqW9jJVmByHBEjAgggkpqjQ6c+7OOUfg/fffx/r1q2DjY2NUSOgsDMqXhonARIwFQFTCjsR0/LlyzFgwAB4O9th76DyKOvjbKpQOQ4JkICFEBiz4zFWXgxF1apV8c8//0j764z9UNgZmzDtkwAJmISAqYWdCGr8+PGYNWsWyvo4Yd+givBy5mEKkySbg5CABRBYcOYZvtjzBAUKFMC5c+ekk7CmeCjsTEGZY5AACRidgDmEnUqlgp+fH3bt2oVmZTywtkcZ2Nsa9zWL0UFyABIgAb0JHLofgfdX34e9gyMOHz6MunXr6m1TUwMUdpqSYjsSIAFZEzCHsBNAIiIiULt2bdy6dQsf1s6PKa2KypoTnSMBEjAugbuh8Wj1x22Ex6dg2bJl6N+/v3EHzGSdws6kuDkYCZCAsQiYS9iJeO7evYuaNWsiPDwcc94tjj5VfYwVJu2SAAnImEBEvAot/7iJu6EJGDduHGbOnGlybynsTI6cA5IACRiDgDmFnYhn//79aNOmDWyhwra+ZVC7mIcxwqRNEiABmRJIVqnRY+09HLwXhbZt22LHjh2wtTX9vlsKO5lOELpFAiSgHQFzCzvh7dy5czFmzBjkd7PHgcEVUdTLQbsg2JoESMBiCXy5NwC/nX6OChUq4PTp0/DyMk8ZJAo7i51CdJwESCA9ATkIO+HP0KFDsXjxYlQu6IxdAyvAzcH0n9g5M0iABExLYPXlEIza5g9vb2+cPXsWZcuWNa0D6UajsDMbeg5MAiRgSAJyEXZJSUlo0aIFjh49ig6VvPFHl5KwNXJBUkNypC0SIAHtCJwJiMa7K+5CBVvs3r1b+vk350NhZ076HJsESMBgBOQi7ERAISEh0rVjjx49wmeNCmFCY1+DxUlDJEAC8iEQEJGE5ktu4nlMMubMmYPRo0eb3TkKO7OngA6QAAkYgoCchJ2I5+rVq1LtqpiYGCzrWkpaveNDAiSgHAIxSSq0W3YLV4PiMWTIECxatEgWwVHYySINdIIESEBfAnITdiKeLVu24L333oOrgy12DSiHKr7Gv05IX47sTwIkkDsBlVqNwRsfYuuNcDRq1Eg6Fe/gII/DUhR2ueePLUiABCyAgByFncA2ZcoUfPXVVyji6YADQ8qjgKujBdCkiyRAAq8jMP3oU0w/EoQSJUpI14Xly5dPNsAo7GSTCjpCAiSgDwG5CjsRU8+ePbF27VrULuqGLf3KwcmO147pk2v2JQFzEth2IxwDNjyAm5sbTp48iSpVqpjTnSxjU9jJKh10hgRIQFcCchZ2cXFxaNCgAS5cuIDeVfNg7rsldQ2T/UiABMxI4GpQLNouu4PYJBU2b96MTp06mdGb7IemsJNdSugQCZCALgTkLOxEPAEBAdJJ2eDgYHzfqghG1C6gS5jsQwIkYCYCz2IT0XzxbTyJTMJ3330nbbGQ40NhJ8es0CcSIAGtCchd2ImARDV6sdE6OSkRf/Usg2ZlPLWOkx1IgARMTyAhRY1OK+7gdEAMunfvLm2tkOtDYSfXzNAvEiABrQhYgrATAf3555/o168fPJ3tsXdgOZTP56xVnGxMAiRgegKjtz/EqksvUK1aNRw/fhwuLi6md0LDESnsNATFZiRAAvImYCnCTlD87LPP8OOPP6JMXifsHVweeZzt5Q2X3pGAFROYf/oZvtr7BAULFpROwBYtWlTWNCjsZJ0eOkcCJKApAUsSdiqVCh06dMDOnTvRtLQH1vUsA3tbnpTVNNdsRwKmInDwXiS6rbkHewdH6ZrA2rVrm2poncehsNMZHTuSAAnIiYAlCTvBLTIyEnXq1MGNGzcwvFY+/NC6mJxw0hcSsHoCt0Pi0WrpHUTGJ2PFihXo27evRTChsLOINNFJEiCB3AhYmrAT8dy7dw81a9bEixcv8ItfcfR9xye3MPl9EiABExB4EZ+MVktu415YAj799FPMmDHDBKMaZggKO8NwpBUSIAEzE7BEYSeQHTx4EK1atYItVNjatwzqFPMwM0kOTwLWTSBZpUb3Nfdw6H4U2rdvj23btsHW1tZioFDYWUyq6CgJkMDrCFiqsBMx/frrrxg1ahTyudnjwOCKKOYljzsnOeNIwBoJfLk3AL+dfo5KlSrh1KlT8PS0rLJEFHbWOGsZMwkokIAlCzuRjuHDh2PhwoV4s6Azdg0oD3dHOwVmiSGRgLwJrLwUijHbHyNPnjw4e/YsypQpI2+Hs/GOws7iUkaHSYAEsiNg6cIuKSkJLVu2xJEjR+BX0QvLupaCrQ1PynK2k4CpCJzyj0LHP+9BBVvs3bsXzZo1M9XQBh2Hws6gOGmMBEjAXAQsXdgJbqGhodK1Yw8fPsSnDQtiYpPC5sLJcUnAqgj4RySh+ZKbCIlJxrx58zBy5EiLjZ/CzmJTR8dJgATSE1CCsBPxXLt2DXXr1kV0dDT+6FISnd7Iw0STAAkYkUB0YgraLbuNa8HxGDZsGBYsWGDE0YxvmsLO+Iw5AgmQgAkIKEXYCVTiFF6nTp3gbG+Dv/uXw9uFXE1AkEOQgPURUKnVGLjhAbbfjEDjxo2xb98+ODhY9uElCjvrm8eMmAQUSUBJwk4kaOrUqfjiiy9Q2NMBBz6ogIJulv3HRpGTjkFZPIFpR4Iw4+hTlCxZUrouzMfH8mtJUthZ/LRkACRAAoKA0oSdiKl3795YvXo1ahZ1w7Z+5eBkx8MUnO0kYCgCW66/wKCND+Hu7o6TJ0+icuXKhjJtVjsUdmbFz8FJgAQMRUCJwi4+Ph4NGjTA+fPn0evtvJjXoYShcNEOCVg1gctPY9Fu+R3EJamwdetW6e5mpTwUdkrJJOMgASsnoERhJ1IaGBiI6tWrIygoCN+1LIqRdfJbeaYZPgnoRyA4JgnNf7+FwMgkTJkyRdryoKSHwk5J2WQsJGDFBJQq7ERKz5w5g0aNGiEpMQHrepRB87KWVQnfiqclQ5cZgYQUNTqsuIOzATHo2bOntNVBaQ+FndIyynhIwEoJKFnYiZSuWrUKffr0gaeTHfYMqoAK+ZysNNMMOzcCahtX2BaqBDvvfLCxiYcq9BaSg4Ngo86tp/K/P2rbI6y+HCatgh87dgwuLi6KC5rCTnEpZUAkYJ0ElC7sRFYnTJiA6dOno7SPE/YPLA9vF3urS7barhKcu38BJ8/XHSRRQ52cBHVCGFQvHkEVcAYJN85AnaBsZaO28YF97ZFwqdMS9nnckHZxiVoFdcQlJByfg/izV6BWNoYcfyZ+PfUcX+8LgK+vr7RvtXBhZRYAp7Czul+LDJgElEnAGoSdSqVCx44dsWPHDjQp5YG/epWBva11nZRV2xSCfc2ucK7XCw55nSBFrxYiLhrqxKTUyW3jADh7wsbBLvX7UEMddwuJe75B7IU7ivwBUNsWg2PnuXCrUhw2yY+ReGIZEu4/AtyqwLH+IDgW9oQN4pByYRKith2wOnF34G4kuq+9BwdHJxw9ehS1atVS5DyQpr9abXjtfuDAAbRo0QKl6nZA3UFTFQuPgZEACciHgDUIO0E7KioKderUwfXr1zG0Zn5Ma1NUPkkwpSc15sL73frSqpQ64TBifvoESXGvHJBeRxZrCqdGw+Fcrkhqu5RAJK7tg9hb4ab01ARjOcO26WJ4NnkTNupnSNwyGDGXnqSNq3aqBtdB8+BUyBk2qkgk7eqP6NOPTOCXPIa4HRKPVn/cRmRCClauXCmVEVLyQ2Gn5OwyNhKwIgLWIuxESu/fv4+aNWsiLCwMs9sXQ79q+awo0/+FWm0OvDs1kFbk1AG/ImLxkmxXodQ2ReHUdxVcy3rABmrg3lS8WL5BUbzUPr3gMXwcHJwA9cOfEbF0ZVYWFb6BV8+OsLUF1FE7ET3vaySnE8KKApIumBfxyWi15DbuhSXg888/x7Rp05QaalpcFHaKTzEDJAHrIGBNwk5k9NChQ2jZsiVs1Cps6VsW9Yq7W0eipSjtYNdsFTyalIcNVEg5MxKRO07nHH+1X+DdsWHqql3EeoTPUtKbJDvYtd4Aj/olYKOOQ9L2jog+F5KFhdq+NlzHzIOztx2gjkHS9s6IPvdc0XMmWaVGt9X3cPhBFPz8/KR6dbZC2Sr8obBTeIIZHglYCwFrE3Yir7/99htGjBgBH1d7HBhcAcW9Ha0k3R5w7LULrhVdYaOOReLm9oi5FJFj7Oo3foB3tzapq1WRmxA+83vFcFLbVYPb6IVwymsHpFxF7LyBSAhVZROfC+y77oT7W97SyqX65mSEr96uGA7ZBTJxjz8WngnBG2+8gVOnTsHDw0PR8b4MjsLOKtLMIElA+QSsUdiJrAphJwTeGwWcsGtgBXg42ik+2Wq7mnD9aD6cvYSY+Rex8/sj4Xl2YkagsIVtvYXwbFM9VdA8+gXhS1YohpG68IfwHDIE9mIhLnoHon+ahOTk7MKzhU393+HVumrq6+vIzYj++TskpygGRYZAVlwMwUc7/JE3b16cPXsWpUuXVmag2URFYWc1qWagJKBsAtYq7JKTk9GqVSvp1WzbCl748/1SsE2rc6HMnKvyD4bXiJGpYiZiPaJmT0NKSvY1PNQ2ReDU/y+4lnaBjToeSfv6IPr4fYWAsQVq/PLqEEngb4hctAgqVQ4npSt+C68eftLKJVIuIPaXoUgIz0kQWy6ik4+j0WnlXahgi3379qFp06aWG4wOnlPY6QCNXUiABORHwFqFnchEaGiodJjiwYMHGNegAL5sWkR+CTKgR+qqPyHPe01gY5PbK0U72FSdDs9OzWBrq4Y6ZB2iFk1HSrxSSsTYwa7NBnjUK5Fa1uXeVISvWJ9jKRN1kdHw/GCgJIihCkDCsi6IffhfiRgD5secph6HJ6L5klsIjU3G/Pnz8eGHH5rTHbOMTWFnFuwclATMS8DVwQZvFnGHq6MtklLUuB0ch5Boy/4Fb83CTswmUf6kdu3aiI6Oxu+dS6Lzm3nMO8mMNroD7Pw2w6NWYdggCckHeiDqyIMso6ltvGFf5wu4tWgOOwexsncEsSu/QGJwvM6eiRp60gnbooYsDK2G6txYRO69pINfznDothduld2l18yqf79ExLrdOdpR5+kLzzEfpwo7dRTi17VG3HXdeejgsFG7RCemoO2y2/g3OF4SdELYWeNDYWeNWWfMVk/gvXfyoWv1AmkcbgbF4Lsdll3XytqFnUimKFzcoUMHONnZ4O8B5VC1kKvi5rraphSch6xLFVeqKCQd+Q7xQf+9TrR1hI1LXtj5VIR9hcaw9/GQBEzK9WWI3b0CKZH6bSgTr3WdB22CawkHA3JVIeXUMET+fV5rm2qbPHDquxtuZYU/KqiufI6IDQdytuPWFe7jvoCDpEsTkbi5GWIuxmo9rhw7qNRq9F//ADtvRUivXvfu3Qt7e0MKcDlGnb1PFHa55CqPqz2K5cn5Tkax2hGbpEJwRDzis92wajmTQYmejmtZDFWKuMHezgZh0clYdCwQ1wJjlBiqVjF1rpYfXarlT+tzJzgWk7c/1MqG3BpT2KVmRFw5Jq4eK+zhgP2DK8DXw5AixPxZVzu3h/u4b+HolNvrVBVUD5YjdudyJD2LNJDjzrApURt2roY8oKKC+sVFpATlfKo3J+fFCqLLwC1wKfmfsLs4DhGbj+QYq9q5Ezw+m5Qm7JK2tUb0Oe3HNRBMg5r54XAgZh4LRqlSpaTDEj4+Pga1b0nGKOxek62i3k6Y2qW0RhuRxQUewZFJOH43HIdvheNFLFWeuX8QhCCf1qVMBjfOP4rCT/v8ze2a2censDN7CozqQN++faUK+zWKuGJbv7Jwlt69KeNRl54Ir37vw06ULgndj/jTF9L2lKnt3WHrVRr2ZetLq3XSFVo3FyNm2zKkKPDznCi+7Dxo438riCqoLo1HxKbDOQs7Jz94fP7tqxW7LS0RcyHK4ifG5n9fYPCmh3B3d8fp06el8ibW/FDYvSb7fm/5oGetglrPjxSVGmtOB2PXv2Fa95VDBwdbG3i62MHGxgbR8UkWuxJZv7QnRjTLeN3SmQeR+OVAgBwwm9UHCjuz4jf64PHx8WjUqJG0ctH9rbz4rWMJo49pmgFsYdNwKbxaVkktTHx+LCK3/pNlaLWtD+ybzYR7g7dha6uC6tGviF62LMeTs6bx3fCjSHv+Bm6Bm7Rip0bKtQmI/GtfzsIu/YqdOh6JG5sj5oplXz9x6Wks2i27g4QUNbZt2yYVIrb2h8LuNTMg8z4kbSfLwZsvsOT4U227mbW9vQ0wv08FuDmlfsJPTlHj8413ERRpeRvrxSvYCW0z/kHbdz0My04EmZWxHAansJNDFozrQ2BgIGrUqIGnT5/ify2KYHTdV3sqjTuyMa17wKHH33B7wy21dMl2P0Sfy/4DtNquAlyGrYCLrzg5EYXEre8j5sIzYzpnBtsecOy7D27lRGFqNdQ3JiF8zc6chZ37+/D4ZGLqip0qDAlr2iH2VqIZ/DbMkEFRSWix5BYCo5IwdepUaQsCH4DCTktht/ZMcFoPZwdbeLrY441CbvD1yr7i+4LDT3DsruXsYXC2B5YMyLiMPWnrfdx7bnknp+xtgTHNi6Girysc7WzwLCoJ8488wcMQy4vF0L+sKOwMTVSe9sSKXcOGDZGYmIA13UujVTkveTqqoVcZCxPfRtzCXq8OTmSx4Qj7TtvhXi2/dGI05doXiPxrj4YjWUozR9h33Qv3tzxTiw4/mImIZatzLnfiMwCeo8aknopNuYe433sg/ol+B0rMRSo+OQUdVtzFuSex6N27t7T1gE8qAQo7LYRdVHwyhq+8nW2PigVdMKpZUeRxy7hRWZSQ+HT9XSTmUDxTbhNRScJObmzl5A+FnZyyYVxf1qxZg169esHTyQ67B5ZDxfwuxh3QiNbVPgPhOWr0f7csbEP0T5NzuGVBOOEI+47b4V5dCDtR4+0HhK/YkKPo0cxtB6BAedg6GvK+UTUQfR+qcF1Op9rBtuU6eDYsnRpj8B+I+G1ujgWKVcU/hvfAvrATwi7pOKJnjkGShb6J/XDrI6y7EibVbzx69CicnZ01S6EVtKKwM5CwE2aKejvia79ScHfOuFF59n5/nH1oGRtUKeys4KceAIWddeT5ZZRffPGF9KqqVF5H7BtUEXldLPQwxdsz4d25mVSYGPemvb4Yr40PnPruhGtZR4Ot2Mmt3InIr7rqz8jzXmOIy0bU8XsQ8+NEJOWwc0b95jR4d2sFW9H22VJEzp+T8y0VMv4RmXcyGJP2B6JQoUI4d+4cChcuLGNvTe8ahZ0BhZ0w1a5KXvSu7ZvB6vpzz7DlUojW2XVxsIWTvTjAkILk7G/LydGml7MdYhK072dMYadLPOIVqiiim6ICohLM+8pAvNp1d7JDeJzh/dA1X7lNKg8nO8QlZp0HFHa5kVPW98Wp/U6dOkmbyxuVcsf6XuXgYMhFJ5PgcoBd+y3wqF0otTDxsQGI2ncjx5HVLm3gMXYKHFyFiklC8v7uiDqmX0kfOQo7FBgCjw8//O/16k3ELeiL+ODsfkfZwbbpSng2rfDfwZMxiNx6wiSZM+Qg++5GoMfa+3B0dMKxY8ekFTs+GQlQ2BlY2JUr4ILJHUplsCpKoPx2ODDLSH3qFERZ8VrEBnjyIgGLj6UetGhRKQ/av+WDAh6p+/bEKVtRZ+zXQ08QlkMZlZolPVC+oCtK+jijhI+zdPhBHHzwfxGPR6HxeBwaj2N3whGblFUhlsnnDL+380lL+ba2NqhewiODr08jEhAQlpDhayFRSViZbr+h+KYh4hG+1yrpIe1bLJrXCa7pLjSPT1JJnG4/i8X2SyGIiM9dYPWsVQDlC7hKdyPGJKjx58mneBqZdbNwTr6LOoYiH/XKeCGfh4NU+iY2MUXap3c/JA6HboYjKBt7uf2i0Sdfr7Nd2MsRNUt6okwBJ5TJ7wZvV3skJatw51kcbgXF4sS9CARGJHLFLrcEKfD74kaKOnXq4N9//8UHNfJhRttiFhVlxsLEL5Cwzg+xN3J6j2gH28Z/wLNZldSVrMQLiPttKBJC9b8XVW1jB2nJy2CPGlClQCxC6vKo7arAZcQSuOS3B9QJSNrVGdGnsh7aU9vkh1PfLXAt6wIb1TMkrO6E2NuWtd/4VkgCWv9xC5EJKVi9ejV69uypCzLF96GwM7CwE6VClg2qlMGqEFQLjmQVdvN7lYeXa2plbPGJesAfNzCkcWE0KOudrVdz9vvjdKZXumJFZljjQninuBLa704AACAASURBVGeuk/VZZCJ+ORiQ5fBAz5oF4fe29sUcBy69kWHvoD7xlM7njL51fSVxqskjxNWm889zLSmT3idhd97BAJy8n7VYaWbf+yy5gcblvdCnjm8GcZnZN7EqKmxeeaJZkSxD5CsnPtWLe2BE0yIQh3pyeoQ4Xnw0EIXzOLFAsSYTTWFtxF2yYoVD3C37U/tiGFAtn+VE6NYJbh9/DUdHG6iTzyJ29nAkRmanhmyBUqPh0asf7EURY3UCkk+MRtSec5YTq1ae2sG2yXJ4Nn0j9e7c+7MQsTzrAQq1V1d4jJoIB1FvP3gpIhf8alHlX8LjktH8j1t4EJaIiRMn4ocfftCKkjU1prAzsLBzc7TFon4VM1jdfz0MS7MpsZFZTOy/8QIt38ibo0dTdj7E9aevNti+UcgVI5sWgber5pXlxSre0n+e4vDt8LRxxKqW31va/4LPTdhpGo+Pmz1+6lYW9qLiqJbP3AMBOPUg56ryugq7nVfCNBa7KpUas/cH4Pzj1++jNFS+skMkai52r1lAo2La2fXnzRNaTjwLbn7kyBG0aNFCWiXa1LcMGhTPuEIvz9BsYfPOLHh2bCytvqtDViDq11+yCBO1U3E41BwGl8atYe8kGiYi5fo0RG3cArWCa8arXZvDbcQ0OHnaAaqnSFjXH7E30m//8YCd3wp41CwBqAKRuK43Ym9aTrWGJBXQbc1dHLkfJV2Zt2XLFqnOKp/sCVDYGVjYidOxX7+b8VXsH8cDceDmKyH1csjMoiO3STp7XwDOPkoVMWLlZ1a3smn15l72DY1Kwo3gGDwNT0ReNwdU8HVB0TwZTwuJV3Of/HU37bVutWLuGNmsqPTKQjxOYjNZuke0z/wCQ7yK/HZHxv0qusbzVhE3fJ6p3lzCf68Pn0cmSpfU+7g7QAgjl3SvZoWLuZ061kXYZZcHsU/taUQifNwc0lZZ07d7EBKHr7ZkvYj8ZRtD5iuzfxULuuLrd0tmO31CY5IRHpuEonmcsuQ1fQcKu9x++pT1/YULF2L48OHwcbXHvkHlUfI11ybKIXK1Z2O49psBpwIOqWU94u4j2T8AUAm1Zgu1nRtsPQrDNl8h2NjbSocl1HG3kXjkR8SdfHUzhRxiMY4PtkD5z+DR/X3YO9hAHf8vEnbNRPy/16F2KA3HeuPhUr8abBGN5GMfI/qgZTH5fPdjLD4bijfffBOnTp2SbpjgkzMBCjsDC7tPWxVH1eIZJ93krQ9w53nWvSCvE0KikO7WSyFISlGhchF3FM/rjD3XQtP2lYk9YW0rZ3x9euBGGFacDEJyOhUmxFrXavnRoWq+DKs5ORVP1ufwhK7xVPJ1xVd+qcIkKCIBO66E4tT9SMSJj2npHvFHaHiTInijsFuGr885EIDTOaza6SvsxGqcyMPOq6Fp/ohXniObFckilDKvqKZ30lj5EmOMb1U0y6t4sZ9uyfFAPAlP3U8oDmd0rJoPrTPNmZc+UthZ35+JUaNG4ddff0WlAs7YPbA8PDJ9aJILEXWpcfDq0wu2DjapJT0yPWq1ClAlAolRUEUEIuX5baQ8OIqEf09AHafjxjW5BK+VH7ZA6YFw8+sPBx/31L2FKhVgY5t6ijjmOhIOTkXcuet6lnzRyim9Gy+/EIKPd/pLd7+KuoziLlg+rydAYWdAYdeonBeGNS6SwaLY0zRy1c1sr+XKSQiJ68h2XA3N0bMCHg74sat4dfnq15wQfStOvSqenLnzwHq+aJHuNa84kPHJujsIicn4fsIYwi63eMS+RLHH71lEovRa9XUl//K7O2Dm+xljF0Wjt1/Jnpc+wk6s0s09GIDLAVn3z9Uu6YExLTJuPt91JTTLgRKRB2PmS9yHO7Vz6QyvJfzD4vHdjoeIScy6UVxcsza0cZEMc0f4SGFnfX8qkpOT0bp1axw8eBBtyntjZbeSOr/Ktz568o1YbeMFuzL1YV+0DGxdHIHEF1AFXUTSnctQZ/M7Qb6RAP88isZ7K+8CtnbYv38/GjduLGd3ZeMbhZ0BhJ1YDelZuyAalst66EGsmhzM5jWsGDY7YXf41ou007E5uda/ri9avflqL57YwP/JX3cQnZDziS+x2vVT94z72H7Z748zmQ5jGFrYaRKPtj8Nn7cpjreKvloVfd3VbboKOyF8Z+x+jGuB2R+KEC+rZ3Uvm3ZyWcTwz70IzD/0JEs4xszXgHq+WfZlTt72QDoFm9NTp5SntOIoTvi+fCjstJ2FymgfFhaGWrVq4d69e/i4fkF83Yz1wJSRWcuP4lF4onRdWGhsMhYsWIBhw4ZZflAmioDCTgthl5yiSjtRKf4m5nV1QGFvxxwPL1x6HI0f9z7OcYTMokOcWp246W62q3vpjUxsVwKV072OzOlwRuaBv+1YCmXSVZ3PbiXNkMJO03i0neuDGxRCs4p50rpdfByJmXsDsjWjq7DLbZVRDDakYSE0qfDKj+uBMZjy96MsfhgzX5ltB0cmSvsnc3syC0IKu9yIKff7N27cQO3atREVFYVF75VA18o5H+BSLgVGJicCUYkpaLP0Nm48i8fIkSMxb948Obkne18o7LQQdtpkU9SNEys+L+JyPoqVWXSIMhTpT6vmNN7cHuWR1z21TIp4/jr7DGLVKrdHrNJUKfJqpevAjRf445+M9Y4MKew0jSc3vzN/v1uNAtJ+sZfPZf8YzNiTVVCJ7+sq7HIqi5Lel/er50end/KnfSkwPAGfbriXJRxj5kucJi7o+eqe4u2XQ7H2bM6v5F86xwLF2s46Zbf/+++/4efnByc7G+zsXx7vFLbca8eUnSnlR6dSq9F3/QPsuhWBZs2aYc+ePbC3f/X3TvkE9I+Qws7Awk7cErHx/DPsu/ki1w2qmoqO9C6Kmxj+GFDRIEe9s1vpMqSw00QcZYc/n7sDqhd3R35PR+RxsZeK7Ip6f46izgEAH4+M5V3MJexaVcqD/vULpYUgVig/zrRaZux8LR9YKcN+ud+PBeLQrawnsDNzprDT/5en0iz8+OOP+Oyzz1DIwwH7B1VEIU/+MVVaji0hnu8PPcFPx5+hTJkyOHPmDPLm5QqytnmjsNNT2IXHJiMoMgFBEYkIDE/EkdsvXrvXLf1wugg7sVl+Wpcy2uY52/ZyE3ZVirihS/X8KFdAsyLF5l6xa1rBGx80fLUnKTthZ8x8CdE7r3f5DLn9aa9/rvX0RAcKO4P8CCnOSP/+/bFixQpUL+yKbf3LwsXeQu+UVVxmrCOgDdfCMXTzA3h4eOD06dOoVCljsX/roKB/lBR2Wgg7saF+zJrbaT3ik1Jy3Q/3uhTpIuyyu7JMl2kgbroQr2EzH+ww14qduGJrTLOi0pVm2j7mWrHTRNgZM19FvR0xvWvZDLhm7H6U7SlerthpO6uss31CQgKaNGki1Qp7v0oeLOyUfX1E66TDqI1J4GJgHNovv42EFDV27NiBdu3aGXM4RdumsNNC2EXFJ2P4ylfCTt+ZoYuwE6db5/TKuEqz7mwwrml4pdVLnyPikiGK12Z+zCHsRIHica2KZynBIV5rn3sUKfkZEZuEhP/q2tUq7ZXhPls5Cztj5kucxp7fp0KGFP52KADH7+V8E8fLxlyx0/enV7n9g4KCUL16dQQGBuKb5oUwtp6vcoNlZLIg8DQyGS3+uImnUUmYPn26tCWAj+4EKOwsTNgJd5cPrJjh+q0VJ55iz/XcD09oMk3MIewmtC2e4VCH8FOcSt17PSzDXbQv/c98aEHOws7Y+cq8x27V6SD8fTUs11RT2OWKyKobnD9/Hg0aNEBCQjxWdSuNNuW9rJoHgzcegfjkFLy7/C7OB8aib9++0lYAPvoRoLCzQGH3Y9cyKOwtbnJOfXIrTqzNFMlO2OVWF+2lfV1WIEWNZXG3bvqL63Mq9Gupws6Y+ZrdvSzye7w6FXvZPwoz9vjnmvLMJ4tZ7iRXZFbXYO3atejZsyfcHW2xZ1AFVMqf8WpCqwPCgI1CYPiWh/jr6gup5I64x9jJ6dXfNqMMaAVGKewsUNhlvkJKFCj+aO1txCbpf32OOMW5dGDGDauz9jzGBf/oXH8cdBF2pXyc8f17pTPY/n7HQ9wIis1xPEtbsTNmvib5lUQF31eHTZJT1Phw5c3XzoXieZ3wtV9JuKa7QorCLtfpbZUNvvrqK0yZMkW6S3b/4ArI68LDFFY5EYwU9JwTzzD5wBMULlwYYpXY15ev/Q2BmsLOAoXdu2/5oEetghk8X3/uGbZcCjHEnMCS/hlX0FadCsLf13J/vaeLsCv3/wWTJ3fMePff6+5cLZnPGeI+XlEC5eUj91exxsyX31s+6JlpLmy88BybLjzPdi4U8nSU7uVNz080pLAzyI+O4oyIQ1adO3fGli1b0KCEBzb2KQuH1KpDfEhALwJ77kSg17r7cHJyxvHjx6V9nXwMQ4DCzgKFnVhVm9mtHHzcXokbUdRx68UQbLz4PMf6eeL38ZtF3FCjhAeCoxJz3IuV+dXhw9B4TN76AEmq168I6iLsxKvfxf0rZbje6sKjKMzal/V1YvXiHhjRtEiG17YifXIXdsbMl4eTHeb1Kpdhz6VgMv9gAP65n/EQRbVi7hjepAjcnLKuulDYGeYXqhKtREdHo27durh27RoG1ciHmW0z3pGsxJgZk3EJ3Hgej7ZLbyMyIQVr1qxBjx49jDuglVmnsLNAYSdcbljWS/ojnfl5EBKHKwExCHgRj9CoJDg52CKPmz3KF3TF20XdkccttbhvUEQCxq3PekuC+J5YEata/NUNFdKKzrNYnH8YJYmqGiU94e5kh9GrbyP97bS6CDthO7OQFF8T17HtuBKCqPgUlC3ggsrF3FC7hGe25VDkLuyMna8PmxRGg7IZ7ykWQv/e8zhcC4iBeFVfrYQH3kh3DV3meUNhZ2W/+bUM9+HDh6hZsyZCQkLwY/uiGFzt1Y0rWppicysnEBaXIt0B+/BFAr788kt8//33Vk7E8OFT2FmosBNuf9muxGv/WL9uuoiCyuPWZ3+nqFjZGde6+GtnW2xiCj5ceQvJ6ZSdrsJOXEo/unlRnWe3JQg7Y+bLV7xebV8iTbRrAlIwe7uYW1pTCjtNqFl3m6NHj6J58+ZQq1KwqU8ZNCzhYd1AGL3WBETFqq6r7uDYw2h06tQJmzZtMsgtSlo7ovAOFHYWLOzEidL2b/mgS7X8WV7F5TZvj94Ox8KjgTk2y7wpP7uG/ZdcR3K6t7O6Cjthe2A9X7R4I/erYyLjkvH7saf4pNWr10GWIuyMmS9xDZsoG1PI6/UnylQqNdacDcY/dyIy1MCjsMvtJ4bfFwQWL16MoUOHIq+LPfYPLi8dquBDApoS+HS3P5acDUHlypVx8uRJuLtnfDOkqR22ez0BCrvX8Ml8s4B/WDwmbLpvsDk1vUtpFPRwREKyGm5Otpi++zGualloWDhT2MsRvWoXQKl8rlk2xad3NiI2GVcDo3HyXiQu5XLK1cHWBl1r5Ee7yj5ZXn+KDdXH7kRkEYb6xiNWCrvXKoCiebKWVRC+/xsUg9UngxEZn4yFfSvA5b9TnSfuR+DXg0+yzYumPmnaLv0gYs+fEJgvYpKk1bJ7z+IwaduDXOeHMfIlBhUFiwfULyQVb7bL5gaP28Gx2Hj+Oa4FxkDsbfy1d0XEJaZIvmtaJiXX4MzYYP2YOnCwSUFMTIwZvVD+0GPGjMHcuXNRMb8zdg8sD89s9mwqnwIj1JbA0vOhGPf3Y+TLlw9nz55FyZK81URbhpq2p7DLhZSbo61UFiJZpYYQF+n3lGkKOad29jaAh4s9EpNVEGLJEOVKhL+ixl0hL0c42NlA3N4QlZCC4IhEhMZmvWkitxjExv8i3k6SeBTXfQlR9Sg0HuFxKVm6GioecVtDQS9HFPBwRHKKCrefxeFZVFKG8YQw8XB2gLjmTdzXm1NeNPVJ03aZg3Z3soVYBXNxsENYXHKOB1dy4mzofIlxxIEKcXo4j6s9HOxsJT5PwuMRFJmRocito70NHGxtER6vve+5zR1Tf5/CzjTEU1JS0KZNG+zfvx+ty3liZffSsLPR/ipA03jLUeRA4PjjKHT+8x5ga4cDBw6gUaNGcnBLsT5Q2Ck2tQyMBKyLAIWd6fL94sUL1KpVC3fv3sXYegXxTfPCphucI1kUAXFIosWS29IH30WLFmHIkCEW5b8lOkthZ4lZo88kQAJZCFDYmXZS3Lx5U7otIDIyEgs6lUS3KnlM6wBHkz0BUc6kzdLbuPk8HqNHj8acOXNk77MSHKSwU0IWGQMJkAAo7Ew/CXbt2oX27dtL2z529CuHGkVe3YJiem84opwIpPx/yaW+f93H7tuRaNGiBXbv3g07O95cYoocUdiZgjLHIAESMDoBCjujI852gFmzZmH8+PHwdXfA/g8qorDHq8Lp5vGIo8qBwLcHAzH7n2CULVsWZ86cQZ48XNE1VV4o7ExFmuOQAAkYlQCFnVHxvtb4gAEDsHz5crxTyAU7BpSDiz1XZsyXDfOPvP5qGIZteQRPT0+cPn0aFStWNL9TVuQBhZ0VJZuhkoCSCVDYmS+7CQkJaNq0qVSbrEvlvFj8XgnzOcORzUrgYmAc2i6/jaQUNXbu3Im2bdua1R9rHJzCzhqzzphJQIEEKOzMm9Tg4GDUqFEDAQEB+LqZLz6uX8i8DnF0kxN4GpmM5ktuIig6CTNnzsS4ceNM7gMHBCjsOAtIgAQUQYDCzvxpvHjxIurXr4/4+Dis7FYGbct7mt8pemASAnHJKXh3+V1cCIxF//79sWzZMpOMy0GyEqCw46wgARJQBAEKO3mk8a+//kL37t3h7miH3QPL4o0CPCkrj8wY14uhmx9iw7UXqFu3Lg4dOgQnJ143Z1ziOVunsDMXeY5LAiRgUAIUdgbFqZexSZMm4bvvvkNxb0ccGFwePq4OetljZ3kTEKdfxSnYIkWK4Ny5c/D19ZW3wwr3jsJO4QlmeCRgLQQo7OSTaXFFYteuXbFp0ybUL+GOjb3LwNHOVj4O0hODEdh9OwK9/7oPZ2cXHD9+HNWqVTOYbRrSjQCFnW7c2IsESEBmBCjs5JWQmJgY1KtXD1euXMGAaj74qX1xeTlIb/QmcONZPFovvY3oxBSsW7cO3bp109smDehPgMJOf4a0QAIkIAMCFHYySEImFx49eiSdlA0JCcGMdsXwQfV88nOSHulEIDQ2SboD9lF4Ir7++mt8++23OtlhJ8MToLAzPFNaJAESMAMBCjszQNdgSPF6rlmzZlClJGND77JoXNJdg15sImcCSSqgy8q7OP4oCp07d8aGDRtgY2MjZ5etyjcKO6tKN4MlAeUSoLCTb26XLFmCDz74QHKwYn5n+TpKzzQiEBiZhMiEFLz11ls4ceIE3NzcNOrHRqYhQGFnGs4chQRIwMgEKOyMDFhP819++SWmTp0KcbCCj+UTKFWqlFTWpEQJ3jIit2xS2MktI/SHBEhAJwIUdjphM2mn6OhoJCUlmXRMDmYcAuIeWDs73glsHLr6WaWw048fe5MACciEAIWdTBJBN0iABMxKgMLOrPg5OAmQgKEIUNgZiiTtkAAJWDIBCjtLzh59JwESSCNAYcfJQAIkQAIAhR1nAQmQgCIIUNgpIo0MggRIQE8CFHZ6AmR3yyDwZiFXlMznAnGrUXRCCk7cDUd8smX4Ti81I0BhpxkntiIBElA2AQo7ZeeX0f1H4Nde5eHtap/GY+6BAJx6EEk+CiJAYaegZDIUEiABnQlQ2OmMTvuOFQu6oJC3E+xsbZCUosK/gbEIidb86L+4Qvud4h7wdLGDrY0NYhNTcPFxpNYrT5ULuyG/h0OajbMPI5Gs0j4ebXqMa1kMVYq4wd7OBmHRyVh0LBDXAmO0MaFX2/m9ysMrnbCbdzAAJ+9T2OkFVWadKexklhC6QwIkYBYCFHYmxL6wTwW4O7+q+3PZPwoz9vhr7EHtUp4Y07xohvY7LodizdlgjW0U8nTEzG5lM7SfvusRrjwxnsgqlscJ07qUyTDm+UdR+Gmf5rFrHGAODSns9CUo//4UdvLPET0kARIwPgEKO+MzThth6nulUdzn1XU6CckqDF1xU+PVsg8aFELTinkyeOwfFo8Jm+5rHEWrSnnQv36hDO2/2Hwfj0LjNbahbcP6pT0xollGQXrmQSR+ORCgrSmd25tC2DnY2kirqeLOxOj4JK1XUnUOjh0lAhR2nAgkQAIkwFOxJp0DfWsXRJsqPhnGnLLzIa4/jdXIj1nvl4Wvl2OWtqNW3caLOM1OAnzUoihqlvRMsxEVn4wRK2/DmG9ixSvYCW0zXjuz73oYlp0I0ihuQzQytrCztwHm96kAN6fUFdnkFDU+33gXQZGav2o3RJzWbIPCzpqzz9hJgAReEuCKnQnnQnYCZ+ulEPx17lmuXvi42WNOz/LZtltw+AmO3Y3I1YaNDSBeB78UH6LD/7F33vFNV9//f70zmnTvUrpLoWUJMkWGypa9RYaCiIrihK8bcHzAgZPhAhEQGbKRjey9Ze/u3dKdpk2a5P373XQmTdukI03T8348Pn986B3nPs/Fvrj3nnNOR2Thp8PxVfatSQORAHizrz9aetvBRsghJacAPx+LR9TDujsl1Le3roWdVASsmNpaZ9p5OyIQnmq+NdbER9bQl4SdNXiR1kAEiEBNCZCwqylBE/qzq7pfnwuDVMzCIAq/8NQ8zNsRWeUovZo7Y8ZTvgbbnQrPws9HqhZnIR5SfD6ymc4YxorCKg208AYk7CzcQbVgHgm7WoBIQxABItDgCZCwM7MLWXRox0DHklk1PI8Za+4iV1n5ZegrT/jgiVAXg9Zm5anw2tp7Va5keHt3jO/SRGdudo2bla+usm9Db0DCrqF7sGr7SdhVzYhaEAEiYP0ESNiZ2cf9WrniBb3ghUUHY3E+KqdSSxaPbwF3R3GFbYwJgPhgUAAe8XUoGcPY08KykzpLhchVqKHizQwOAHvHJhYJkFdg+otAY4QdO1G1lwiQmWe60K3tq1h2bc5Y5xeoKQjDyK1Gws5IUNSMCBABqyZAws7M7vV0EOPHZ1vozHrodgb+OJVYoSWG+ug33nA+GTuvpVU4Bnvbtuy5MK0wKv62XE7F1suplRLoEuSI0CZ2CHKXItBdqn2fxwIDYjPytZG0MWn5OHE/E/KCypXehK5eCPWyg0AA5Cp4rDmTiMRsZaVzMyHXt5UrWvvYw8dFgiZONtocgOyUMz1Xhbh0hTYfoP7HhN9fZ5N0TkErEnYutkL0a+WG7s2d4eUo1ka0svyA7P1fxMM8HLmTiaQK7GRX20Pbe4ADIBBw6FTmJJbZlJil0NpY9nuYU4C/zhtOT9PezwG9w1zQ1MUG3k42ELEyGYDWngcpeWCRxCfvZ6FAUw+q2sx/T6ozHQm76lCjPkSACFgbARJ29eDRhWNC4OsqKZk5JVuJdzY+qNCSp0Jd8NITPiU/vxIjg71UgBZediV/dishFwv2RFc4Biup9dGQIJ2ff7IjEg9S8wz2cZQI8cqTTdEhoDSCtqLBmf2LDsdVGgxhzIlZ2fFZEuUp3b21gq463/cHYnEppvQU1ND8IgGH5x731gkm0Z+LnU6yZMaG8vxN6NIEQ9vrRjkbY+sLK29DqS4VZ+52IrzYqyna+5de0Vc0TkKmAh9tjSBxZwAQCTtjdh+1IQJEwNoJkLCrBw9PeqwJBuulPXl7w32kVlCF4rXevugR4lxi6bpzyZCIBRjT0bPkz9gp2itrbld4bTe+cxMMf7RUhFSW5qR1UzvM7O0LF7uKr371sbH5V55KxNF7mQaJmiLsQjyl+GRYsPZ0rrrfjwdjcaHM9bb+/ElZSoOpYwzNp9Hw+PFgnI5QZO3YKeTQdh4mm1hW2Ak5YO7QILRoUirSqxrw/c3hiMvUPQmsqk9j+DkJu8bgZVojESACVREgYVcVoTr4OTuN+nCwbl63FScTcPiOYVG0dEILuNqXiqw52yK0pbk+HR6sY903+2NwJVZm0OLPhwcjxMu25GcVRdKyk7rvnmle7hQrLacAt5NzkZiphJu9GGHetvBzLU22zAYuUGkwa+MDpMvL59QzVtixK+MvRjdDU2fdk7o7ibm4mSjXzsGuhbs2c9KWRCv7ZcoLtCdhigINvtkXg7QydujPrw8pT6lGYpYS7vZindJjxe0iH+Zhznbd6OWO/g6Y2ccPxWZIylxzF/PQvyhmV7yf74oqmX5oO3dM6Foa0MJ+kJKjxKn7WcjOU8HJVoQwbzu0amqnvSZm34dbwxGjd8VbB9u0wQ1Jwq7BuYwMJgJEoA4IkLCrA6hVDcl+///2XEudtCdno7Kx5GD5Sgz6JcDY1SCLouUN5KTbfyMNf54t/37L3kaAXyeHad+BFX8/HYnD6fDytVInd2uCQW11rxcP3U7Hn2eSdCpkMI0xtqMnhj/qoSOwDt/JwIqT5d8LGivsDAWXrD2bhD030nWwtvCy1SY9Lps6prI3gxUJO3Yax3IJ7r6eVhKU0SnAEeyUtOzYbPLKkklXN3hizpBAtGpqX7K26/EyfLsvplxwCntzN6qjB4LcbfHZPxFVvmmsag9a489J2FmjV2lNRIAImEqAhJ2pxGqp/az+/jqP7WX5arz6191yFSD6tnTBtJ6l7+suRWfh+38Lc9a90c8P3cpUkWDvr97dHF7OQiZUZg3wL/lzFnzAqk3kKHSjP1nwwDdjm2tPA4u/isRi8c9f6O6Nfq3dStqrNTxm/X0fD3N1T+2MFXb6ZdOuxcnw9b4Yg9SHPuKOCY+VnnaxE7uZ6+4bbGtI2LFTuiWH43A1rnyd3MeCHPFmv1JmbNDd1x5i3XnDyaSrK+x+fz4Mtjal9YNNqURSS1vRaoYhYWc1rqSFEAEiUAMCJOxqAK8mXfu2dMW0nro1W+duj0CEXjWG1/v64vHg0vd1a84kYd/NwtMrFkE5vVep6GN/9ub6rUdYkwAAIABJREFUe0jTE1VTHvfGgDal4ut+ihyf/lN6HVi8Dv127HRw1sb7kCkqTi/CHv5/P56JwdJoW0PpW4wVdv8bEYxmnqVXxixw4UxE+ZNFZrO3kxjfPaMbYfzS6tsGT7P052cCdOG+GNxIKC/q2NhsNd+Nbw4vx9ISbpUlgq6OsGNzrJrWSuct4S9H4nDSwElqTfZaY+lLwq6xeNry1qnJT0NMvBzOgf5wFRmyT4nM2CgkZGtg6+4Hf28HGGxmeUsjixogARJ29eQ0DwcxFumlPfn7QjL+uaqbsuTnSaFwti39T8AHW8IRm1H4cN5QmbHlxxPKBTDoR+FuvpSCbf89LLdy9u6Pvf8r/g7eSsdKI+q5fj4iGCFlxNj6c8nYdV1vHRNDdd6uVSTYfp0cCkemkoq++buicDvJcC1dQ2KqoiAUfWFnyEZ9IC/1aoqnwlxL/riyyOPqCDs28IJRzbRvBos/FmH89b5oqjFbjb+XJOyqAY261IiAJvUUlr4/G1+tPY9EJWDX4hks2fkXpoUV/TdMlYCjS+fhs6WbcCIiG9pgePFj+OLaKXzYsvSkvkZGUGcioEeAhF09bgl9wcVOj74sk7LEz0WCr8eGlFjIHtO/qldhQn+MM5FZWHqotLyYq50ISyfq1pg1dDLIJlnybCjcHEpF1cYLKWBv5qr6Zvbx1Ul8bCgvn7Endl+OaoaAMkKnshOsYHcp5o8qLZHGAiteWH0HvIE0b8bOX3at4zp5YmSH0sjjiq66WZ/qCrsXezZFn5al4pGNxU5K995Iw4Gb6VVWJKnKN43p5yTsGpO363+t6qjNmDF0KlbczEXpf3IE8JqyFRGrRsA29Qj+9+xkLDiSgLJpPoWBr2H/vZ/Qt/QyoP4XQxZYFQESdvXozsldm2BQu9JABZVag5f+vFuS42xga1c83730uvZcRDYWH9YNsNAPdtB/q6dfY7ai8mMsGvWPqS1LIi9rguW/mGx8e0DXTmOF1Ru9fdGtTGqXe0lyfFYmirSsXfppYAxFrha3N3b+suMPaOWKKWWqhFSWb7C6wo4J769Gh8BBWv5f7wqVBszn7OQ0XO+Kvib+sda+JOys1bOWty5N8m680e8ZLM/oiQ+WfI1JzhswcdBCXFbysOmxEDe3dsLvQ0ZhaXY/zPlmHp5pfh9fjZmI3++oYD9qNaK3PofSxzGWtz6yqGETIGFXj/4zlPbk673RJclw3+7nhy5lgiP+OJmIQ3onaO397PHe07qpU+btiEB4ar52Zfo1ZlmViF+PJZRbtb+rBF+NKT0drAmWmgg7Q+8GmbBZfz6pJEcfi8gd1cETozt46AjRiiJy2VqqI+z0bakLYcdsY/vg9T6+OlfQ+vxZupe155LLvcGsiZ+srS8JO2vzqKWuR4OYP8eh+we5eHffNrzVzhZQbMPzvmOxJo2Hw9Nz8BF+w/yoEVh78FeM9GWvaTWIWP0qXl0dicApv+KXKc1AF7GW6t+GbxcJu3r0oaG0J8WRl+w/Bb9MDtM5yXl30wMkZOmW4TJUKmzTxRRsv1L4hm7xhFDtW7zir6K3bSx9iH5evOqg4XleWx5NPyefscKKibZFz+razOxg19BRaflg4we4SXXy+rGfs4TLH26JQEZe+Rx6li7smH3s5G7yY03QNdhJJy1NWR+wgI8NF5Kx57pu6pfq+Mka+5Cws0avWuqa5EhJVsGziZO2pCDSf8cQ35exJ5+Dg7sr1HgUC47uxjttq1c5x1JXTXY1DAIk7OrZT/ppT5h4+XhbBALcJPhydOkJWkZuAV5fbziVx/tPB6Cdn0PJSu4k5eJ/u6Lh42yDb8Y1L/lzlrNtxl93Db7bYtGti/Xe4rFgjhvxhqNGK8LGrnr1o3JNEVa+LjZYMLKZTk1bY1z0479xuBBtOHrWlPnLzmWuE7uyc7KUMwNbu6FHC2eDJ3hM2LJE1IZStBjDyZrbkLCzZu9a9tqUJ95Gmz6L8ID9u1LgjkE/ncPOGSF0KmfZbrNa60jY1bNr9fPUsV/cM9fe0xaln9zNu8S6ylJtDH7EDZMeK23LTnZeWXMX7H1d2Tdi95Ll+Gxn+TQnxZOsfqGlTtqSP08nYv+tqoMnjEFo7IndJ8OCEGpCeS32ru6vM0m4k2y45m2xbcbOX9/Crnh+sYDDE6EuGNHBU+fElf08LiMf72+JMAZ7o2pDwq5RuduCFqtB5HdPoeW7J6DkOUg6fYJzZz9Be8pnYkE+alymkLCrZ3972IuwaIJu1CqrCtE9xAkdApxKrDOUxqT4h4bex31/IBa9WjijS3DpGGWvaA0t+5uxIfBxKb06qCo5sSnojBFWrELGb8+Flbybi89QgL057BXqArZGLycb7TUle+vGIlTZ6ealqJxySZ0N2WXM/Pr9anpi9+k/kbifUrngrIohK/H27tMBOulkWIJplq8v3/Ctc1VDWu3PSdhZrWstfGFZWD8uCJM2Z4LnnPD0L7ew5xXfwita+ohAPRAgYVcP0PWn/HpMCPxcSwXVyQeZ6BjgCLsyFQkqys9WPJZ+PVmWcqRbMyedmq+sxmxkWmFQhaHv/wb46YhJlnbj7Q33aqV8lTHCSj8QhJXX+mqv4aoTprrNmPlrIuzYW8eVL7TSGeK7/TG4XEHtXlPsb93UDh8PCSrpwk5131p/X6cWrinjWWtbEnbW6lkLX5fyGN5s1RdLItSAbT8svb8fM7UBE/QRgfohQMKufrjrzKqf9kTfpNQcJd7++0Gllr7cywdPhrlU2CZTrsLMdfcqHWNYO3c8q1eQvqpTPmPxGSOsHgt2wpt9/XSG3HA+GReicpCUrRs0Yuy8xe2Mmb8mwo71XTFFt/6voRq3ptrN2uu/t2R/9ua6eyTs9GCSsKvO7qI+NSWguf81erb5AGcKAHGnz3D57Dy0pWvYmmKl/jUgQMKuBvBqq2ubpnb4qMyJjP64x+5mYtmJ8ilKyrZjp3Nv9NEVRWV/fvxeJn47XvkY7NTp22da6LzpYtd+O/57iC3/pRpM/MvmYP82beNrj86BjkjOURqM2jRGWDlLhfhpUqjBXHr5BRoo2P9Uhf/LV2kgV6iRlKXU/u9KnAwpOQUVusSY+Wsq7PSvstlV8ac7IlGgMZAxGUDPECe88qQvzkVnY/OFlAqrTcx40ge9WpSKdmNEem3tzYY0Dgm7huQt67E146/RCHx+G3J4IXyn70LE8qdBuYetx78NcSUk7CzAa4bSnpQ169ej8TjxIKtSS9lbrJ8nh0LA8oUY+BYfisO5yIqjRou76Cc0Lv5zFqRwLS5X+3A/LacAErEArvYibaBDez+HkvQjSVkKzN4UXs4CY4XV1O7e6N/a9NSdLOL3dEQW/j6fgnR5+cdnxs5f1nBT3tixfu8OCMCjAaXRyezPWF1e9g5QKhagc5ATHCRCvLHunvZd4KgOHhjbyUs7JRPQV2NztKeTyVlKKFS89k1hn1auOmXeWNsLkdn48ZBuAmgL2Mb1bgIJu3p3QSM0QIFDr4ViwC8x0HC2GPRbFHa/5EXv6xrhTrCkJZOwsxBvzOrvi06BzgatMfba7fPhwQj5//no9D8WJftqBWlODE348eBAtC5TM9YUROz0bPam8tfGxgorFg3663NhWiFUnS82PR+f74ws9y7Q2PlrIuw6+jtg9sCASs2WK9VaX6g0wNB27pigd/Vd1ZpZvr452yPxUFbx6WRVY1jrz0nYWatnLXhd6luY36095l5UAeL2mHvuIj7vQPewFuyxRmEaCTsLcbN+2pNisyo6ATNk9tiOnhjVsbS2aXGb4rx2xi5VyAFD2rljTEdPnfQnxvSv6MrXWGGlH8DB5mT1b5VKHmIxBxsBBxuRAE2dbeDpaPjC4+jdDCw/kahjrrHz10TYsb7zhgYhzNuuUlRTVtyCigdCPKT4ZHgwhALj4ufYdfT3B2JwM1FujCsaXRsSdo3O5fW/4LSVGB74Inbm8hB4TcHWyFUYUflf//q3mSywegIk7CzExex92f9GhWjft7FkxK72Ym2VhY0XU/DP1TSjrGQJiT8ZFqytVlE8hkrN49dj8TgTUfU1rP4kbLyJj3kh2MMOLnYV/ys0S67C9QQZzoRn40oFUaBfj2mGJo422itGe4kAX++LwXW95Mf6tVmZPStOJpSrYlFsJ0vm+4ivAyZ09YJtmQhidlX85t+6yZyNmV9//Z0CHDFrgH8Jy/CUPMz7J7JSX7ATx7GdPTG4rXu5ChLMnyfuZ+m8dfRzscGgR9y1FSfKRkGXnYQJugM3M7Dn+kPkKNRG7YXG2IiEXWP0ev2uWbHvZYQMXY54NQdpn8UI//d1+FTvsqF+F0KzWxUBEnYW5k4XWyEK1DykIgHkBRrkFbDXWMZ/7Imdi60IigI1bMVCrRBQqg0/3jd+VIDlmGM57thJmVjIQZav1o7N3oOlGXjTpj+2iAMcbUVQqjRawSovKG/Tt+NC0NS5NO2LoZM3QzYbqi/72l93kZVfKoKMmd/Q2A4SAdj7PcYyPU9VYQCJfl8WiOLrItFW/2C597LzVYhOy0dmnmFhxn4X+LlJ4OUogaO08DdDmqwAKTlKPMwp0J7w0Vc5ARJ2tEPMS0CNa592RefPLqMAIrR67wSufN2NAifM6wSazQABEna0LSyCgFQErJjaWseWj7dHIOphxXn3ihs/FuSIN/v56/SdseYunW5ZhGfNZwQJO/OxpplYxFMylg1phhn75OAFbpj4dzTWjtUNniJORKA+CJCwqw/qNGc5AuwKerFeBY6F+6KNqok6vWdT9G7pWjJmcrYSszZWnvePXGB9BEjYWZ9PaUVEgAiYToCEnenMqEcdEGCXj789H6bzzoyVDfvuQEyF+d3YdScL8ihOGVJsFqvc8cvRynP21cESaMh6JkDCrp4dQNMTASJgEQRI2FmEG8gIRuC9gYFo72+vA4OlarmVmIs7iXJk5anA/j8L5GARsV2CHOHI7nDLfCwdyNztkUildCCNblORsGt0LqcFEwEiYIAACTvaFhZDwNtJjHlDg+FcSQRuZcayIAcWbXsjIddi1kSGmI8ACTvzsaaZiAARsFwCJOws1zeN0jJWF3V6Tx+DiZYrA3I3SY6tl1NJ1DXKXVO4aBJ2jdj5tHQiQARKCJCwo81gkQSYwOvT0hXNvWy1KVDKVqJged0eypRIzWH/U+HUgyw8SM2zyHWQUeYjQMLOfKxpJiJABCyXAAk7y/UNWVaGAAuUkIgE2pqquUrTcvsRyMZBgIRd4/AzrZIIEIHKCZCwox1CBIiAVRAgYWcVbqRFEAEiUEMCJOxqCJC6EwEiYBkESNhZhh/ICiJABOqXAAm7+uVPsxMBIlBLBEjY1RJIGoYIEIEGTYCEXYN2HxlPBIhAMQESdrQXLIFA/9mnMHVwACb11S1zaAm2kQ2NgwAJu8bhZ1olEbB6AiTsrN7FDWKBTNg92twZ37zatkHYS0ZaHwESdtbnU1oREWiUBEjYNUq3W9yiSdhZnEsanUEk7Bqdy2nBRMA6CZCws06/NrRVMWHXoYUzFs6gE7uG5jtrsZeEnbV4ktZBBBo5ARJ2jXwDWMDyWVnDge+epqtYC/BFYzaBhF1j9j6tnQhYEQESdlbkzAa6lMS0fDz/xSW0C3HCd6890kBXQWY3dAIk7Bq6B8l+IkAEtARI2NFGqG8CF+9m4MNlt9A22BE/vN6uvs2h+RspARJ2jdTxtGwiYG0ESNhZm0cb3np2nEzE0m0RcHeywYZPujS8BZDFVkGAhJ1VuJEWQQSIAAk72gP1TeDn7RHYfz4VcoUKO7/oBqlEWN8m0fyNkAAJu0bodFoyEbBGAiTsrNGrDWtNL33znzZwYueZJMx7Pgzd27o3rAWQtVZBgISdVbiRFkEEiAAJO8vfA8evPsThy6mWb2g1LJQr1Pjvfhb+/KgTVuyJxo2IbLQMcKjGSJbfxdVBjOnDgmAvFVm+sY3QQhJ2jdDptGQiYI0ESNhZtlcLVBo8+/kFhPjYI8DL1rKNraZ1Ib4OGPRYEyQ8zMPO00lga7bGb/+FVLw0NBDDezS1xuU1+DWRsGvwLqQFEAEiwAiQsLPsfXD6RhoWrLmHjZ91oZMey3ZVldYt2xmFa+FZWPp2+yrbUgPzEyBhZ37mNCMRIAJ1QICEXR1ArcUh1/wbg/O3MrDkLRIDtYi1XoY6czMd8/+8i11fdQPHcfViA01aMQESdrQ7iAARsAoCJOws241frr0HAQe8PzHUsg0l66okEJssx7SF/2HtnM7wcpVU2Z4amJcACTvz8qbZiAARqCMCJOzqCGwtDfv6j1fRra0bJvfzr6URaZj6IsDeDg774Ay+eKUNOrZwqS8zaN4KCJCwo61BBIiAVRAgYWfZbhw15yzeGhOCpzp4WrahZJ1RBJ5bcBHP9PbFsO4UQGEUMDM2ImFnRtg0FREgAnVHgIRd3bGt6cgKpRpDPzyLRW+2R+tA60wBUlNGDa3/7J+vo22wM14YFNDQTLd6e0nYWb2LaYFEoHEQIGFnuX7OlhdgzNzz+OWd9mjuR8LOcj1lvGUfLruJ4KZ2eHlYsPGdqKVZCJCwMwtmmoQIEIG6JkDCrq4JV3/89Gwlxn92Acvf7YAgb7vqD0Q9LYbAvBW30MRNipmjmlmMTWRIIQESdrQTiAARsAoCJOws140pGQpMmn8Rqz7oCF9P60xObLn068ayz1bdgZujGG+MCambCWjUahMgYVdtdNSRCBABSyJAws6SvKFrS45chQ2H4zC+jy+c7MSWayhZZjQBVhpOLBKgVzuqh2s0NDM1JGFnJtA0DREgAnVLgIRd3fKl0YkAEWgYBEjYNQw/kZVEgAhUQYCEHW0RIkAEiAC9saM9QASIgJUQIGFnJY6kZRABIlAjAnRiVyN81JkIEAFLIUDCzlI8QXYQASJQnwRI2NUnfZqbCBCBWiNAwq7WUNJARIAINGACJOwasPPIdCJABEoJkLCj3UAEiAARoDd2tAeIABGwEgIk7CzXkQqFAhcvXoRGo7FcI8kykwn4+PggJITy2JkMro470IldHQOm4YkAETAPARJ25uFs6izZ2dl47LHHcOfOHVO7UvsGQGDlypWYOnVqA7C08ZhIwq7x+JpWSgSsmgAJO8tzL8/zGDVqFHbs2IGmjmJ0C6A6sZbnpepZlJWvwuHwHEgkEpw6dQqdOnWq3kDUq9YJkLCrdaQ0IBEgAvVBgIRdfVCvfM758+dj7ty5CHKV4ND0ULhKRZZnJFlUbQJLzyRj3sEE+Pv74/Lly/Dw8Kj2WNSx9giQsKs9ljQSESAC9UiAhF09wjcw9d69ezFkyBDYijjsfyEUbZpQjVjL8lDtWPPi1ihsu5mBvn37Yv/+/RAKhbUzMI1SbQIk7KqNjjoSASJgSQRI2FmON8LDw9G5c2dkZmbi99FBGN3G1XKMI0tqlUBugQYD/riL2yn5ePfdd7Fw4cJaHZ8GM50ACTvTmVEPIkAELJAACTvLcIpcLke3bt1w/fp1vPqYJxYM8LMMw8iKOiMQma5A7xX3kJ2vwt9//41nnnmmzuaigasmQMKuakbUgggQgQZAgISdZThp4sSJWL9+PXoGOmLr5BCIBJxlGEZW1CmBA/ezMOHvCNja2uH8+fNo06ZNnc5Hg1dMgIQd7Q4iQASsggAJu/p34w8//IBZs2bBx0mMI9ND4WlvU/9GkQVmI/D18UR8fSwJzZs31+YtdHZ2NtvcNFEpARJ2tBuIABGwCgIk7OrXjUePHkW/fv0ggAZ7prZAJx/7+jWIZjc7AQ3PY/LGCOy7l60NnNm5cyc4jk5sze0IEnbmJk7zEQEiUCcESNjVCVajBo2Li0OHDh3w8OFDLBoagOc6uBvVjxpZH4FMhRr9f7+L8HQFPvnkE3z66afWt0gLXxEJOwt3EJlHBIiAcQRI2BnHqbZbsXJhPXv21F69Tenojh+GBNT2FDReAyNwOzVfGymbq9RoT+2GDh3awFbQsM0lYdew/UfWEwEiUESAhF39bIXp06djxYoV6Oxrh51TQiER0tVb/XjCsmbdfisD07ZEwcnJCZcuXdK+u6PPPARI2JmHM81i5QQ8fVzQ2U2gu0pehTv3shFZYLmLb6h2GyJKws78+2zZsmV45ZVX4GkvwpGXWsHHkSpLmN8LljvjJwfjseRMClq3bq2NlLW3p3eX5vAWCTtzUKY5rJwAh7Yd/DAqxBbNnUUQFR1YaPIy8cOGBFxWW+ryG6rdhnmSsDPvPjt79iyeeOIJ8GoVtj/XHN2pDqx5HdAAZlNpeIxbH45jETkYN24cNm7c2ACsbvgmkrBr+D6kFVgIAY4TY8iQ5pjgzZQdj4K4JMzal4F0C7GvIjMaqt366yFhZ76NlpycjI4dOyIhIQFfDvDDK495mm9ymqlBEUiTF6DP7/cQm6XUVqVg1Snoq1sCJOzqli+N3qgICNB7YCim+7MrWR6x/0Xi40v5sNgDuxLfNFS7dTcXCTvz/GVTqVTauqDHjx/HM4+44teRQeaZmGZpsASuJsoxaNV9KDXAgQMHtPuHvrojQMKu7tjSyI2MACeQYvyoYAxz5QBejeMH7+G3aL5CClIbIQqU6moIPw4s72uusuKxTUFvqt2mjG3OtiTszEP7rbfewuLFi/GItxR7p4bBTqz3ttQ8ZtAsDYzA+qsPMfOfWLi7u+Py5csICKDo6bpyIQm7uiJL4zY6ApzEBbMnNEUHEQdeJcOqv2NwMM8wBkdvT7zTxxlZl2Lwy10llEbT4hDc1h+zHwH2HYzDrlSN0T0ramiK3TWerA4HIGFXh3CLhl67di0mT54MN1sRDk0PQ6ALVZaoe+rWM8O7+2Kx4sJD7TX+qVOnIJVKrWdxFrQSEnYW5AwypSEQ4ODt46RN7eDvKIQNr0F2jgL3Y7JwReKJL/u7ggXHqtJT8fm2VIQbOFTjOAmGDmmG8ewtnkaFi6ej8fMdhRHiToCQR/wwu4s9nDkg+W4sPjkpQ45R2Gput1HT1GMjEnZ1C//KlSt4/PHHoVDkY/PEZujdjMpF1S1x6xtdqdZg5JoHOBubi6lTp2LlypXWt0gLWBEJOwtwApnQMAg4ebpifHdP9PRgka88NCoNclUcbCUCCKFGuoyHm6MIHHhk3YvF28dlFYo1gZ0Dnhvoh/7uAq24u3w2GktvVSbuBGjR3h/vdLLTirqUyAR8czQLCUYc2NWm3ZbsKRJ2deed9PR0dOrUCVFRUZjbxxvv9Ghad5PRyFZNICmHBVPcRZKsAD/99BNee+01q15vfSyOhF0RdTd/dwzyFUFYiRfUMhl23chFVhWesvd2w4ggMVhGJ16jxMUrGbhdyV2b0M0ZY0OlkNTC3PWxiWp7TqG7M55pIYW4GgPzPMDzPJQqHvlKFbLkBUjOVCAmowDyaj9J4+DbvCne6uUMXyGgyJZh9/kUHIhWIIcHHNxd8GK/pujiyEGb6YTX4PrpB1h4W4XKdJfQ1gHPDvTFIA+hVtxdOReNpTcVyC+3bgFCH/XHrI52cDRJ1NWN3dVwi1m6kLCrG8wajQaDBw/G/v37MbSlM/4c16xuJqJRGw2B83EyDPvzAXhOiGPHjqF79+6NZu3mWCgJuyLKzdr44eUOjvCTFv1y1qfP88h+EId3juUY+MVb2pjjbDBsaDM800RQ+EtenYu/NkVjr6xid3q2CcTCbvawqTBhO4/c8Hi8eSS70rnNsWHMMYckyBdL+jrDvgYJ7Jm4AzgU1p/moVIWIDpBhtP303EiWolcExbiEeSDD3s7w5uJurQMLNmXhP/03s75dQjGgk62WjEPTT52/BOBjQ+rnkQgtce4AX4Y5lUo7q5diMXi63ll/CxAyw7+eKujPZzAm3RSV5d2V70y87cgYVc3zD/66CN8+eWXCPWQ4MC0MDhJKvvnb93YQKNaH4E/Lj7E/+2Nhbe3tzaYomlTOgWuLS+TsCtDknNwwuvDfNFNR1HwUMtl2HAwAftTqo5gtPVriq8GuMKjOFDMCGHHBIhrE1dMeaoJOhef+hTZxSvzcfxsAjbcz0d2tU+camu7mG8cJzcn9O/ijeF+hQl/VRkZWH1Jhhy+UO15t2iC8UFiQKXEneg8pKp5CEVCuLnaIsRFCOTkYMu5dMSKpWjj54guAfbwtAE4nkdGcjrWnEjBuayqgQodXTBreFM8asuBV+Ziw85o7Mooz8G5VQAW93DQCjtTExMLpHYY2c8Po71FWnF380IMfryejzwI0aqjP97pYAd7E0WdOew2324wbiYSdsZxMqXV1q1bMWbMGK2Y+/fFULRwp8fupvCjtpUTeP2faKy7mo4ePXrgyJEjEIurc09DlPUJkLDTIxLUsRk+6ygtPHnRHvYYd63GmrJEr/0GNsMUP2HhaR37jBJ2hU1dWgbg+x4OkBR15jUKHD0YhT9i1JVe6VnrthZInfHGOB90lXBQJSXhvd0ZSNaexAEtuzXHR21twJWr7sDBw9cTLz3pjtbCfOw9FIMNCWpwtrZ4omMTjA+zhaMAWrG+ek8sDmVWRk+Ix3uHYGaISCsIY69FYe6FPBiqEObTPhhfdmEndjyU8UmYvde0xMScxBbD+vljnDebS4XblxKwX+yBl9ubLuoA89ltSXuPhF3teuP27dvo2rUrZDIZ1jzTDEPCKFiidgnTaHkqNQavegCW527mzJlYunQpQakFAiTs9CCK/Zri+wGFkY3aT5OPbdsjsNmI8gHiJk3wxRA3+AjK3CGaIOwkwb5Y0qfoCpLnkXArBp+eyTXp2rAW9oR2CJGNCF4OYjhJBZAKOYgERVfLFU3AFyA6Lh8pmqpPwYy1kUWPjhreDGM8TRF2haPb+3rjs6dd0SQ/B79ti8NJuVZ6w6+lL97t7qg9Uc1PTsanuzIQWyQW9e0SuXrgk5FeaMZungpkWL05BgcM3uFy6NgrFLPCmKDnEXslEh9fND0xMWcjxaCjD6aDAAAgAElEQVS+fpjgY4OSLcSbdv2q9Z2Z7TbWn3XdjoRd7RHOzs5Gly5dcO/ePczu6YWPe/vW3uA0EhEoQyA2iwVT3EGaXIVVq1ZhypQpxKeGBEjY6QEUunjg09FeaFZylSrDqg0x+LeCfGSl3YXo2S8EM4JYVGSZzwRh17xLCOa1l2gDOHhZJn7cloCLihp62OjuAvj4OeOJEEc84m0LXwcBxIUP1Iz71DKs3FBx3jbjBiknrTBocAtM9jFd2LHT00GDQzCxKYek69H46Jy8KEJVgI69mmFWmA04jQpH/72P5bGGxWiLriGY2475g0deTCJmH8g0GDijfVc5LATjvYxLTFwZC87GHlNGBKCfM3vryUOVnob5O1Jw34TyFfVhd/X8W7u9SNjVDk/2PnX06NHYvn07+oY4YcOEZhCa8t+C2jGDRmlEBI5FyTDmr/sQ20hw+vRpbZ47+qpPgISdHjuB1AXvTvBBu+L3weoc/L42FkeqyCArdPPEZyM8ESzgkavQwE5adB1rpLDjhHaYPDoQT7NcFrwaV09F4ts7SrNcwUpdnfFsDy/0biIuKWBv8paqI2HXf1ALTPU1Xdgx+wM7BuPzjrYQ5KTjy01JuFUUospOtD4d5aX1VeqtaLx7Wl7uelV7WjiyGca4M3/wuHvuAebfKDDoD07ihLfG+6KLTWFi4tV/G/MPgfKEOU6Ejo8HYGYracl1PNsL4dfi8N2FqqOx2Yj1YbfJe6WOOpCwqx2wCxYswJw5cxDkKsHhF0PhYlvyMKV2JqBRiIABAkvOpOCTg/HaihSXLl2Ch4cHcaomARJ2+uDETnh7ki+6sBf77DNK2AnQ5ckQvNVCDD4/B9vvCTG8nV3hOz0jhZ1TMz983dsJThwPVdpD/G9HKh4YkaOsmn4v6ebi3wSzn3JDs+KHfeDBqzVIy1QgQaZGjkKlre9X5Q0rr8DpC+m4Y+gBWrWNFKEmwk4S4odfnnKChJdj3aYo7C7K5MsJ7DBlXCD6O3JQxSfizb0Z5U7iylZjgEaJ/XvC8WeS4ZM9uyAffNvXRZtfrrLExJWe1HEidO4RgFfDWNobDeLvp+K0jRtGBbK0OcaLO3PbXW3X1kFHEnY1h7pv3z5tahOpiMP+F0LRtoltzQelEYiAkQRe2BKJHbcy0a9fP7C9KBRSBLaR6HSakbDTo8YJHfHaBD90lxYLOxn+WB+DQ+WTi5X0FDi74eORTdBSBCTejMEvSi/M61iU+sIIYVf4liwYoz0F2uvBk0fC8Vtk3QdMSD098MEgT7QoyrOiUeTj9JUU7LwrQ5zxNa6qs++M7FNDYRfggx/6u8CZV2DP7nCsTS6ctuyplio5BR/sSkOi3js7oacXvhjmAT92Jc98uDEGe3MNCTshnujXHC8HFb6vy74fh7eO5RhRRaIUAbs27tYzAC+HSmADDaJvx+O7MzlIF0jQp3cAni8SdxHX4/Dt+cpP7sxpt5FONFszEnY1Qx0REaFNQpyZmYllowIxtq1bzQak3kTARAIypRoD/riHO6n5eO+99/D111+bOAI11/6O4wsTftXqd+jQIa3iDn58OB6f9mWtjl3Xg3ECe7w4PgC9i1OeqHPx59/R2K99fG/o49C2ewjea20DgVKGNVtjEd6qWclbOWNO7JyDfbGgjzNcOR75icmYtycD8bXvFl3jxXaYNDwQg1wL33Jp8nKxbm8s9qbX+naogctqLux+7O8CJ16BfbvDsaZY2AlsMWlMEAY5F57YvWMggtXGxxvfPO1WmLZGnYPf/orFcQOnkZImXvh8iDv8WLQDr8GN0w/wdRWJicsCYaLu8V4BeKkFE3VqRNyIx3fnZCgO1uWEuuIu8kYcvjlXsbgzl901cGqddSVhV320crkc3bp1w/Xr1/HqY55YMMCv+oNRTyJQAwLh6Qr0XXEP2fkqbNq0CWPHjq3BaI2zKwk7Pb+za7rJY4reurGfaeRYuzEKeypIMCywd8Hs0U3xqAR4eCcWH5yUIejx5vi4jY1RCYo5gRSjRgRhtLa0lAL/7o3E6sS6v4P1eSQQ87vaF77l0qhw4nA4lkXV/SmhaX/NaibsbFv44+cnHGHD52HLtkhsLco/J7R1wezxTdFeBKTcisZ7Bt7Y6QskQ+8sOaEUo4cEYZRXUcSwRoFdO8OxPtW4VXICMXr0CsD05hKIocb9q7H47qK8XO1XJu769gnAcwGF17KRN+LxbRnxV3Y2c9ht3OrM34qEXfWZT5o0CevWrUOPQAdsndwC4uLgseoPST2JQLUJ7L+fhQkbImBvb49z586hTZs21R6rMXYkYacv7Dgpxo0OxgjXoqtYjRwbt0Rhh8E6Yhyadw7G3PZSCNVybNwehX8yOTzSoznebyU2Stg5h/jhy6ec4MzxyIlKwEcHs2BEZpUa7VVOYIuJo4Mw2IWtkYciIRkf7i3NEVejwWu1c82EXUjnZpj3qBRCRRaWro/HWVWhca4t/bGwhyPseBUO77+PFfHlTylZMMz/RnoikP2C0+Rj6/YIbCnjGBbo0LlHIF7yysMVtTO6e3Lg1LrX9h6+TvBMz8ZtAxHVnMAGvZ4MwLRmNhCxvHVXYvHj5bwKU9swcdevTwAmF4u7m/H49mzpyV4x9rq2u1bdW8uDkbCrHtAff/wR77zzDnwcxTj8cii87GyqNxD1IgK1SOCrY0lYeDwRzZs3x8WLF+HsTHkUjcVLwq6csLPBiOEhGOdZLOx0T3vKNhdInTBzjC+62QKZ4fH44Eg2csChfc8WeLdlUdqTSt7YsdO6cSOCMZxFXqrysG1nJLakGeu66rcTe3phfvH7MV6Dq6ce4Ns7ldc1rf5sNelZfWFXnILkGS8gPyYR7x3I1ApmTmKPqcMD0NcZUCalYN7udMQZuPbmhA54ebw/nrArjIqNuhKJzy7la9/OCSRS9Onui2d9VNi0LxVuvQMx1IVFxObgj/WxOKwordFqGxGHN44VRW0UoWC29eodgBeDbSDkCytNLL6eX2W+Qibu+vcNwHP+YgigRtT1OHx1PlfnhK8u7a6JJ83Rl4Sd6ZRZnc6+fftCAA32TG2BTj72pg9CPYhAHRDQ8Dwm/h2JA/ezMGzYMOzYsQMcpd0xijQJu3KYxBg8pDkmNS0WdvnY/k8ENhmo++n/aDA+72QLMV9YG7SwTdlktZVHxbq28McXTzhqI2HT7sbiwxOyKn+5G+XVKhqx2rTfdrMvTG1S6YlkbcxWkzGqL+ycA33wWV9neEKNowcfYHmMBsWnXs8HiAGFDOt2x2JvRkVvCjm07tYM77eRaDnxGhViYmWIUYsQ7GMPH+Rj+8FobE0WYejQ5ni2SWGVktTEHNzXSPGorw00KWlYejAF18ud2AkQ1sEfsx61wf3zMVh6U2F0DWC2hgF9AzDRR4PjR6KxMlpfkNel3TXxZd33JWFnGuO4uDhtvrDU1FT8ONQfz3eg9BKmEaTWdU0gK1+Dvn/cQUSaAp999hnmzZtX11Naxfgk7Mq5UYjeA1tgun/RIxP2bmpXONan6DbkbBwwfXQAnnLgkRudgA/+Lb5C5dDpiVC8E1p5HjtOaIsJI4MwmF35KmX4a1sM9uke7NTRBtO9KtbkZeD7DYn4z4QEuHVkmIFhqyPsOPg288JL3d3QQsIj8V48vjiRA4WbM8b1bIK+nkJo8uTYdigW/yRrKs0TKJDYYWR/P4xqItKpBJGTlon1x5NwrCjQpElLf3zagwn0wiUwEfjgdhKWX8hGfNH1b/nFCeDnJkRKeoFJEbTaccQStHZT406y4VPWurXbfN43dSYSdsYTUygU6NWrFy5cuIDnO3rgxyH+xnemlkTAjARup+RjwMq7yFVqsGvXLgwZMsSMszfMqUjYlfMbh579wjAjqPhBvOEcZl5tAvFFN3tIeQUO7Ikok+OMQ+cnW+DtFpVfxbqH+ePLno7a4u7x16Mw53ye6b/gq7XnBOjVPxSvBBauT5WWinnbHyK6rqNwq2Vr1cJOoFLidkyeNopUJBLB01WKQAchOHUBbt1IwfYkDm1auKJ3sBROnAaJ0Wn468xDXDVYGqy8kRwnRHCgE9q5iSAsKEDCQzmuJSr1TlY5+Pq7oKuXCMhX4l5MNm7m1G90cUO1u1rbpKgTCTvj6b300kv4/fff0cnHDrumhkIiNKHKjPHTUEsiUCsEtt/KwLQtUdp3duy9HXt3R1/FBEjYGRB2XZ4KxVvNi07cNCoc3HcfKxNKf1GzKhGTtFUigPyEZHysE3hgQNjp5UDT9h8ViKfZu6y8bPy6NQ6nqixZVlvbWITBQ1qUXDXzGjVkCh41liEaOdZvi8PxSvL9mb6CqoWdfvpKlovvyv1MHL6dCVloEOa0kxZepSrl2HUoDpviVbDIw0nT4VAPPQIk7IzbEsuXL8fLL78MT3sRDk8Pg68TBUsYR45a1SeBeQfjsfRMijZClkXKsohZ+gwTIGFnQNjpBD9oVDhy4D5+jyuVPuxt3FdPOMKBZ7VGH2B5bNn0JOWF3Z8bY7C/THJbz1YBWNDdAfbQIPxSJD67ojCb2GAP90vehNXm34p6KikmUOXj/O1c5Lk64HFfCWwK8nHsZCz+iCiAGgL4BXvg+W7uaG0H5GVkYuXBJJzOrrGMrU1yNFYtESBhVzVI9guRXcFq1Cpsn9xcm96EPiLQEAioNDzGrnuA45EyjB8/Hhs2bGgIZteLjSTsDGBv2a05PmprA+1pkJ6w4zgpRo8Kxig3QJXyEHN2PtSLqtQXdjKs2lCmdqjYDlNHBmqLvPM5Gfh2WyKumrHKg07Beu17MA3kBbUgdNRyrNsai6NmPrHj8jLxw4YEXFYL0LprEN59RAobVR627onCltTCdQkdnPDyIF/0cAbUWZlYuisRF8x2Qlovf68b5aQk7Cp3e0pKijZYIj4+HgsG+OLVx7wa5T6hRTdcAg/lBejz+z3EZSnx7bffYvbs2Q13MXVoOQk7A3CL859pa71q2KncfSyPLRQJDkG++EpbJUKNM0cf4Odw/aS+5YVd2ZJk3m0C8b9u9rCDGtdOR+Cb24YLy9edz0UYOLgFnvcpfFOjSkrCe7stMYcds67qq9hSYQdwNvZ4YVQA+joCytRUfLbzIaKKDlOlnp74aLAHQsRAVlQC5h3KwsNa0LN15yca2VQCJOwqJqZSqbRpTY4fP46xbV2xbFSQqXipPRGwCAJXEuV4etV9qDTAv//+iz59+liEXZZkBAk7A97wfTQYX3QuqvXKF+DoAXbdyoOddg0ZEoJnvQF1+kP8b3sqHpQrEqEv7HJQXLWgrPBQpadh/o4U3Df7gy8OPfqG4dXgwuAJdVYaFmxJxt26L3ZRjX1vmrBjE7Br7i+6O2iF88Xj4Vh0vzhylENIxyDM6WALG16F00fD8UuEpVXaqAYi6lJCgIRdxZvh7bffxqJFi9C2iRT7XgiDHZWWoL85DZjAuqsP8fo/sXB3d8fly5cREBDQgFdT+6aTsDPA1KNtIL5jed7Yz8oIO6mvN74c6AYvTo3LJ8Lxwz1D6SZ0hR2vysavf8XhpArwaRuI+Y/ZQ1LPwiK0awjmtJMUXjUX5OC39bE4bsbrYOO3senCrjQwBVBnpuOLbcm4WySetcJ6ZAD6OBX+7MvtybhTYToS462klpZBgISdYT+sXbsWkydPhqutEIdeDEOQq8QyHEZWEIEaEPi/vbH44+JDdOrUCSdPnoRUKq3BaNbVlYSdAX86tgzAkp4OELOflVzFitD36WZ4wU8ITVY6vtqahFsGT9v0hJ0yGz+tjcNZkQOmjyrMe6eopOKBObaXQ3M//PikE2yL6sTqR/2awwbj5jBd2LFxPVr644seLJWMGtdOReKbO8qSfHVlf3brXCS+vFH6M+NsolaWSoCEXXnPXL16Fd26dYNCkY9NE0LQJ8TJUt1HdhEBkwgo1RoM+/MBLsTl4oUXXsAff/xhUn9rbkzCzoB3JSF++OUpJ0iKhA+Lil2l9MQXQ93hy/G4dTYcX96s6G2crrDTKLKweF08EtsE4fMudrDhlTi4LxyryqRPMfcG4xzc8PHYJmilLT3BI+lGDD44m4sCcxtS5XzVE3Zl08lostOxcGsSbhSdzLHE0BNHBmGQKwdenonFWxJwQVGlIdSgARAgYafrpIyMDG2wRFRUFOb0bopZPb0bgBfJRCJgPIHEbBX6rLiDZFkBfv75Z7z66qvGd7biliTsDDjXJsgXS/o6w6FE2EXiTlgwZgSLAFkGvtuSiP8qVEF6wo5FbW7LQdcRfuhlD8iiE/Hxv5kwUKHMbNuM48R4elAIJvkUvrPT5GVh6ZZ4nKvViNbaWE71hJ3uqZ0Gt85F6JzMlZRygwZR/0Vh3uV8s6WbqQ0qNIZhAiTsSrloNBoMHjwY+/fvx5AwZ6weFwwB1dmkvzpWSOBsbA6G/xkOTigCq338+OOPW+EqTVsSCTsDvMR+TfH9AFe4sapiGhWOncuAf1dPNBMYk3dO7ypWloE196R4toMtxJo8bN8Zic31qeqK1usY5IsvWXQvWyPPI/luLD45KdMpKG/aVqqL1qURvPrRu8Upabj8LCxeH48LetfipUmkOa0YZ2llrhSdzHECKcaNCMZwdw68Igcrt8XhsIxCZOvCg+Yck4RdKe2PP/4YX3zxBUI9JDgwLQxOEv1U3ub0DM1FBOqWwPKLKXh/bzyaNm2qDabw9m7cp9Mk7AzsN6F3E3w9xB1NtTeVGmTIebjYC8HnZeHnLfE4U+nJlt6JXUEBMjViuEl4pN+Pw4fHciCr2z1u1Ojs1K7fwGaY4ldUYYNX4/7VOCy5lIs0C9E4zMYhQ5pjgjdXLi1LSa7BgsI3jKcNBEGUvqfT4MHFSHx+tTQRtHOwL+b3doabgEfavTh8fDzHwkStUW6kRmUIkLArhLF9+3aMGjVKK+YOTAtFqAc9Kqe/KNZPYOaOaKy/lo6ePXvi8OHDEIu1r+Qb5UfCzoDbRR6emD/cE/7sNKv443nEXo3Cxxfzqri200t3UtSfV+Zi3Y5o7MmynH3GOTjj9aE+eMyB017JspO7nPQcHLmdhYvxuYjK0dTrFSUnsMPkMax0WyXCTiPHxi1R2GGAKzu1mzw6EANZMmi962YmGp/qF4wXA0WARokThyKxPJrSn1jO7jTdEhJ2wJ07d9ClSxfIZDL8OS4IQ1u6mg6SehCBBkggT6XGoJX3cC0pH6+//jqWLFnSAFdROyaTsDN0YufigU9He6FZGWGnUeTgj62xOFJl8XgDwo7nkXgzBnPO5sLSnrFJXV3wcn9vdHUqfG9XVsiq1BrIFGoo1YCmilM8TiPH5l0JOF2LgQiugT6Y19cFXgJAlfoQXx3NRKF+4xDYzh8zw2zA8RpEX4/BgvNyGHJN6akdj4Q7cVh8o/TUjrd1wtR+nmgrBTT5edj2bwy2J2tKImhr568YjWIuAo1d2OXk5GhF3d27d/FOjyaY28fHXOhpHiJgEQRiMpXou+Iu0uQqrF69Gs8//7xF2GVuI0jYGSDOokbnjPNGy5JnKTySbkbj4zNyI4RZeWGnycvGsm1xOCE3t3uNm09kZ4cBXb0xopkEDgIdeWfcAKxVLdaKFbk6Y1JndzzuL4GjMfbwGqSnyHD0ShK2xermFmRRsM+OCMIQt6JTyUpWxKuUuHEjCcsuypBu/MqppYUQaMzCjud5jBkzBtu2bUOfEEf8PSEEQgqWsJCdSWaYk8DRyByMXfsANhIpTp8+jQ4dOphzeouYi4SdATcIpC54d4IP2hULu4Jc/LUtGnuzjfGZnrDjNYi4HIlP/ys9KTJmlPpoY+toh07BDmjtZQt/Fxu42wphK+Yg5jhU+TuiFoWdtJkffu5dlG7GaBCFaVveO5tb7vrYMaApPu/HTv6qFq3qzIf439YU3LfIShxGw2iUDRuzsGOBEixgItDFBkemh8HFVptenT4i0CgJLDqdhM8OJSIwMBCXLl3SVqhoTB8JO0MndgJ7jO7njuZFV7GKlHQsvywzeNVnaLOEtvPDKJ+izqo87DmeiusWWdmhMW11Wqu1E2iswo6lNBk0aBCkIg77XwhF2ya21u5qWh8RqJLAC5sjseN2Jvr374+9e/dCKGw8keEk7KrcHtSACBCBhkCgMQq7yMhIbUklloz4t5GBGPeIW0NwFdlIBOqcQI5SjYF/3MOd1Hy8//77+Oqrr+p8TkuZgISdpXiC7CACRKBGBBqbsMvLy9OWC7t27Rpe6eqBLwf614gfdSYClklAAF5sB04E8PkycCak43qQlo9+K+4hW6HGpk2bMHbsWMtcYi1bRcKuloHScESACNQPgcYm7CZPnoy1a9eie4ADtj3XAuKy6ZnqxwU0KxGoBQIC8K4dYNO2D2yaPQph0yAIbG3BMUVXkAl17EkoTv8Gxb0Eo+baey8bk/4Oh729Pc6fP4/WrVsb1a8hNyJh15C9R7YTASJQQqAxCbtFixbh7bffho+jGIdfDoWXnQ3tBCJgFQT4wLfhPPU5sCdx/MNDyDu6GQXJGeCY2Os1AxI/F0CTCOWWKZDfMK6M05dHE/DNiWS0aNECFy5cgLOzs1WwqmgRJOys2r20OCLQeAg0FmF3/Phx9OnTBwJosHtKKDr72jUeJ9NKrZ4A3+xDOD8/DgLZfuQu+wgF2WXuXu0Hwm7mAkgcOPAJvyJ72TJoNFVnO9DwPCZsiMC/D7IxfPhwbXUWrspUDw0XNQm7hus7spwIEIEyBBqDsIuPj0fHjh2RkpKCH4b4Y0pHD9oDRMCqCPDuA2H7WDvwUeuhuBWntzYHiCcdgH2YFMjfC9nCj6EyUE7SEJCMfBX6rbiLyHQlPv/8c8ydO9equJVdDAk7q3UtLYwINC4C1i7slEolnnjiCZw7dw7PdXDDoqGBtepgnnOFsGUPCKUCQBEO1e2brMpgI/nEgFdbiDy8wSEbmqTrUKcblbjUAB8BeKcwiHx8IRDIoUm6UYOxhIDPU5B0GAyx6Ahy/9lVqU94SVMIm4ZA6OgAKNOgTroBTVaeFfnQATYTD8CupRTI3gbZD/+DSl1mefYDYTtuIBC+D4r/jkAjK9BZ+60UOQb8cR95Kh67du3C4MGDrYhN6VJI2FmlW2lRRKDxEbB2YffKK69g2bJl6ORjh11TQyERVn0FZdIuaP4RnCaNgVDIgc/eCtkP83V/aZo0WENpLAACxsN+4BSI/bxKErHzGjn48L+Qu305VDlllUPl6+Jd+8Bu6JuQhPiDK0qIzquzoL62CLm7tkOjqzMqHYx37wPbga9DGhoIaFKhvrkcuds2G7x65B06QtL3Vdi2fRScRFhSHpLX5EETsxeKf5dAEWtBhcqruT14SS/Yv/E9JE4cNNc/QtamA7ojeY2E/YgZEPt5AspYqM4tQe6xQ+DLcN96MwPTt0Zp39mx5MUhISHVtMZyu5Gws1zfkGVEgAiYQMCahd3vv/+Ol156CR72Ihye3hJ+TmITyFTdlBeGwfalVZD6SLSigM8/gNxvPkCBCUKk6lksrYUYXPu5cBw2BEIWe1Igg0YtBScVFwkjHnzSOuSs/A5qIw69eOfBsJ/+CWzsU1Bw7m8UpNtA/OhkiP1cwKEAqhufQrZ5rxGnoDbg2r0Px6EjIJAooLmzEvL9f0GVbrjSOO82BPbPzYGNuw2Qdhz5J7ajID0fAt+BkPYaDqGdAFBGQLH1NeTdSrE0J5hgjxSC3svh9FQbcPJTkP/+FhRphkoEScGFToDdwGkQe9qCS/4Hso0LUZBaym/uvwn46Wwy2rZti7Nnz2ojZq3pI2FnTd6ktRCBRkzAWoUdS9HQq1cvqFUF2PZcCHoGONaylwXguiyC05AeEBSnTFGdQu43b0BphKCpZWPMNJwQXLv5cBzZH4KMg8g/sAyKexHQwAHCNq/DYfjYwitpXg3NpVnI+udE5adrnDtsntkE+9YOUJ+cjpx/r2nb89KusHtpCSSeYnCaFORvnIi8W5VVoraBoOtXcBz0JDikQnXoPeSeulahGOTFrWE7bTmkvrZA9mHkrngPBRmlYocPfANOz0+FSMyBzz+HvN9mViCGzIS92tMIgLAP4DR+DASa+1BsmIG8B5mV+0TaGpLR38EuzAvIOQX5mvegTC4UdyoNjzFrH+BElAzPPvss1q9fX23LLLEjCTtL9ArZRASIgMkErFHYsSAJFizBgibmD/DFa495mcylqg68bW/Yz1wIG5ubUGW1griJGFBfhXzxdCgyjL+GrGoeS/o5zzlD3Od72PscR86mVdDkl15r801ehtMrr0AkKvqzghuQ//YCFCkVsyjpw92G/JcpZdoKgDb/g1N/T6gub0b+pcPQ5Fb02p+Jl4/gNH4UhAIZCg7PgOz47UqwCYDuv8BlYBdwyEPB/omQnY7Wa+8A0chNcOjYBBw00Fx9H1lbDlmSK4ywRQA+8GU4TZwOkSgGiq0zIb+ZaEQ/gBe3gHTSb7Bt5gyk/QPZis+hkhU+HH0oL0Dv5XcRn12A7777DrNmzTJqzIbQiIRdQ/AS2UgEiECVBKxN2KlUKm2dy6NHj2JsW1csGxVUJQPTG9hA2H8NHHsGQ318BvK9foJ9Kwk49QPkLZ+A/ATrFHaFnATgOY1eJQMxRIM3w6Gbf8k7NUAD9blXkb37QgV4hRD0XQenJ1sA6euRs+RbqNVlo04MzVN+KN5hAOxnLCh8P3b/K2T/tbnyQAnho7CduQy2HiLwytOQ//gGlEWipezofLMP4Pz8MxCyA0jFKcgXvWmwnel7xxw92BvIF+EwcQbE4miTRF2xdbzzMDi88glsHADNzXnI3rinhOt/CXkYtPoeVBrg4MGD6N27tzkWVedzkLCrc8Q0AREgAuYgYG3Cjp0g/PDDD2jTRIp9L4TBvg5KS/Duk+D4yiyIFLsh+/kLcMMPwr6NHThNIhSrR0IeadWP7MptS17YBXZv/QypswYFF4+E8NEAACAASURBVHdB0HFkYTCJbBdki+ZCpSgfsMJzwZC+9Dfs/IRQ3/gI2Rv3V2O7O0A0dC0cuviDU1V9Qsgm4IPehfPUCVrBhtjFyPx9lUEhyEv6wv6dhZDYcQCfC+W2oci90kACKfymwWHyzGqLumIBL+j+G5wGdgLHJ0Kx7lnI7+WU+OivK2l4c2cMPDw8cPnyZfj7N/zSfCTsqvFXkLoQASJgeQSsSdixNz8TJ06Ei1SIw9PDEOQqqXXgPHsXNnYd7NvaQrV3MmRnUyAaux8O7RzBabKg+HsA5LcbmbDzewvOL06BkEtE/prXoXlqPWwDbcBpMqDYNBzym7nl/WA/EvbvzIWNjQoF+0dDdireZF/xbpPg+NosiG14qG98jJxN+6sIshBC2GctHJ8KLbxi/W82srYdMzgvz/lCOm0r7AJZwI0G6ktvInvHaZNtNHcHjecYOE59D2LbBCjZ9esN40qIGbKTl/aG/esLIXESgI9egqw/dEXwrN0xWHU5DZ07d8bJkychkdT+3zdz8iNhZ07aNBcRIAJ1RsBahN21a9fQrVs35OfnYeOzIejb3KlOmPFB/wfnKRMgSFuDnN9+hLrABsLRe+D4qBs4XgHl9gHI/a/0ZKNOjLCoQQXgeq6A84D2gOos5N+/AWXLH+EyrAc4TqMVXIZO4/gW8+AyaSQEiINi1RjIo0wVw0II+v8Np17NtAEWivWjIL9bedQKyzkombwbdi2khRG3hyci52h4BTRtIBqzFw7tXQsjnqO+R9bKv4yIzq2+c3ipH4R+rSHy8IHAwRWcxBYQicBxgpKUMtpTR14N9dXvoYjUFcy8tAtspy2C1DMbBTtnIPdylI4xLChFOrAXNKcXoSDVmAzFNhAO3QrHrj7g1NHIWzke+THKkjEVah7DVt/DxXg5pk2bhhUrVlR/8RbQk4SdBTiBTCACRKDmBKxB2GVkZKBTp06IjIzEx081xexe3jUHY2AEXhAC6bQ/YeuXC+Xm8ZDfyAAghnDoDjh2ZUl6C1CwdwRkZ5LqZH7LHNQB4mf3aCNb8XANcn76ESrJYDi89RlsbAXg8w8h94d3UaCjuQQQPLkaTn3bAIojyP3u/1CQb1pWZ17YAXav/wapuwjI3Q3ZD3NRUKo5DJ/CCR+F3cxlkHqIWJ4WFOwaDNn5tAqwCiHotwFOT4QUCjuD7wBr7hFtgut2z0LacSDEAf4QGJNnUSuyxiE/plScaQNbhq2GQycvaE69iexDV6BDVOwNce9v4NBVjPw/nkV+rDHCDuBbzIXLpFEQCNibyRnI3n1RZ9EJOSr0WX4bKbkq/Prrr2B5IxvqR8KuoXqO7CYCRECHQEMXdhqNBkOGDMG+ffswKMwZa8YFQ1An9SwF4Dp8C6cRT4KLLXstJYRw4GY49ggsPAU6Mhk5R+7XYJcJwD22BE59HykTiFCD4Yq7Zm5Bzm+L9QIUaj4uL+wEu7d+hdRFUOatnCPEz+yAfVuXwuvpzcMhv1H2FJOJwb2wb20PJPyM7GXLjapdWtZa3mcmnF56ESIhD/7efGSt3VblaRovHgCHd7+EjZS9m8uDcnMf5F5XVABBAK7H73Ae+GhRjsJ9yPnmI6hNPVisBDErA2Y35j1IfF0LT+R4NficSKiS4wGPxyFyEUATvhsFGbqTcqp7yP93MzRltBnv/wacpk2FqCphqL4O+dIXjE7fwkuHwGHW51pmfOqfyPl5Ubk9dCZGhhFrHoATinDs2DE8/vjjNd9Y9TACCbt6gE5TEgEiUPsEGrqwmzNnDhYsWIDm7hIcfDEMThJh7UPS5lbrAbvXfoDEMQHKPydCHikvmkcIwVN/wqlPK3BQQ33mFWTvvVwDG5igWA7ngR1qVdjV2YmTy2Q4vTULImEBCg6MhexkbOHa23wB53FPa0969NOF8MKOsH/jV0jcBFBfegvZO06ZyIuJ31/gPLgLOK4AqhNTkfNvZSlOioa3HwuH2R9BzA7sNNnI3zgQebcqFnbosgQuwx4v9IPqOGRfvV3lqaDRC/EvjFoV2QvB8QXQxO1C3pFVUITHguMFQMfv4TKiF5C4Ajkrfq5CULLnALvh+Kh71XtGdQa5370OZa5xJ6TslNr25XWw9RGDV11C3uJXoMgsn+B42flUfLA/Dk2bNsV///2HJk2aGI3CUhqSsLMUT5AdRIAI1IhAQxZ2O3bswMiRI+FgI8S/L4YhzKOuHm+LIei9Ck5PhYG//Smy/y5be1QAQfdlcHq6Y5UP8o11FO/WAeImrsY2N66dMgbqiAdVnmoZN1hpK771F3B55mkIEA/F6tElEcG8tD/s3/5KG1XK5/yDnB8/LREnvPMkOL41C2KhEqpdQ5FzoaLr0IqsEUM0Yqf22pHj86HcNgC5V2RVms67TIbjm7OKhF06FOsHQ3634vtbrsMiOI/qVSTsTiP3m9drJfk07z4aDtM+hNhRCE6TDfXpOZAdPKFzasls1QpmQS4K9oyH7Fz1gyCqBFNpAynE4w/Avo0DOE3lzF7dEY2/r6VrE4MfOnQIYnHtVnqp2Tqq7k3CrmpG1IIIEIEGQKChCru7d++iS5cuyMnJwepxQRjWspaFUBnf8a5j4TTjA4hEtyFfNhWKZL08dZ2WwGVED3Dgwd+ai8wNexqA52vDxNJ3aFAcRe53s8u8lWPXsf/Avq0zOE0S8teMRF54oYjiW82Hy/jBEPDhyPv9WeTHm5b3j+c8IHl+N+xDxIAmGfl/DUfeg6rvSHnX5+D05jsQsUNddkW8YSDkdyoWdvyj38N19FNFwu4U5AvfgMJwhTKjYfKCYEinrIJtsKNWlKrOvA3Z/vPlBDcvehIOs7+HjT3AxyxF1oqVtS7KjTNaCOHTW+DYPaDKN6TyAg0GrbqL60n5eOONN7B48WLjprCQViTsLMQRZAYRIAI1I9AQhR0Tc127dsWdO3fwdo8mmNfHp2YQKunNcy4Qj14Ph/Ye4C++jax/yl8b8m0XwnVcP+07KT5iIbJWb6inX8J1hsHgwNoo0+f2wK65BIj/GVnLftddd/tv4Ty6DwScWlsyLPvAVQClKUeQtw8535r+bo0XhMJ2xhrYerNqH8aLw8KTwtmFJ3Z8DhQbB0J+sxKl1nEJXEYywc5iLQ5D9tX/1bAOMKuoMb/oipoHn/gHsn//CRoDmlSbG/BtlhtQCF5xCLn/r72zAI/q2trwNzOZuCHBiiS4F6dQ3CmuRUPR4lZKoRS3Foq7O71FglPc3Ys7QUIheFwmmf/fJzBYgJmMnZl8+3nuc9tkn73Xetdp5psta43/2XTbwAa9JuKowRJ4VsorHTVI2Pa+9MkR7r6MQeX51/A8UoNly5ahVatWBs1mzc4Udtakz7lJgARMRsDWhJ1Wq0Xjxo0REBCAilk9sKpFNqjMclkiAXF85l7wausPB0UwYg8tR0wiKdmUPtXgXLRAgrB7MBOv5n0gcEwWLXkNpFWVhmufKXAWVR9O9cSrjR/keXOrB7feQ+DopIA2SFySmI94rQfUzf+BWx5XIIkpRN67ERt3GREz2yD6yZdX/bQuDeDx8+DXwk5s4VZC+LlPCTslUGImvGuXSLg8EbYRYROHQaPfZdJPCOH0cPJfA9dsLlDEP0HUqqaIvJx40mOt6lu49Zki5ZCD5jTCp/6ImETOtpn/jXj3EsmXqokkWLP39is0WXkbjk7OOHLkCAoXLmx+M00wA4WdCSByCBIgAesTsDVhN3bsWPz666/I7O2IPR1yI6WLeS5LiMhoFb5w/mEZXPzcvnwo/XUotU+XInTGxzcHrR9pM1jg0x4eXbvBQRmJmI21EX5apH9527SKtHDyXwfXbM6A5hQipnRCdNibkl6KJF800aqKwbXXLDh7q4C4i4ic8QOinn58oP9Dj7UOleH28x9wchFlJ76UmuZtfj5pxe7xQryaNc3g27vv8XBvDPfeA+HoqJDSp4RNH/9Joah1qQ/3foPhqFYAmhMIn9jFSiXNxBnSOfCsUTThDOmpHni18egXX6bJhx9jxJ6HyJIli1SZImXKlF98xtodKOysHQHOTwIkYBICtiTsduzYgRo1asBJpcC2tjlQMJ2rSRgkPogSKDAWXo2qQhFzGlEbFkET84mbhCnrwLVmjYS6oiHrEDZpJDRfXkD6pO1adz+ovN1N6ptC8xRxj/QrAq/3xAX/hHejSlDEX0fknBaIevShuHotjqp+DQVel+W6URlufQfB0SECMeu+Q/g5w5M5v79idwWRs/0R9eG5x0Sc0Kryw6XrQrj4JOSx0+xvi9Ddlz/h7rtpbERKlRF4uXyD3mgS66jNOQTeLeon3BT+TNUL6UtFltelz1SANmo7wscPNHIbOKmmvytw4xF3rDNCtr6fy+5TI/+w5g42XnmJatWq4Z9//oFSKQS1fBuFnXxjQ8tIgAQMIGArwk4kHxZJiEUy4tn1fdG0gPkuS0gfrE7F4frjdDil0iLuQAeE7r74aao+HeDRtat0KF8bvRPh434x4kPYVtKdvD1Uj9ANCJs0PNHVJ226zvD8sRMcVKIsVy+8OlcKXj+0gAqXXm+hfnml7UPwWmUOuHRaLqXgQCLJej8dKA84ttgG19wu0kWX+AsD8Wr1jk90d4a66Xa45ffQ62zZl/+Te3drV4/VwvJL4FU5X8I28IPpeDVvoZXObaqgrLwCnuVFGTYhhtsgdPfVL7sLIDQmDlUXXMf1p1EYMGAAxGq7nBuFnZyjQ9tIgAT0JmALwi4yMlJKevrvv/+iU3Ef/F4jo97+Ja2jCsryC+FZqQAQsglhM4dC85lqVVrPZvDo3T/h7JbmCML/7I6YN2nuDDbANoSdVpEeLm3Xw9nXAbg2HC9XbEzU0/dW1x4vRMiJXPCo/S0Quh5hk0YkaWUz4dLGP3DN/roe7f9qfvZ261vD3r0IAGj/m4uQObMS3V4VFzRcf1wG5/Ti5m0wolfUQ8SNT+W80yfI76x8fSE5slbhB+cOf8Mlk8NrMfW5lUV95jamjxoOdTfCvVjahFuxW2oh7PhTvQe88SxKEnch0XFYu3YtGjZsqPezlu5IYWdp4pyPBEjALARsQdi1bt0ay5cvR6nMbljfOifUZt7R0XrVh0eXwVC7aKDZ0xKh+29+lv17h/INzOyf2MBacQtDaeKzg9o4KOL1S0qrz4umdaoO976j4egci9htjRB29FN51pzhUH8D3Iv4AJqziL3gBXVhPyhujcHLpWuTuAqlhqrOBngUf13GzQCxoc3YC17t20Al8MaeTLiUkMj9Ba1HY3j0Ggi1OA8Xsh6hU0YYWXXinRU7bSii/hbJkRO/uBGfuQ+827aCSlSRiD2PyFnt9DpDqE/cDO/jAXWrHXDL6SRd+IheUddggbv12ku0WnUH7u7uOHHiBPLkyWO4GRZ4gsLOApA5BQmQgPkJyF3YiVxYvXr1QnoPNfZ0zIW0buZNeipWg9QNV8D963RQ6HkwX+vRDB59Xq/Yxf+H6CX1dYl6zR9Bc86gAnybwqW4N2I2zkRctHSNQGpavwHwatMUSs0JREzrnKg40nUuMA5ejapIaU+0YudVbMvqWy0iUfeUkKpC1C4FhUK/m5o6u19v4zpnUEOhjUTstmYIO/q6WoZuLiUUxafBS4wvth8PtUs0xYc2z2/w+i4hHYr2wWyErtrwWaGqzToQXv5NoFIKMdwAYUc+FsNapS+c/JfANavIcxeH+NOJp9gxZ9TfHVurKgiXbvPhktoBWhHrJF7iGLPvIf48+Bg5c+bEyZMn4enpaSkX9J6Hwk5vVOxIAiQgZwJyFnYHDx5ExYoVoUQ8NvlnR4mMpr1Q8HFclFDkGQyPpnWl1RJtzB6E//Hl3GXaHL/Bq2VD6fIEtLGI3flOaS05B/+ztimhKDUTntWLQ6mIQdzhrgjZcfb1E85Q1fobHiUzQHuuP14F7P3sSFqPJvDoPQBqccNTtPhXiFpV85MrVnohS9MJHl06J5xrvDcFrxYs0XP1TwnkHQavprUhneV/sQmh84ZBE/Z2NVPrXBKunabCKbUDFC82InTOSGgiEjkLmGs4vJrXkcbRPl2G0BmTP1uLV6suDdeeU+DspUwQggvmv9dfq0gFdbUpcC+dFwqFFtonqxA2/4/PHgPQi5URnRKOGfwMtYMC2kfzEDJ7ZpJuBsdptWj+v9vYdTME9erVw7p166AwY5qipLhMYZcUanyGBEhAdgTkKuyCgoJQpEgRBAcHY2KtTPihSGozs1MBWX+Ee9P2ULu+ESDPELOtOyKOX0tUNGgVHlD5Vodzzd5wTOf6OiWKFtrnmxG+aCRiXxmR9MzM3uozvNajFtw7D5NKXyEuGJrDUxF58QYUGVvApWZdqCL3Inx+f8R+Ib+adGas499wzSgOIUKqORphZF42rTIfXLougksaB2ijDyJicm/9658qUsDxu7lwK5FNWvGLf7AWEdsWISboFRQ+ZeFc82c4+6YCoi8iamU3RAUmfnNXm2MQvFs2ShB2r9YgdPLYzwo7QAlF4T/hWa98gli+tgSRR3ZDE6aAMm0JOBZvASe/dJJN2uCNCF8xFrEvvlxRQ59YJrlPwT/h1ejDJNNJG+1FlAaV519H4ItojBw5EqLOs5wahZ2cokFbSIAEkkxAjsIuJiYG5cuXx7Fjx9CqcCpMrZ05yf7p96ASKD0L3tWKQaF8u92YoEJiEXeqD0I2fZB8V/wqbWd4dhY3Pj+eRRt9CBFTelkp95h+XuvVK2MbuDXvCrWH+m0uP7FF+GQnIteNQkyQPrdE3qYOgUi7cn4awjduTNLKz1ubVVBWXALPCqIiQihi1jdA+NnnerkkhVWRGg5lB8KtTHkonZUJ5eDEGUSlEgptLOL/24rITRMRE/TpdCxv05cYkufODcpiv8CtSg2oXMXliHebFtqo+4g9PReR+7Yi3pi7GnqT+FxHL6i/Xw+3fF5A3HVEzWuJqIdG5PEBcOlxJKotuo4ojRZbtmxBzZo1TWKpKQahsDMFRY5BAiRgdQJyFHadO3fGnDlzUCSDKza3yQ7nxJST1cklHwO0Tr5Q568Ah1QpoIh9Bk3QUcTevAEYkKlEm6EeXHKrEXt2M+JeGFlw9TV6rWcDuHcbBEcXBbR3JuDV4pV6bse+jZ3WJRvUOb+ByictlA7x0IYHIe7eYcTcewjFF+6aKApPhlf9cgnn/E73QsiGj8vNfeot0TpnhkOOUnBIkw5KcRsoWuQZPI/YWxegjTEArBlfQ61XE3h0GwC1M4BbY/Fy6RqD+SZm3pqLL9Fp3R14e3vj1KlTyJYtmxm90H9oCjv9WbEnCZCAjAnITdgtWLAAHTp0QGo3B6myREZP816WkHFoaNoXCThCVXU5PMpkB+LvIXpZc0Te/kxemi+OZ0gHZzjUWw/3ommAmAuInNcO0XokSTZkBuv2dYaqxgp4lPID4m8jenFrRN41HdtBOx5g1vEnKFCggLQy7+pqzmTj+pGksNOPE3uRAAnInICchJ24LVemTBnEaWIR0CobymbxkDk9mmdtAlrnUnDtNFm66ICHCxCyYCbiLXAsTZvye3j++DMcnF4hdkdnhB25YW0UJp1fm6ETPNv9CAe1BvGn+yFk00GTrNa9MVITr0XD5bdw6G4omjdvjpUrV5rU/qQMRmGXFGp8hgRIQHYE5CLsnjx5Il2WePDgAUZWzYhu3/jIjhUNkimBzF3h0bo9HBxjEXfmF4Ru3G9SEfKh11qXonDxnwTn9DHQ7PsZ4fvPmnU+i1N3KwOXtuPhlMYReLgYoYunIi7qg7OnJjDqSXgMKs6/jochsZg4cSL69OljglGTPgSFXdLZ8UkSIAEZEZCDsIuLi0OVKlWwb98+NMznjfkN/WREiKbIn4BIUzMQ7o0awkEdCc3pEQjbshPaONMlZH6XgVadA86Nf4bqyu+IOHdb/ngMsFCbojJcvx8KpwxuwJMNCF82GrEvjbsw8bnpTz8Mx3eLbyAeSuzatQsVKlQwwFrTdqWwMy1PjkYCJGAlAnIQdj/99JP0jT1vGidsb5cbbuYuLWEl1pzWnASUgG9buDfsBAeRJ+6/zYjcNhsxgY/NOandjK1V+cChaGe4VqwDlTjudms+QgPmIy7M/Bc5lp59it6b78PHxwdnzpxBxozmLhmYeNgo7OzmdaYjJJC8CVhb2P31119o0aIFvJ1V2NMhF3xTOCXvgNB7owhoXfLCsUIPuBQtDqU6Frg1AyHLlxmZWsUok+T/cJrmcPPvAbWHE/DqFKIPTEPU6YsW3V7us+Uelpx5hmLFiuHQoUNwcrL83wEKO/m/qrSQBEhADwLWFHYXLlxAyZIlERUVif81y4qq2b30sJhdSEAPAm65of76O6jVxxCx/+MchHqMkGy6aJ1KwLlmWeDmNkRfuQjEmf483ZdgRsdpUWfJdZwKikD79u0xf/78Lz1i8t9T2JkcKQckARKwBgFrCbsXL15I385v376NgRXS4+ey6azhPuckARKQCYGgkBhUmn8NT8I1Uh7LTp06WdQyCjuL4uZkJEAC5iJgDWEXHx+P2rVr459//kGNnN5Y3tQXSpnVjTQXb45LAiTwaQJH7oWh3rKbUKocIGpFixV9SzUKO0uR5jwkQAJmJWANYTdkyBCpVmT2VE7Y0T4XvJ0SqcllVq85OAmQgFwJzD4RjF+3ByFDhgzSZYq0adNaxFQKO4tg5iQkQALmJmBpYbdhwwbUr18fbo5K7GyXE7l9XMztIscnARKwMQI/rg/E6gsvUK5cOezevRsODg5m94DCzuyIOQEJkIAlCFhS2F2/fl06VxcaGopFjf1QL4+3JVzkHCRAAjZGICI2HjUWXcPFx1Ho2bMnpkyZYnYPKOzMjpgTkAAJWIKApYRdWFgYSpQogStXrqBX6bQYWjmDJdzjHCRAAjZKIPBFNCovuIYXkXFYvnw5WrZsaVZPKOzMipeDkwAJWIqApYRd48aNsXbtWlTw88Dqltmg4mUJS4WY85CAzRLYcysETf66BScnZxw9ehSFChUymy8UdmZDy4FJgAQsScASwu6PP/7AgAEDkMnLEXs75kZKF16WsGSMORcJ2DKBiYceYdTe/+Dr64vTp08jZcqUZnGHws4sWDkoCZCApQmYW9jt3LkTNWrUgKMS2NY2BwqmE/WK2EiABEhAfwL+q29j89VXqF69OrZu3QqlUqn/w3r2pLDTExS7kQAJyJuAOYVdYGAgihYtiufPn2NWvcz4vmAqecOgdSRAArIkEBIdh2oLr+H602gMHDgQY8aMMbmdFHYmR8oBSYAErEHAXMIuMjISpUuXxrlz59CxeCr8USOzNdzjnCRAAnZC4PrTKFRbeB1C5AUEBKBBgwYm9YzCzqQ4ORgJkIC1CJhL2Pn7+2PZsmX4JpMb1rfODkeV6bdOrMWM85IACViHwOarL+C/OhDu7u44ceIE8uTJYzJDKOxMhpIDkQAJWJOAOYTdtGnTpNxT6dzV2NspF9K6qa3pIucmARKwIwKj9gZh4qFg5MyZEydPnoSnp6dJvKOwMwlGDkICJGBtAqYWdocOHULFihWh0MZhk392lMjobm0XOT8JkIAdEYjTavH9X7ew51aoVMVGbMsqTJA+yazCTu3kim/ajrajMNAVEiABuRI4OLsPXF1dER4ebrSJDx8+RP78+fHixQuMqf4V2hb1MXpMDkACJEACHxJ4GaVB5XnX8DA0FiNGjMDgwYONhmRWYWe0dRyABEiABAwgYCph17RpU6xevdqAmdmVBEiABIwncOHCBelLpTHNLMLuxo0bmD59ujF28VkSIAESMJiAk5MTxo0bZ/BzHz4wcuRIDBkyBBkyZIAYk40ESIAEzEngzp070u178YVS/N0xpplF2BljEJ8lARIgARIgARIgARJIGgEKu6Rx41MkQAIkQAIkQAIkIDsCFHayCwkNIgESIAESIAESIIGkEaCwSxo3PkUCJEACJEACJEACsiNAYSe7kNAgEiABEiABEiABEkgaAQq7pHHjUyRAAiRAAiRAAiQgOwIUdrILCQ0iARIgARLQl8C5qw8RGR2L3H5pkMLTRd/H2I8E7JYAhZ3dhpaOCQIvQiJx6eYjPAwOQZPqBU1SroVkSYAE5EPg8bNwNO69FKER0ZKwy5zOGyW/zozcfj6S2MuQxjT1N+XjMS0hgc8ToLDjG2I3BMIjonHpVjAu3Xwsibldx27qfBvdqwZqlcttN77SERIggbcENu27gsHTtieKxMPdCbl90yB31gShJwSfb4YUUKmUREgCdkmAws4uw2r/TkVFa3D1TjAu33qMyzcf4+LNxwh8+CJRx8sXz4opA+raPxR6SALJmECXkQE4eu6eXgScHFXIkUUIvddiL6sPcmRODSdHB72eZycSkDMBCjs5R4e2SQRiNXG4de8ZLtx8JIk4sSJ36/4zxMVr9SK0Y14HpEnprldfdiIBErBNAs9fRaBhr6V4GRolOSBW6cXPxBfAq7ef4E7Qc8R/5m+GSqmA71cp31vZy+XnA083Z9sEQquTLQEKu2Qbenk6Lv7wij/Ab7ZThZC7dvcpYmLjPmmwp5sT8mZPi7zZ0kr///fWf3Hy4n2p/7CuVVC/snEFleVJilaRAAl8SGD7kev4ZcJW6cdlivhi+qD6ui7RMRrcuPdUEnmS2LvzBDfuPkF0zKf/toiHxRm93FnTIJdvGuTJKrZyU/OLIl89WROgsJN1eOzfuPuPXiZspb7eUr18OxiRUbGfdNzZyQF5s6ZB3uzpkDdbGuTPng6Z03vr+h88HYgeY9ZL/16qUGbMGtzQ/iHSQxIgAR2B/pO2Yseh63p9sYuLi5eOcAiR92Zl79qdYISER3+WaMEc6bD092akTgKyJEBhJ8uw2LdRZy4HYd7a49K5uNCwT/8BVTuokCtLamkVLp/0v3TImjEllEpFooDCwqPRsPdSBD8Ph5uzGmsm+yO9j4d97BxtZQAAIABJREFUw6R3JEAC7xF4GRqJRr2X4dnLCLi7OGL1pNYG/x0Qt+jfrOq9EXzBz8N081QtlQPj+9UieRKQJQEKO1mGxb6NOnHhPjoNW/uek0KsZcuUCvlfb6mK/8+eOTXUapXeMIbP3Il1uy9J/Qd2rIjva3yt97PsSAIkYD8E9p+8jV6/b5QcKlkwE2YPaWh0qqMbd5+iSd/l0pg9W36Ldg2L2w8wemJXBCjs7CqctuGMWFkr4z9LMlZsrYrt0jxZfeDspE6yA8f+vYfOIwKk54vly4h5wxsZ/Yc8ycbwQRIgAasSiInVoESz6TobTPFF791jHjN/q4/ShX2t6iMnJ4FPEaCw47thFQL1ui/G3f9emuQcnDiT17j3MgQ9CZF8GdWzOmqXz2MVvzgpCZCAdQlcvR2MQVO34db955IhDg5K9GpVBq3rFDHKsLlrjmPmX0elMfYs7ISUXq5GjceHScBcBCjszEWW436WwKDJ/2DLwWsQyUMPLuliFK2gx6/Qd9xmXAt8Io2jUAAtaxVG9xbfSiuCbCRAAvZPQBMXj3lrTmD+muO6VEi5fH2kL3o5sqQ2GsBPf27G7qM34ZPCDTvndzR6PA5AAuYiQGFnLrIc97MEVmw+i/GL9kt9Ns34AZnSvb3ZmhR0ItfdwnWnMW/NMWg08dIQmdN5YWjXqiiaL2NShuQzJEACNkLg5r1n+G3qNul2q2giJ504A9epSUmIS1imaN91WSiVJixb1A/Tfq1niiE5BgmYhQCFnVmwctAvETh75SHa/rZK6vZHn5qoXibXlx7R6/c37z3FkOk7cPlWsK7/9zUKSlsxri6Oeo3BTiRAArZBQOS9XLThFGb/75iUyFw0v69SYnTP6tJtelO1kPAolPOfLQ3XsXEJdGte2lRDcxwSMDkBCjuTI+WA+hCIio5F6VYzpUzwresWwU9tyunzmF59RG6qxRtPY87fx3SJjb/y8cSQrlVQsmBmvcZgJxIgAXkTuPvwBYZO34Fz1/6TDBVHMFrVKYLuzUubvDSYSHjecWjCTf4/+9dGlZLZ5Q2H1iVrAhR2yTr81nVepA4QKQSK5v0KC0Y2Mbkxdx48x7AZO/Hv9YQ//KI1rJIfP7UpCzdXJ5PPxwFJgATMT0Cr1eJ///yLqcsOITJGI02YKa0XRvSohsJ5vjKLAcs2ncGExQeksbfOaidVo2AjAbkSoLCTa2SSgV1v8s65OKtxeFnXTyYeNgaFWBFcufUsZqw4ovsQEHVjh3SpjDJF/IwZms+SAAlYmIA44zZ0xk5dyUAxfdPqBdHHvyzE3xFztd+mbMPmA1dNctnLXDZyXBJ4Q4DCju+C1Qis3n4eo+fukeZfO7m1lKDYXE2ULhOrd6cvB+mmqFM+D/q3Kw8Pdxb5Nhd3jksCpiIQsOsiJizaj/DXJQfTpnLH8G7V8M3X5j9e0aj3Uil9SvH8maQcmWwkIGcCFHZyjo6d23bl1mM07/+X5OWI7tVQt2Jes3ostnBWbT+PyWIL5/WHQ2pvVwz6sTIqlshm1rk5uGUIxF2ZgO+qT8CFhB26hKbMjHarDmBUaV6esUwUTDvLkxfhEKv7h84E6gauVzEvfm5bHu5u5j9SERWtQelWM8xyHti0pDgaCSQQoLDjm2A1AuIWW+mWM6XbbOLm6sCOlSxii9jOGTF7F0S1ijet+rc5MaBDRaTwdLGIDZzEPAS0zy9jz56D2DRtCKYdCIZIfKPwqIeFd9fjhxTmmZOjmo/AlgNX8fuCvbqa0qm8XTGkcxWUL57VfJN+MPLF6/+h1cC/pZ+O7lUDtcrlttjcnIgEkkKAwi4p1PiMyQi06v8XLt56jPw50mH5781MNq4+A63bdRETlxxAaESM1F2Iul86VESNb3Pq8zj7yJhA9Hp/ZGy0DE/jAYcSo3HhyK/IbZp0ZjL22n5MexESidFzdmPXsZs6p6qVzoFfO1WCt4dlv3y9e2QkYEprZM1oviMj9hNBemJNAhR21qTPuTFm7h5pe1QkET2yoqvJkonqi/bxs1CMnL37vW2eSt9kx6COFZHK203fYdhPZgTCVjZC+lYBCNOqkKXLdlyfWRnciJVZkD5hzp7jN6X/JoW4E83L3RkDO1Wy2heukbN3Ye3Oi3BxdMDhFd3McsnLNiJDK22FAIWdrUTKTu3csOeSdMtNtP+Nb4HcWdNYxdNN+65g/MJ9CAmPluYXpc5+aVeBNWetEg1jJ43D6cFFUGrUecQqPFB/8V2s8+c+rLFUzf28SAI8bv4+6fbpm1aumB+Gdqli1S9ZLfqvlBKeF8yRDkstvKtgbuYc3z4JUNjZZ1xtxitRCqhxn2WSvYN+rIQm1QpazfanL8KlFcQ9J27pbBDlg377sQrSpuLqndUCY/DET7Cgti86bomAVl0SYy4cxsBc3Ic1GKMFHzhyNhDDZu5E8PPwhC9Wro7o17Y86lXKZxEromM0WLzhDFrXLvRehRpRf7ZUixnSOWCRVkVsBbORgNwJUNjJPUJ2bp/IMydunImbZw0q55Nqu77bAoOeS6toBXOmtxiJ7Yeu4fcF+3RbQeJDpm+bcmhQJb/FbOBERhCI3ooO2epiQVAcVL5dsf3aDFTmPqwRQM33aERkDCYsOSBtdb5pIn3JsK7VkC61u/kmfmfkyzcfY9DU7bgT9ByNqubH4M5VdL8VCdRFInXRhnSpIiU4ZyMBuROgsJN7hJKBfe1+W4UzVx4iZ5bUWDWxlc5jURqszaBVuHTzkVQqqFuz0nB2crAIkeevIvD7wn3Ycej6ex844kYes85bJAR6TaIJf4wHD14gxiUNfDOnlM7RxZ4fipLFRuBsrALuDZbgbkBrpNRrNHayJIFTlx5IdZ3FLXXRxH/bff3Lokn1glCI+mBmbmIVbu7q41gYcBJx8VrdbDN/q4/ShX2lf9+07zIGT9sh/fNf45ojTzbT1Z81s3scPhkToLBLxsGXi+t/Lj6A5ZvOQKVU4PDybjrxtmTDaUxaelBnZpb03tKKXpG85ikblBgPcZBbJFF+9jJC+rWbsxo9W5eRtmUs8eEjlxjJyo74JzixdALGzfoLO88+QEhsPKBQwjldIdT8oR96pVmK7/puQ4TWEd+MPY9DA3KBG7HyiaBYnZ+64hBWbjmnM6pIngwY3r0aMqXztoih1wOfSILtWuCTj+YTlWkCJreWcuSNW7hPstNBpcTRFd2gVvNNskiAOIlRBCjsjMLHh01BYOvBq/h18jZpqCVjmuLrXBkgCnyLLZCY2LiPpmhW82v0alXGrCWE3p00NCwK4xbux6b9V3Q/LpYvI4Z2rWKxDyJTcLaHMeKD92BkS3+M2R2EGDjAO3sZVC2XD+kdQ3Bt/ybsuvIKUCoRHxcHqHzRdedVTK9o/iS29sDWEj6InHC/TduBwIcvpOkc1Sp0b15aWpFXKs2/Sid2ARasOymt1Gk0IsshkC1TSozuWUOqKT123l7pZ6Iqzcie1dF+8GqpWs2HuwmWYMU5SCCpBCjskkqOz5mMwL3/XqJu98XSeKLEV7OahdB+8Cqcvfqf9LPZQxri8q3HmPX3MekQs2hiO1TclitZ0PzlhN44euB0IEbNfnvAW6Q/6NayNFp8V9giH0omA26jA2mf70b/qg0x8UwItOqvUGP4Ciz8uTzSvdmdDzuMX0pVwPiLGoiNNYVnAyy5G4DWllkEslGqljE7NjYOs1Ydw+L1p6QKDqLlz5YWI3pWs1heuDsPnmPItO24cPOxNL/Y7W1Tryi6NisFR7UDRGWaziMCcPz8fen3E/vXxpBpOxAWGQNR6UKsKLKRgC0QoLCzhSglAxvL+c+SLknULpdbOscyftF+yWtx3mbQ65to4g+zqPcqvlm/aeIw809tysLN1TKrMmHh0dJh73W7L+lsKJQrPYZ1qwrfr3iSy2yvavw9LGtaCm0DHiJOkQLlx+7G1v6F4frehDE40DsfKk+5CVFRzOGbsbh0aABycvfMbGHRZ2Cx7SkuJ4iLCKKJbc2OTb5B+4bFpH82dxNCcsXmM5i28ohuB0Ac6xjRo5q0O/Bue/Q0DI17L5XE3LtNfOFsUauwuU3l+CRgEgIUdibByEGMJSC+KYsSX+KcnaODCpExGqT38cDaSa3fSz8g/ZHechbTVx5GdEzC6p0oBi5urH37+sCzsbbo87ywdfisnfjvSajUXWwp/fj9N/ihblGoLPBhpY+N9tMnHsFr/FGk2QoExSnhXXECju7snUgliRgc/6UQyoy7Ag1U8Ou+E1enVWRiYiu9CGLbc37AKcxbfQwibYhoObKkxqge1ZDLzzL5Kh88eoUh07dLl7PetObfFUKvVt/C2UmdKJn1uy9i2Mxd7/1u0agmKJzHcmd7rRQyTmsnBCjs7CSQtu7G1BWHpdtp77ZZQxqg1NdZEnVNbN+KwuDi/MubVrdCHvRrVx6ebs4WwSFSNUxedkiqnPGmie2lYd2rInvm1BaxIVlMormAkaWKYeipGGgdsqP7jnOYVjGxvIJR2NIhG+oueIh4hScaLL2LgFbch7XGO/Lhtqc4P/dD/WLo0vQbi11AWLXtX0xcelBKpSTaVz6eGNqtKkoUyPRFJD3GrMfB04G6fkeWd33vC+YXB2AHErAiAQo7K8Ln1G8J7Dp+E/3Gbdb9ILGcdh/yEmdi/t52HlOWH0JkVKz069TervitcxVUsGCRcJG2YfiMnbj/+JVkg4ODEh0bW26ryd7fo5j9vZCvylTc1CjgWGgIjp8chkKJZb2Jv4tJFXOi74EYQF0Kv188iF+4D2vR1yOxbU/fDCmkVbr8FspFKbZTh83cIe0AvGniyEa/H8rpLc6evAhH1Q7zpMfFtu2G6T9YlCMnIwFjCFDYGUOPz5qMgKjZWr3TAt14h5d10fvcXFBwCEbM2qk79CwGqVEmFwZ0qGCxguFR0bGYvvKItE2sfZ0SK5evD0Z0r2qxbSeTBUNWA0VjX8+8qDztNuLhiAKDjuDMqKJINJth2Bq09P0eK5/FQ+XXHbuuTEMFyxy9lBUxaxmT2LZny1qF0aOl2Pa0TP5JUaLwz0X7ERqRcEYuTUo3KUVSUo5pbDlwFYOmbEO1Mjkxrs931sLKeUnAYAIUdgYj4wPmIlC5/VwpX9z0QfVQpoifwdOs3XkBk5Yc1B18TuHpggEdK6J66ZwGj5XUB/699lC64HEnKCGdgzgz2K5hcXRrXjqpQybv5zQXMKxkEQw/o4FClREdt9zEnOqJq7XYA32Qt/JkaWXPq8EyBAa0BDdiLfP6iGMJ1TsvQGhYQq1l8d/e+H61INICWaKJcoAjZu/CgVN3dNPVKp8bv7SvYNTRjL5/bEK+nOnQvkFxS7jBOUjAJAQo7EyCkYOYgkCPMRuQwsMZI3pUT/JwYuVv5OzdOHTm7fmYyqWy49cOFS1WSFzUnazScZ7uQ06c/TPGpyTDsIcHQ1ei0VetEBCqBRwrY3rgDnRLn9hNyhgc/qkAKky8Dg0cUfr3SzjwS3YmJrbgOyDOyIqzsqJZMj3IP6IE4Ly9eBUWpROVgztXRqWS2Y32XlSgefQkFHmzs+KE0TA5gMUIUNhZDDUn+hKBgF0XUfWb7PBwN/7ygygFNH7hfimFimhe7s74uV151C6f50tmGP17IS4b9VomrRyKM3/rpvibxCejDbPBAeKujELprwfjRCyg9GmDdfcWo25ir8fL9fghfyMsCYqHQpUV3XddxlTuw1o04uIWbNtBq3D+xiNp3rnDGul1USGpRr4MjcSYuXuw48gN3RBVvsmOQT9WllYM2UgguRKgsEuukU8GfosD0KPn7sa+E7d13pYr5offfqwMUTbIXK376PW6FcPJA+pa9CKHuXyy1rgxJwag8Ld/4LJGFJLoiu3XZqCyKAj7XovCqaFlUHbkaURpExITL78bgBbch7V42ETFmKY/LZdSESWWrshUBu09cUtamRcraqJ5uDthYIeK+K5sblNNwXFIwGYJUNjZbOhouL4ExFbNH/P34mVowlaNh6sTfvqhLOpXzq/vEHr3e7douLjA8Xufmno/y44fE9Cc+g1FSo/GhVhA9VUHbL41DzU+OGIXfnQIyteYiUDlCzx7GQ910RE4e2ww8lnmvD7D9gGBFZvP6hKM63O73RCAIeFRGL/g/fJ+ZYv6SXksfVIklgLHkNHZlwTsgwCFnX3EkV58gYD4Zv/7/L3vbduUKpQFQzpXkVYWTNHEAe6GvZZK279iK2jdVH+L3co1hf1yHCM+aBqq+PXC3lgtFM5VMOP2dnR554yd5s4K+FftjLOVuiDvhgkICNbCvcFSPAxoBRFVza1l6Ds+DO0ndMHX/Ny3SIhFGqKOQ9dCpAESbdqv9VG2qK/Rcx85d1e6mBT8PEway93FUcpbWb9SPqPH5gAkYE8EKOzsKZr05YsEdh+7gdFz9+q2cNyc1ejtXxaNqxWAQhSPNKL1GbcJe4/fkkb446fvLHob1wiz5f1ozHH0/7oMxl/VAAo3lBl9FLsHFpCqSbw6vwi9mvXAhhS/YNsSbwzM2xN7YxVwrTkXgVs7wOvOanSr8wOWRjbEkn1L0CyT+ctXyRum5awTKYga91km5Zc09pypuHEryvit3XlR50DJgpkwrGs1k30psxwZzkQC5idAYWd+xpxBZgRCQqMwbuE+bD5wVWdZ8fyZMLRLFWRM55Uka8V278BJ/0jPilu4E/rVTtI4fOhDAnE4P6o0Sg05gQhxfk7ljRylSyJz7B2cPX0TkTk7YsX2majvvgINM7XBulAtFOp0+LpMDkScPYI7bjUx6Z+/0a3A+1Vlydn8BNbsuIBRc3ZLE9Usmwtjext+LEGs+g2ZvgMPg0OkcUQ+vD7+ZdG0ekGjv4iZnwBnIAHrEKCwsw53zioDAiLn1ag5uxD8PFz3odGjxbcQtSRFCSR9m9jmFVuw4gyfuH0bMKW1xVKr6GujTfcLP4uJjb7DgB2PEPs6+TMULshacyDmLhiEyumUQPwTrGtfFN8vuZ/QR6GCd8E2mLxyBtrkNf6WtU3zs6LxXUauw9H/30IV7c9+tVClVA69rBFlwKauOISVW87p+hfOnQEjelRDpnS8FaMXRHZKtgQo7JJt6Om4IBAWHo0/Fx/A+j2X3vkASY9h3aohS4YUekHqP2GL7uzeqJ7VLZJSRS/D7KlT3GOcCvgLm07eR6RLRhSq2hANy2TBe5It+ha2z1uC3UEOyFikBr5vUAJpeYHCqm/B42fhaNx7KUIjEs6drp3cGim9Pr96ev76fxg8dTvu/vdSst1RrUL35qXRqk4Rg75wWdVxTk4CViRAYWdF+JxaPgSO/nsXI2btwn9PQnUfJl2blULrOkWgUn36bNa7NW7F7bxpv9aTj1O0hARkQGDTvisYPG27ZEmlEtkw8Zc6iVoVGxuHmX8fxZINpyFqzoqWP1tajOhZDVkzppKBJzSBBGyDAIWdbcSJVlqAgDikPWnZIazefl43W4H/zzg/rHs1ZMv08QeLOKvXoPdSqQyah6sj1k7xN2t+PAsg4BQkYBYC714sGt2rBmqVez/f3NXbwfht2nbcvPdMmt/BQYlOTUpKpbw+98XKLMZyUBKwcQIUdjYeQJpvegInL97HsJk7EfQ44cD2mw+Zdg2Kw+Gd1btBU7dhy/6ECxjDulYxS14803vHEUnA8gTePYfq6eaENZNbS1+CNHHxmLfmBBasPSH9s2g5sqTG6J7VkdPXx/KGckYSsAMCFHZ2EES6YHoCUdGxUt3Lv7aeg/b1gf1cfj4Y2b2a9IFz4HQgeo5ZL00s8uHNGtzA9EZwRL0IzN0UiG/zp0Q+P0+9+rOTdQjsOHoD/f/cIk3+bRFf9GldVtqivXI7WPqZuLAkvjz92KQk1GqVdYzkrCRgBwQo7OwgiHTBfATOXnmI4TN3IvDhC2kSsWLXpHpBbDt0DS9CIiHy4K2Z7M98WuYLwRdHrvrTYRTM5okJXQt8sS87WJfALxO3Yvvh6x8Z4fdVSozsWQ35s6ezroGcnQTsgACFnR0EkS6Yl0B0jAaz/j6KpRvP6A51v5lxUKdKktBjsx4BIey+zuaFP7uavkSc9byyz5nFudSGvZfi6cuEGq8iJ3jL2kXQo0VpODnyCrN9Rp1eWZoAhZ2liXM+myVw8eYjDJ2+A7fuP5d8EEmN5w5ryESpVo4ohZ2VA2Dg9CJ/ZM+xG5AprZeUl65wnq8MHIHdSYAEPkeAwo7vBwkYQECkZJi75jj+t/Uc/hrfMsmVKgyYkl2/QIDCzvZekc37r6ByyexwcVbbnvG0mARkToDCTuYBonnyJBASHgVPN1Y0sHZ04uK0qNH/CApm9cSEbjxjZ+14cH4SIAHrE6Cws34MaAEJkEASCTwIjkTbP84gv58nJnWnsEsiRj5GAiRgRwQo7OwomHSFBJIbgWOXnmPwwivImckdM3p/ndzcp78kQAIk8BEBCju+FCRAAjZLYO3+IMzeGAhPFwesHVXSZv2g4SRAAiRgKgIUdqYiyXFIgAQsTmDqmls4duU5nryMwdoRJeDpxsP4Fg8CJyQBEpAVAQo7WYWDxpAACRhC4KcZF5AlrSs2HX2Eid3yo0BWL0MeZ18SIAESsDsCFHZ2F1I6RALJg8CTl9FoPeoUxnbOhyX/3EO2DG7o0Shb8nCeXpIACZDAJwhQ2PHVSHYETl59gYVb7yJWk1B0nM02CbwIi4WHswMWDSyC7SeDMeHvm8iS1sU2naHVOgJCoA9omZOJv/lOkEASCVDYJREcH7NdAt0n/wtXZxVK5Utpu07QcumDv1hOb2RM4wJNXDwOnn+Gl2GxJGPDBKI08Vi05S7GdsqHojm9bdgTmk4C1iNAYWc99pzZCgTuPo5Ah3FnMbdfIfild7OCBZySBEjgcwSGLrwCZ0clBrbKRVAkQAJJIEBhlwRofMR2Cew8HYw5GwOxZngJ23WClpOAHRNYf/AhNh5+hIUDitixl3SNBMxHgMLOfGw5sgwJLN52F+duvMLkHgVlaB1NIgESEGdghyy4gs1/lIJKqSAQEiABAwlQ2BkIjN1tm8CoZdfgrFaiX7Mctu0IrScBOyXw37Mo+I85jSUDiyBDal6GsdMw0y0zEqCwMyNcDi0/Al0mnkPZQqnRolJG+RlHi0iABBAXr0WtAUcxom0elMiTgkRIgAQMJEBhZyAwdrdtAnV/PY6fm2VD2YKpbdsRWk8Cdkyg/R9nUKtUOjQsl8GOvaRrJGAeAhR25uHKUWVIICY2XloJmNy9APL5ecrQQppEAiQgCPw86yLy+nqibc3MBEICJGAgAQo7A4Gxu+0SCI/SoP6g45jeuyByZfKwXUdoOQnYOYGBcy9J6Yg61fG1c0/pHgmYngCFnemZckSZEngVpkHjoccx56dCyJqBOexkGiaaRQIYsuAy0qVyRtf6WUmDBEjAQAIUdgYCY3fbJfD0VTSajziFhf0LI1NaV9t1hJaTgJ0TGL74ClJ6OLL2r53Hme6ZhwCFnXm4clQZEhBF47tO/hfTexZE2pTOMrSQJpEACQgCU1bfhIuzCp3q+BEICZCAgQQo7AwExu4kQAIkQAIkQAIkIFcCFHZyjQztIgESIAESIAESIAEDCVDYGQiM3UmABEiABEiABEhArgQo7OQaGdpFAiRAAiRAAiRAAgYSoLAzEBi7kwAJkAAJkAAJkIBcCVDYyTUytIsESIAESIAESIAEDCRAYWcgMHYnARIgARIgARIgAbkSoLCTa2RoFwmQAAmQAAmQAAkYSIDCzkBg7E4CJEACJEACJEACciVAYSfXyNAuEiABEiABEiABEjCQAIWdgcDY3XYIREREYMaMGQgODrYdo2kpCZDARwQUCgXq1KmDsmXLkg4JkMAXCFDY8RWxWwIrVqxAq1at7NY/OkYCyYlA/vz5ceHCheTkMn0lgSQRoLBLEjY+ZAsEFi5ciPbt2yN7lpQY0beiLZhMG0mABD4gEBurQZt+G+Dr64s7d+6QDwmQAFfs+A4kVwJvhN3gHuUwom+l5IqBfpOATROIjY2DY86RFHY2HUUab0kCXLGzJG3OZVECFHYWxc3JSMAsBCjszIKVg9oxAQo7Ow5ucneNwi65vwH03x4IUNjZQxTpgyUJUNhZkjbnsigBCjuL4uZkJGAWAhR2ZsHKQe2YAIWdHQc3ubtGYZfc3wD6bw8EKOzsIYr0wZIEKOwsSZtzWZQAhZ1FcXMyEjALAQo7s2DloHZMgMLOjoMrB9f69euH69evQ6PRwNXVFS1atEDDhg0/a9qGDRuwdOlSREZGwtHREc2bN8f3339vsDsUdgYj4wMkIDsCFHayCwkNkjkBCjuZB8jWzUudOjWePXumc0OIu8uXLyNLliyJunb37l3ky5cP4eHhut93794d06ZNMxgFhZ3ByPgACciOAIWd7EJCg2ROgMJO5gGydfOmTJmC3r17v+eGKA20cePGRF2rW7cuNm3apPudu7s7Ll26hMyZMxuMgsLOYGR8gARkR4DCTnYhoUEyJ0BhJ/MA2bp5cXFxKFmyJE6fPv2eK2K7VYi4d9u6des+2qadNGnSR8JQXyYUdvqSYj8SkC8BCjv5xoaWyZMAhZ0842JXVp05cwYlSpSAEHlvmtiKFVuyYmtWtNDQUOTNmxcPHjzQ9SlatCiOHz8OlUqVJB4UdknCxodIQFYEKOxkFQ4aYwMEKOxsIEj2YGLfvn0hVt/ebQMGDMDYsWOlH/Xp0weTJ0/W/VqIuRMnTqBIkSJJdp/CLsno+CAJyIYAhZ1sQkFDbIQAhZ2NBMrWzQwLC5NW5O7fv69zRa1W49y5c4iKivpoRU8IwQkTJhjlNoWdUfj4MAnIggCFnSzCQCNsiACFnQ0Fy9ZNFRcm6tWr954bZcqUkYTdqVOndD8XFyXENq2bm5tRLlPYGYWPD5OALAhQ2MkiDDTChghQ2NlQsOzB1EaNGiEgIOCIjbcTAAAE4UlEQVSzrmzevBm1atUy2l0KO6MRcgASsDoBCjurh4AG2BgBCjsbC5itmxsUFIQ8efJIlyUSa02aNMGqVatM4iaFnUkwchASsCoBCjur4ufkNkiAws4Gg2brJk+dOhW9evX6yA1RZSIwMBDp06c3iYsUdibByEFIwKoEKOysip+T2yABCjsbDJqtmzxnzhx07tz5IzfETdiLFy8id+7cJnGRws4kGDkICViVAIWdVfFzchskQGFng0GzZZMfPXokbcW+fPkyUTfKlSuHffv2QaFQGO0mhZ3RCDkACVidAIWd1UNAA2yMAIWdjQXM1s0VZ+jWrFnzWTfmz5+P9u3bG+0qhZ3RCDkACVidAIWd1UNAA2yMAIWdjQXMls1NLN3JsGHDpMTFr1690rmWMmVKXL16FT4+Pka5S2FnFD4+TAKyIEBhJ4sw0AgbIkBhZ0PBsmVTQ0JCpATF4lbsm1asWDGpZNiMGTPQs2fP99xr1aoVli1bZpTLFHZG4ePDJCALAhR2sggDjbAhAhR2NhQsWza1W7dumDlzps4FpVKJY8eOoXjx4lINWSHyRBWKd9vOnTtRpUqVJLtNYZdkdHyQBGRDgMJONqGgITZCgMLORgJly2YeOXIEosKEVqvVudGpUyeI27Fv2tGjR/Htt9++1yd79uy4cOECnJ2dk+Q+hV2SsPEhEpAVAQo7WYWDxtgAAQo7GwiSLZsYExODwoULSyXC3rRUqVLh+vXrEGfp3m3iwoQQY++2QYMGYdSoUUlC8EbY/da9LEb0rZSkMfgQCZCAdQkIYeeUaxR8fX1x584d6xrD2UnABghQ2NlAkGzZxOHDh0NckHi3zZ07Fx07dvzIradPnyJXrlx4/vy57ndqtVraohXn8wxtixYtQrt27Qx9jP1JgARkSMDPzw+3b9+WoWU0iQTkRYDCTl7xsCtrxLdrkWxYrNq9aSVKlJDO1n0qT11iyYvFOTtx3s7Qdu/ePdSoUeOT5csMHY/9SYAErENA/L0QK/pDhw61jgGclQRsiACFnQ0Fy9ZM3bZtG2rWrKkz29XVFYcOHZK2Zj/V4uPjUalSJezfv1/XJUuWLFKpMTYSIAESIAESIIHPE6Cw4xtiVgJiW1WkOhHlwkReOn0uQohLFo8fP0ZkZCRE/di0adPCwcHBrHZycBIgARIgARKwBwIUdvYQRfpAAiRAAiRAAiRAAgAo7PgakAAJkAAJkAAJkICdEKCws5NA0g0SIAESIAESIAESoLDjO0ACJEACJEACJEACdkKAws5OAkk3SIAESIAESIAESIDCju8ACZAACZAACZAACdgJAQo7Owkk3SABEiABEiABEiABCju+AyRAAiRAAiRAAiRgJwQo7OwkkHSDBEiABEiABEiABCjs+A6QAAmQAAmQAAmQgJ0QoLCzk0DSDRIgARIgARIgARKgsOM7QAIkQAIkQAIkQAJ2QoDCzk4CSTdIgARIgARIgARIgMKO7wAJkAAJkAAJkAAJ2AkBCjs7CSTdIAESIAESIAESIAEKO74DJEACJEACJEACJGAnBCjs7CSQdIMESIAESIAESIAEKOz4DpAACZAACZAACZCAnRCgsLOTQNINEiABEiABEiABEqCw4ztAAiRAAiRAAiRAAnZCgMLOTgJJN0iABEiABEiABEjg/wA7RY7tzfZzFwAAAABJRU5ErkJggg==)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Pt7Nr6a7tItO" + }, + "source": [ + "Fine-tuning a model involves updating its weights (also called parameters). LLMs have a lot of weights. The Gemma 2 2B that is being used in this notebook has 2,617,270,528 parameters!\n", + "\n", + "Changing all of them can take quite some time and requires a lot of resources.\n", + "\n", + "To mitigate this issue, you are going to use a technique called: [LoRA: Low-Rank Adaptation](https://arxiv.org/abs/2106.09685)\n", + "\n", + "This technique, in summary, helps lower the number of trained weights needed by a lot, making fine-tuning more accessible.\n", + "\n", + "The key parameter used is the `rank`. In this notebook it set to 4 but you can use higher numbers to get better results but, of course, needed more resources.\n", + "\n", + "**TIP**: Train your model with lower ranks and evaluate the performance improvemnet on your task. Gradually increase the rank in subsequent trials and see if that further boosts performance." + ] + }, + { + "cell_type": "code", + "source": [ + "!pip install peft\n", + "from peft import get_peft_model, LoraConfig, TaskType\n", + "import torch.nn as nn\n", + "\n", + "lora_config = LoraConfig(\n", + " task_type=TaskType.CAUSAL_LM,\n", + " r=lora_rank, # Using your predefined lora_rank\n", + " lora_alpha=32,\n", + " lora_dropout=0.1\n", + ")\n", + "gemma_lm = get_peft_model(gemma_lm, lora_config) # Enable LoRA for the model\n", + "\n", + "print(gemma_lm) # Hugging Face models don't have a summary method; use print() instead\n", + "\n", + "tokenizer.model_max_length = token_limit # Set token limit in the tokenizer\n", + "\n", + "from transformers import AdamW\n", + "\n", + "optimizer_grouped_parameters = [\n", + " {'params': [p for n, p in gemma_lm.named_parameters() if not any(nd in n for nd in [\"bias\", \"LayerNorm.weight\"])], 'weight_decay': 0.01},\n", + " {'params': [p for n, p in gemma_lm.named_parameters() if any(nd in n for nd in [\"bias\", \"LayerNorm.weight\"])], 'weight_decay': 0.0}\n", + "]\n", + "optimizer = AdamW(optimizer_grouped_parameters, lr=lr_value) # Use AdamW optimizer\n", + "\n", + "\n", + "loss_fn = nn.CrossEntropyLoss() # Define the loss function\n", + "\n", + "def forward_pass(input_text):\n", + " inputs = tokenizer(input_text, return_tensors=\"pt\", max_length=token_limit, truncation=True)\n", + " outputs = gemma_lm(**inputs, labels=inputs[\"input_ids\"])\n", + " loss = outputs.loss\n", + " return loss\n", + "\n" + ], + "metadata": { + "id": "YQiQxLFKfyzx", + "outputId": "5a5b1398-61cd-48ba-d354-71e4eee37213", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Collecting peft\n", + " Downloading peft-0.13.2-py3-none-any.whl.metadata (13 kB)\n", + "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from peft) (1.26.4)\n", + "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from peft) (24.1)\n", + "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft) (5.9.5)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from peft) (6.0.2)\n", + "Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.10/dist-packages (from peft) (2.4.1+cu121)\n", + "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from peft) (4.44.2)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from peft) (4.66.5)\n", + "Requirement already satisfied: accelerate>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from peft) (0.34.2)\n", + "Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from peft) (0.4.5)\n", + "Requirement already satisfied: huggingface-hub>=0.17.0 in /usr/local/lib/python3.10/dist-packages (from peft) (0.24.7)\n", + "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.17.0->peft) (3.16.1)\n", + "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.17.0->peft) (2024.6.1)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.17.0->peft) (2.32.3)\n", + "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.17.0->peft) (4.12.2)\n", + "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft) (1.13.3)\n", + "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft) (3.4.1)\n", + "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft) (3.1.4)\n", + "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (2024.9.11)\n", + "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (0.19.1)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.13.0->peft) (3.0.1)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.17.0->peft) (3.4.0)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.17.0->peft) (3.10)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.17.0->peft) (2.2.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->huggingface-hub>=0.17.0->peft) (2024.8.30)\n", + "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13.0->peft) (1.3.0)\n", + "Downloading peft-0.13.2-py3-none-any.whl (320 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m320.7/320.7 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hInstalling collected packages: peft\n", + "Successfully installed peft-0.13.2\n", + "PeftModelForCausalLM(\n", + " (base_model): LoraModel(\n", + " (model): Gemma2ForCausalLM(\n", + " (model): Gemma2Model(\n", + " (embed_tokens): Embedding(256000, 2304, padding_idx=0)\n", + " (layers): ModuleList(\n", + " (0-25): 26 x Gemma2DecoderLayer(\n", + " (self_attn): Gemma2SdpaAttention(\n", + " (q_proj): lora.Linear(\n", + " (base_layer): Linear(in_features=2304, out_features=2048, bias=False)\n", + " (lora_dropout): ModuleDict(\n", + " (default): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (lora_A): ModuleDict(\n", + " (default): Linear(in_features=2304, out_features=4, bias=False)\n", + " )\n", + " (lora_B): ModuleDict(\n", + " (default): Linear(in_features=4, out_features=2048, bias=False)\n", + " )\n", + " (lora_embedding_A): ParameterDict()\n", + " (lora_embedding_B): ParameterDict()\n", + " (lora_magnitude_vector): ModuleDict()\n", + " )\n", + " (k_proj): Linear(in_features=2304, out_features=1024, bias=False)\n", + " (v_proj): lora.Linear(\n", + " (base_layer): Linear(in_features=2304, out_features=1024, bias=False)\n", + " (lora_dropout): ModuleDict(\n", + " (default): Dropout(p=0.1, inplace=False)\n", + " )\n", + " (lora_A): ModuleDict(\n", + " (default): Linear(in_features=2304, out_features=4, bias=False)\n", + " )\n", + " (lora_B): ModuleDict(\n", + " (default): Linear(in_features=4, out_features=1024, bias=False)\n", + " )\n", + " (lora_embedding_A): ParameterDict()\n", + " (lora_embedding_B): ParameterDict()\n", + " (lora_magnitude_vector): ModuleDict()\n", + " )\n", + " (o_proj): Linear(in_features=2048, out_features=2304, bias=False)\n", + " (rotary_emb): Gemma2RotaryEmbedding()\n", + " )\n", + " (mlp): Gemma2MLP(\n", + " (gate_proj): Linear(in_features=2304, out_features=9216, bias=False)\n", + " (up_proj): Linear(in_features=2304, out_features=9216, bias=False)\n", + " (down_proj): Linear(in_features=9216, out_features=2304, bias=False)\n", + " (act_fn): PytorchGELUTanh()\n", + " )\n", + " (input_layernorm): Gemma2RMSNorm((2304,), eps=1e-06)\n", + " (post_attention_layernorm): Gemma2RMSNorm((2304,), eps=1e-06)\n", + " (pre_feedforward_layernorm): Gemma2RMSNorm((2304,), eps=1e-06)\n", + " (post_feedforward_layernorm): Gemma2RMSNorm((2304,), eps=1e-06)\n", + " )\n", + " )\n", + " (norm): Gemma2RMSNorm((2304,), eps=1e-06)\n", + " )\n", + " (lm_head): Linear(in_features=2304, out_features=256000, bias=False)\n", + " )\n", + " )\n", + ")\n" + ] + }, + { + "output_type": "stream", + "name": "stderr", + "text": [ + "/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:591: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n", + " warnings.warn(\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hQQ47kcdpbZ9" + }, + "source": [ + "Note that enabling LoRA reduces the number of trainable parameters significantly.\n", + "\n", + "From 2,617,270,528 to **2,928,640**\n", + "\n", + "To monitor the learning progress, you will evaluate the model at the end of each epoch and save the lora weights." + ] + }, + { + "cell_type": "code", + "source": [ + "import torch\n", + "import os\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Define a custom callback-like function to handle actions at the end of each epoch\n", + "class CustomCallback:\n", + " def __init__(self, model, lora_name, lora_rank, text_gen):\n", + " self.model = model\n", + " self.lora_name = lora_name\n", + " self.lora_rank = lora_rank\n", + " self.text_gen = text_gen # text_gen function for evaluation\n", + "\n", + " def on_epoch_end(self, epoch):\n", + " # Save LoRA weights at the end of each epoch\n", + " model_name = f\"./{self.lora_name}_{self.lora_rank}_epoch{epoch+1}.lora.pt\"\n", + " self.model.save_pretrained(model_name, token=access_token) # Save model with LoRA weights locally\n", + "\n", + " # Evaluate the model using text generation\n", + " print(f\"Epoch {epoch + 1} finished. Running evaluation:\")\n", + " self.text_gen(\"Write a title\")\n", + " self.text_gen(\"Write a poem\")\n", + "\n", + "# Assuming train is your DataLoader and gemma_lm is your model\n", + "callback = CustomCallback(gemma_lm, lora_name, lora_rank, text_gen)\n", + "\n", + "# Training loop with callback-like behavior\n", + "losses = []\n", + "for epoch in range(train_epoch):\n", + " epoch_loss = 0\n", + " for batch in train: # Assuming `train` is a DataLoader or similar iterable\n", + " optimizer.zero_grad()\n", + "\n", + " inputs = tokenizer(batch, return_tensors=\"pt\", max_length=token_limit, truncation=True, padding=True)\n", + " labels = inputs[\"input_ids\"]\n", + " outputs = gemma_lm(**inputs, labels=labels)\n", + " loss = outputs.loss\n", + "\n", + " loss.backward()\n", + " optimizer.step()\n", + "\n", + " epoch_loss += loss.item()\n", + "\n", + " losses.append(epoch_loss / len(train)) # Store average loss per epoch\n", + "\n", + " # Run custom callback at the end of each epoch\n", + " callback.on_epoch_end(epoch)\n", + "\n", + "# Plot training loss over epochs\n", + "plt.plot(losses)\n", + "plt.xlabel(\"Epoch\")\n", + "plt.ylabel(\"Loss\")\n", + "plt.title(\"Training Loss Over Epochs\")\n", + "plt.show()" + ], + "metadata": { + "id": "YKpmDIfXh1Kx", + "outputId": "2422fc27-260d-4e7d-ce4b-bfa4234bec7d", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n" + ] + }, + { + "output_type": "stream", + "name": "stdout", + "text": [ + "Epoch 1 finished. Running evaluation:\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "O Cortiรงo\n", + "TOTAL TIME ELAPSED: 7.93s\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a poem\n", + "model\n", + "\n", + "TOTAL TIME ELAPSED: 4.38s\n", + "Epoch 2 finished. Running evaluation:\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "A Relรญquia\n", + "TOTAL TIME ELAPSED: 8.04s\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a poem\n", + "model\n", + "O Primo Basรญlio\n", + "TOTAL TIME ELAPSED: 8.19s\n", + "Epoch 3 finished. Running evaluation:\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "O Primo Basรญlio\n", + "TOTAL TIME ELAPSED: 8.21s\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a poem\n", + "model\n", + "O Primo Basรญlio\n", + "TOTAL TIME ELAPSED: 8.18s\n", + "Epoch 4 finished. Running evaluation:\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "A Sibila\n", + "TOTAL TIME ELAPSED: 7.24s\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a poem\n", + "model\n", + "O Primo Basรญlio\n", + "TOTAL TIME ELAPSED: 8.21s\n", + "Epoch 5 finished. Running evaluation:\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "A Sibila\n", + "TOTAL TIME ELAPSED: 7.31s\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a poem\n", + "model\n", + "O Primo Basรญlio\n", + "TOTAL TIME ELAPSED: 7.77s\n" + ] + }, + { + "output_type": "display_data", + "data": { + "text/plain": [ + "
" + ], + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTlUlEQVR4nO3deVwTZ/4H8E/CEQ5JAJVLIqhYUFFUPADPVq1V60pr16MqaD22W+zqtt2ubH+97Hbpsd3eq1arVK31rLr1xgNPqIpi0aotlVMIqBzhPpL5/YGkTYFwCEwSPu/Xa15tJs8k34ch5sPMM89IBEEQQERERGQmpGIXQERERNSaGG6IiIjIrDDcEBERkVlhuCEiIiKzwnBDREREZoXhhoiIiMwKww0RERGZFYYbIiIiMisMN0RERGRWGG6IjMT8+fPh7e3dom3feOMNSCSS1i2IqBG1v3d3794VuxQiPQw3RI2QSCRNWmJjY8UuVRTz589Hp06dxC6jSQRBwKZNmzB69Gg4OjrCzs4O/fv3x8qVK1FSUiJ2eXXUhoeGFpVKJXaJREbJUuwCiIzdpk2b9B5v3LgRMTExddb36dPngd5n7dq10Gq1Ldr2//7v/7BixYoHen9zp9Fo8PTTT2P79u0YNWoU3njjDdjZ2eH06dN48803sWPHDhw9ehSurq5il1rHqlWr6g2Qjo6O7V8MkQlguCFqxNy5c/Uex8fHIyYmps763ystLYWdnV2T38fKyqpF9QGApaUlLC35cTbkvffew/bt2/HSSy/h/fff161fsmQJZsyYgdDQUMyfPx8HDx5s17qa8nvy1FNPoUuXLu1UEZHp42kpolYwduxY+Pv7IyEhAaNHj4adnR3+8Y9/AAD27t2LKVOmwMPDAzKZDL169cJbb70FjUaj9xq/H3OTmpoKiUSCf//73/jiiy/Qq1cvyGQyDB06FBcuXNDbtr4xNxKJBEuXLsWePXvg7+8PmUyGfv364dChQ3Xqj42NxZAhQ2BjY4NevXphzZo1rT6OZ8eOHQgMDIStrS26dOmCuXPn4vbt23ptVCoVFixYAE9PT8hkMri7u2PatGlITU3Vtbl48SImTpyILl26wNbWFj169MAzzzxj8L3Lysrw/vvv46GHHkJUVFSd56dOnYrw8HAcOnQI8fHxAIDHH38cPXv2rPf1goODMWTIEL11mzdv1vXP2dkZs2bNQkZGhl4bQ78nDyI2NhYSiQTbtm3DP/7xD7i5ucHe3h5/+MMf6tQANG1fAMCNGzcwY8YMdO3aFba2tvD19cUrr7xSp11BQQHmz58PR0dHKBQKLFiwAKWlpXptYmJiMHLkSDg6OqJTp07w9fVtlb4T1Yd/6hG1knv37mHSpEmYNWsW5s6dqzu9ER0djU6dOuGFF15Ap06dcPz4cbz22mtQq9V6RxAasmXLFhQVFeFPf/oTJBIJ3nvvPTz55JO4detWo0d7zpw5g2+//RbPPfccHBwc8Mknn2D69OlIT09H586dAQCXL1/GY489Bnd3d7z55pvQaDRYuXIlunbt+uA/lPuio6OxYMECDB06FFFRUcjJycHHH3+Ms2fP4vLly7rTK9OnT8e1a9fw/PPPw9vbG7m5uYiJiUF6erru8aOPPoquXbtixYoVcHR0RGpqKr799ttGfw75+flYtmxZg0e4wsLCsGHDBuzbtw9BQUGYOXMmwsLCcOHCBQwdOlTXLi0tDfHx8Xr77u2338arr76KGTNmYNGiRbhz5w4+/fRTjB49Wq9/QMO/J4bk5eXVWWdpaVnntNTbb78NiUSCv//978jNzcVHH32E8ePHIzExEba2tgCavi9++OEHjBo1ClZWVliyZAm8vb3xyy+/4LvvvsPbb7+t974zZsxAjx49EBUVhUuXLmHdunVwcXHBu+++CwC4du0aHn/8cQwYMAArV66ETCZDcnIyzp4922jfiVpEIKJmiYiIEH7/0RkzZowAQFi9enWd9qWlpXXW/elPfxLs7OyE8vJy3brw8HDBy8tL9zglJUUAIHTu3FnIy8vTrd+7d68AQPjuu+90615//fU6NQEQrK2theTkZN26K1euCACETz/9VLdu6tSpgp2dnXD79m3dup9//lmwtLSs85r1CQ8PF+zt7Rt8vrKyUnBxcRH8/f2FsrIy3fp9+/YJAITXXntNEARByM/PFwAI77//foOvtXv3bgGAcOHChUbr+q2PPvpIACDs3r27wTZ5eXkCAOHJJ58UBEEQCgsLBZlMJrz44ot67d577z1BIpEIaWlpgiAIQmpqqmBhYSG8/fbbeu2SkpIES0tLvfWGfk/qU7tf61t8fX117U6cOCEAELp16yao1Wrd+u3btwsAhI8//lgQhKbvC0EQhNGjRwsODg66ftbSarV16nvmmWf02jzxxBNC586ddY8//PBDAYBw586dJvWb6EHxtBRRK5HJZFiwYEGd9bV/MQNAUVER7t69i1GjRqG0tBQ3btxo9HVnzpwJJycn3eNRo0YBAG7dutXotuPHj0evXr10jwcMGAC5XK7bVqPR4OjRowgNDYWHh4eunY+PDyZNmtTo6zfFxYsXkZubi+eeew42Nja69VOmTIGfnx/2798PoObnZG1tjdjYWOTn59f7WrVHFfbt24eqqqom11BUVAQAcHBwaLBN7XNqtRoAIJfLMWnSJGzfvh2CIOjabdu2DUFBQejevTsA4Ntvv4VWq8WMGTNw9+5d3eLm5obevXvjxIkTeu/T0O+JIbt27UJMTIzesmHDhjrtwsLC9Pr41FNPwd3dHQcOHADQ9H1x584dnDp1Cs8884yun7XqO1X57LPP6j0eNWoU7t27p/tZ1u63vXv3tnjQPFFzMNwQtZJu3brB2tq6zvpr167hiSeegEKhgFwuR9euXXWDkQsLCxt93d9/udQGnYYCgKFta7ev3TY3NxdlZWXw8fGp066+dS2RlpYGAPD19a3znJ+fn+55mUyGd999FwcPHoSrqytGjx6N9957T+9y5zFjxmD69Ol488030aVLF0ybNg0bNmxARUWFwRpqv/BrQ0596gtAM2fOREZGBuLi4gAAv/zyCxISEjBz5kxdm59//hmCIKB3797o2rWr3nL9+nXk5ubqvU9DvyeGjB49GuPHj9dbgoOD67Tr3bu33mOJRAIfHx/dmKWm7ova8Ovv79+k+hr7HZ05cyZGjBiBRYsWwdXVFbNmzcL27dsZdKjNMNwQtZLfHqGpVVBQgDFjxuDKlStYuXIlvvvuO8TExOjGIjTlH3cLC4t61//2aEJbbCuG5cuX46effkJUVBRsbGzw6quvok+fPrh8+TKAmi/rnTt3Ii4uDkuXLsXt27fxzDPPIDAwEMXFxQ2+bu1l+j/88EODbWqf69u3r27d1KlTYWdnh+3btwMAtm/fDqlUij/+8Y+6NlqtFhKJBIcOHapzdCUmJgZr1qzRe5/6fk9MXWO/Z7a2tjh16hSOHj2KefPm4YcffsDMmTMxYcKEOgPriVoDww1RG4qNjcW9e/cQHR2NZcuW4fHHH8f48eP1TjOJycXFBTY2NkhOTq7zXH3rWsLLywsAcPPmzTrP3bx5U/d8rV69euHFF1/EkSNHcPXqVVRWVuKDDz7QaxMUFIS3334bFy9exNdff41r165h69atDdZQe5XOli1bGvwy3bhxI4Caq6Rq2dvb4/HHH8eOHTug1Wqxbds2jBo1Su8UXq9evSAIAnr06FHn6Mr48eMRFBTUyE+o9fz88896jwVBQHJysu4qvKbui9qrxK5evdpqtUmlUowbNw7/+c9/8OOPP+Ltt9/G8ePH65y2I2oNDDdEbaj2L9rfHimprKzEf//7X7FK0mNhYYHx48djz549yMrK0q1PTk5utflehgwZAhcXF6xevVrv9NHBgwdx/fp1TJkyBUDNfC/l5eV62/bq1QsODg667fLz8+scdRo4cCAAGDw1ZWdnh5deegk3b96s91Lm/fv3Izo6GhMnTqwTRmbOnImsrCysW7cOV65c0TslBQBPPvkkLCws8Oabb9apTRAE3Lt3r8G6WtvGjRv1Tr3t3LkT2dnZuvFTTd0XXbt2xejRo7F+/Xqkp6frvUdLjvrVd7VXU/YbUUvxUnCiNhQSEgInJyeEh4fjL3/5CyQSCTZt2mRUp4XeeOMNHDlyBCNGjMCf//xnaDQafPbZZ/D390diYmKTXqOqqgr//Oc/66x3dnbGc889h3fffRcLFizAmDFjMHv2bN3lx97e3vjrX/8KAPjpp58wbtw4zJgxA3379oWlpSV2796NnJwczJo1CwDw1Vdf4b///S+eeOIJ9OrVC0VFRVi7di3kcjkmT55ssMYVK1bg8uXLePfddxEXF4fp06fD1tYWZ86cwebNm9GnTx989dVXdbabPHkyHBwc8NJLL8HCwgLTp0/Xe75Xr1745z//icjISKSmpiI0NBQODg5ISUnB7t27sWTJErz00ktN+jk2ZOfOnfXOUDxhwgS9S8mdnZ0xcuRILFiwADk5Ofjoo4/g4+ODxYsXA6iZKLIp+wIAPvnkE4wcORKDBw/GkiVL0KNHD6SmpmL//v1N/r2otXLlSpw6dQpTpkyBl5cXcnNz8d///heenp4YOXJky34oRIaIco0WkQlr6FLwfv361dv+7NmzQlBQkGBrayt4eHgIL7/8snD48GEBgHDixAldu4YuBa/v0mgAwuuvv6573NCl4BEREXW29fLyEsLDw/XWHTt2TBg0aJBgbW0t9OrVS1i3bp3w4osvCjY2Ng38FH4VHh7e4OXKvXr10rXbtm2bMGjQIEEmkwnOzs7CnDlzhMzMTN3zd+/eFSIiIgQ/Pz/B3t5eUCgUwvDhw4Xt27fr2ly6dEmYPXu20L17d0EmkwkuLi7C448/Lly8eLHROgVBEDQajbBhwwZhxIgRglwuF2xsbIR+/foJb775plBcXNzgdnPmzBEACOPHj2+wza5du4SRI0cK9vb2gr29veDn5ydEREQIN2/e1LUx9HtSH0OXgv/296f2UvBvvvlGiIyMFFxcXARbW1thypQpdS7lFoTG90Wtq1evCk888YTg6Ogo2NjYCL6+vsKrr75ap77fX+K9YcMGAYCQkpIiCELN79e0adMEDw8PwdraWvDw8BBmz54t/PTTT03+WRA1h0QQjOhPSCIyGqGhobh27VqdcRxkfGJjY/Hwww9jx44deOqpp8Quh0h0HHNDRCgrK9N7/PPPP+PAgQMYO3asOAURET0AjrkhIvTs2RPz589Hz549kZaWhlWrVsHa2hovv/yy2KURETUbww0R4bHHHsM333wDlUoFmUyG4OBg/Otf/6ozKRwRkSngmBsiIiIyKxxzQ0RERGaF4YaIiIjMSocbc6PVapGVlQUHB4d6725LRERExkcQBBQVFcHDwwNSqeFjMx0u3GRlZUGpVIpdBhEREbVARkYGPD09DbYxmnDzzjvvIDIyEsuWLcNHH31Ub5vo6GgsWLBAb51MJqtzPxpDHBwcANT8cORyeYvrJSIiovajVquhVCp13+OGGEW4uXDhAtasWYMBAwY02lYul+vd0ba5p5Zq28vlcoYbIiIiE9OU733RBxQXFxdjzpw5WLt2LZycnBptL5FI4Obmplt+e9M4IiIiItHDTUREBKZMmYLx48c3qX1xcTG8vLygVCoxbdo0XLt2zWD7iooKqNVqvYWIiIjMl6jhZuvWrbh06RKioqKa1N7X1xfr16/H3r17sXnzZmi1WoSEhCAzM7PBbaKioqBQKHQLBxMTERGZN9FmKM7IyMCQIUMQExOjG2szduxYDBw4sMEBxb9XVVWFPn36YPbs2XjrrbfqbVNRUYGKigrd49oBSYWFhRxzQ0REZCLUajUUCkWTvr9FG1CckJCA3NxcDB48WLdOo9Hg1KlT+Oyzz1BRUQELCwuDr2FlZYVBgwYhOTm5wTYymQwymazV6iYiIiLjJlq4GTduHJKSkvTWLViwAH5+fvj73//eaLABasJQUlISJk+e3FZlEhERkYkRLdw4ODjA399fb529vT06d+6sWx8WFoZu3brpxuSsXLkSQUFB8PHxQUFBAd5//32kpaVh0aJF7V4/ERERGSejmOemIenp6XpTLOfn52Px4sVQqVRwcnJCYGAgzp07h759+4pYJRERERkT0QYUi6U5A5KIiIjIODTn+1v0eW6IiIiIWhPDDREREZkVhhsiIiIyKww3rUhVWI7r2by9AxERkZgYblrJwaRsjHrvOF7ZndR4YyIiImozDDetJNDbCRJIcCm9AAlpeWKXQ0RE1GEx3LQSFwcbhA7yAACsPZUicjVEREQdF8NNK1o0qicA4PCPKqTdKxG5GiIioo6J4aYVPeTqgLG+XSEIwPozPHpDREQkBoabVrb4/tGb7RczUVBaKXI1REREHQ/DTSsL6dUZfd3lKKvS4Ovv08Uuh4iIqMNhuGllEokEi0f3AABEn0tFRbVG5IqIiIg6FoabNvD4AA+4yW1wp6gCexOzxC6HiIioQ2G4aQNWFlIsGOENAPjydAo62I3XiYiIRMVw00ZmDesOe2sL3Mwpwqmf74pdDhERUYfBcNNGFLZWmDm0OwBg7albIldDRETUcTDctKEFI7xhIZXgTPJd/JjFG2oSERG1B4abNqR0tsMkfzcAwLozPHpDRETUHhhu2ljtpH7/S8yCqrBc5GqIiIjMH8NNGwtQOmJYD2dUawVEn0sVuxwiIiKzx3DTDmqP3mz5Pg3FFdUiV0NERGTeGG7awTg/F/TsYg91eTW2X8gQuxwiIiKzxnDTDqRSCRaOqrklw/qzKajWaEWuiIiIyHwx3LST6YM94Wxvjcz8Mhy+liN2OURERGaL4aad2FhZYG6QFwDgi9O3eEsGIiKiNsJw047Cgr1gbSnFlYwCXEzLF7scIiIis8Rw0466dJJh+uBuAHhLBiIiorbCcNPOFo6suSw85noOUu6WiFwNERGR+WG4aWc+Lp3wiJ8LBAH4krdkICIianUMNyKondRvZ0Im8ksqRa6GiIjIvDDciCCopzP8u8lRXqXF5vg0scshIiIyK0YTbt555x1IJBIsX77cYLsdO3bAz88PNjY26N+/Pw4cONA+BbYiiUSiO3rzVVwqyqs0IldERERkPowi3Fy4cAFr1qzBgAEDDLY7d+4cZs+ejYULF+Ly5csIDQ1FaGgorl692k6Vtp7J/d3hobDB3eJK7E28LXY5REREZkP0cFNcXIw5c+Zg7dq1cHJyMtj2448/xmOPPYa//e1v6NOnD9566y0MHjwYn332WTtV23qsLKRYMKLmlgxrT6dAq+WkfkRERK1B9HATERGBKVOmYPz48Y22jYuLq9Nu4sSJiIuLa3CbiooKqNVqvcVYzBymRCeZJZJzi3Hypztil0NERGQWRA03W7duxaVLlxAVFdWk9iqVCq6urnrrXF1doVKpGtwmKioKCoVCtyiVygequTXJbawwe1hNPWtP87JwIiKi1iBauMnIyMCyZcvw9ddfw8bGps3eJzIyEoWFhbolIyOjzd6rJeaP6AELqQTnfrmHq7cLxS6HiIjI5IkWbhISEpCbm4vBgwfD0tISlpaWOHnyJD755BNYWlpCo6l7BZGbmxtycvTvqJ2TkwM3N7cG30cmk0Eul+stxqSboy2m9HcHAKzj0RsiIqIHJlq4GTduHJKSkpCYmKhbhgwZgjlz5iAxMREWFhZ1tgkODsaxY8f01sXExCA4OLi9ym4TtZeF7/shG9mFZSJXQ0REZNosxXpjBwcH+Pv7662zt7dH586ddevDwsLQrVs33ZicZcuWYcyYMfjggw8wZcoUbN26FRcvXsQXX3zR7vW3pv6eCgT1dEb8rTxEn01F5OQ+YpdERERkskS/WsqQ9PR0ZGdn6x6HhIRgy5Yt+OKLLxAQEICdO3diz549dUKSKao9erPl+3QUlVeJXA0REZHpkgiC0KEmWFGr1VAoFCgsLDSq8TdarYAJH57EL3dK8H9T+mDR/bBDREREzfv+NuojNx2JVCrRBZoNZ1NRrdGKXBEREZFpYrgxIk8M6obO9ta4XVCGg1cbnruHiIiIGsZwY0RsrCwwL9gLQM2kfh3sjCEREVGrYLgxMvOCvCCzlOKHzEKcT8kTuxwiIiKTw3BjZDp3kmF6oCeAmhtqEhERUfMw3BihhSNr7hZ+9HoOfrlTLHI1REREpoXhxgj16toJ4/vU3CD0yzM8ekNERNQcDDdGavGomqM3uxIyca+4QuRqiIiITAfDjZEa1sMZAzwVqKjWYlN8mtjlEBERmQyGGyMlkUh0t2TYFJeG8qq6d0knIiKiuhhujNgkfzd0c7TFvZJK7L58W+xyiIiITALDjRGztJBiwQhvADWT+mm1nNSPiIioMQw3Rm7WsO5wsLHErTslOHEzV+xyiIiIjB7DjZHrJLPE08O6A6g5ekNERESGMdyYgPkjvGEplSD+Vh6SMgvFLoeIiMioMdyYAHeFLaYGeADg0RsiIqLGMNyYiEX3J/Xbn5SN2wVlIldDRERkvBhuTEQ/DwVCenWGRisg+ixvyUBERNQQhhsTsnh0zaR+35zPgLq8SuRqiIiIjBPDjQkZ+1BX9HbphOKKamw7nyF2OUREREaJ4caESCQS3dibDWdTUKXRilwRERGR8WG4MTHTBnZDl07WyCosx4GkbLHLISIiMjoMNybGxsoC4cHeAGouCxcE3pKBiIjotxhuTNDcIC/YWElx9bYa8bfyxC6HiIjIqDDcmCAne2s8FegJgJP6ERER/R7DjYlaOLInJBLg+I1cJOcWiV0OERGR0WC4MVE9uthjQh9XAMCXZzipHxERUS2GGxNWO6nfrku3caeoQuRqiIiIjAPDjQkb4uWEgUpHVFZrsSk+TexyiIiIjALDjQmTSCRYPKrm6M3m+DSUVWpEroiIiEh8DDcmbmI/V3g62SKvpBK7LmWKXQ4REZHoGG5MnKWFFAtH1tySYf2ZFGi1nNSPiIg6NlHDzapVqzBgwADI5XLI5XIEBwfj4MGDDbaPjo6GRCLRW2xsbNqxYuM0Y4gSchtL3LpbgmM3csUuh4iISFSihhtPT0+88847SEhIwMWLF/HII49g2rRpuHbtWoPbyOVyZGdn65a0NA6ktZdZ4unhXgCAtac4qR8REXVsooabqVOnYvLkyejduzceeughvP322+jUqRPi4+Mb3EYikcDNzU23uLq6tmPFxmt+iDesLCQ4n5qHKxkFYpdDREQkGqMZc6PRaLB161aUlJQgODi4wXbFxcXw8vKCUqls9CgPAFRUVECtVust5shNYYOpAR4AeEsGIiLq2EQPN0lJSejUqRNkMhmeffZZ7N69G3379q23ra+vL9avX4+9e/di8+bN0Gq1CAkJQWZmw1cJRUVFQaFQ6BalUtlWXRHdopE1l4UfvKpCRl6pyNUQERGJQyIIgqiX11RWViI9PR2FhYXYuXMn1q1bh5MnTzYYcH6rqqoKffr0wezZs/HWW2/V26aiogIVFb/O3qtWq6FUKlFYWAi5XN5q/TAW8778Hqd/votnRvTAa1Mb/xkSERGZArVaDYVC0aTvb9GP3FhbW8PHxweBgYGIiopCQEAAPv744yZta2VlhUGDBiE5ObnBNjKZTHc1Vu1izhbdn9Rv24V0FJZViVwNERFR+xM93PyeVqvVO9JiiEajQVJSEtzd3du4KtMxuncX+Lo6oKRSg63n08Uuh4iIqN2JGm4iIyNx6tQppKamIikpCZGRkYiNjcWcOXMAAGFhYYiMjNS1X7lyJY4cOYJbt27h0qVLmDt3LtLS0rBo0SKxumB0JBIJFo6qmdRvw9lUVFZrRa6IiIiofVmK+ea5ubkICwtDdnY2FAoFBgwYgMOHD2PChAkAgPT0dEilv+av/Px8LF68GCqVCk5OTggMDMS5c+eaND6nI5k20APvH74Jlboc+5Oy8MQgT7FLIiIiajeiDyhub80ZkGTKPj+RjPcP30Rfdzn2/2UkJBKJ2CURERG1mEkNKKa2MWd4d9haWeDHbDXO/XJP7HKIiIjaDcONmXK0s8aMITWnozipHxERdSQMN2bsmZE9IJEAsTfv4KecIrHLISIiahcMN2bMq7M9JvZ1AwCs49EbIiLqIBhuzNzi0TWT+u25nIXconKRqyEiImp7DDdmLtDLCYO7O6JSo8WmuDSxyyEiImpzDDcdwOL7t2TYFJ+G0spqkashIiJqWww3HcCj/dzQ3dkOBaVV2JXQ8B3UiYiIzAHDTQdgIZVg4ciaWzJ8eSYFGm2HmreRiIg6GIabDuKPQzyhsLVC6r1SHL2eI3Y5REREbYbhpoOws7bE3KDuAIC1p3hZOBERmS+Gmw4kPNgb1hZSXEzLx+X0fLHLISIiahMMNx2Ii9wGfxjoAQBYdzpF5GqIiIjaBsNNB1N7WfjBq9nIyCsVuRoiIqLWx3DTwfi6OWD0Q12hFWqunCIiIjI3DDcd0OJRNZeFb7+YgcLSKpGrISIial0MNx3QSJ8u8HNzQGmlBl+f5y0ZiIjIvDDcdEASiUQ39uarc6morNaKXBEREVHrYbjpoKYGeMBVLkOOugLfXckSuxwiIqJWw3DTQVlbShEe4g0AWHv6FgSBt2QgIiLzwHDTgc0Z5gU7awvcUBXhTPJdscshIiJqFQw3HZjCzgozhigBAGs5qR8REZkJhpsObuHIHpBKgFM/3cENlVrscoiIiB4Yw00Hp3S2wyR/dwC8JQMREZkHhhvCovuT+u1NvI1cdbnI1RARET0YhhvCoO5OGOLlhCqNgOhzqWKXQ0RE9EAYbggAsHh0zaR+X3+fjtLKapGrISIiajmGGwIAjO/jCu/Odigsq8KOi5lil0NERNRiDDcEALCQSrBwZM3Ymy/PpECj5aR+RERkmhhuSOepQCWc7KyQnleKI9dUYpdDRETUIgw3pGNrbYG5QV4Aam7JQEREZIpEDTerVq3CgAEDIJfLIZfLERwcjIMHDxrcZseOHfDz84ONjQ369++PAwcOtFO1HcO8YC9YW0hxKb0ACWl5YpdDRETUbKKGG09PT7zzzjtISEjAxYsX8cgjj2DatGm4du1ave3PnTuH2bNnY+HChbh8+TJCQ0MRGhqKq1evtnPl5svFwQahgzwAAGtPcVI/IiIyPRLByG4H7ezsjPfffx8LFy6s89zMmTNRUlKCffv26dYFBQVh4MCBWL16dZNeX61WQ6FQoLCwEHK5vNXqNic/5RTh0Q9PQSIBYl8aC6/O9mKXREREHVxzvr+NZsyNRqPB1q1bUVJSguDg4HrbxMXFYfz48XrrJk6ciLi4uAZft6KiAmq1Wm8hwx5ydcBY364QBGD9GR69ISIi0yJ6uElKSkKnTp0gk8nw7LPPYvfu3ejbt2+9bVUqFVxdXfXWubq6QqVq+MqeqKgoKBQK3aJUKlu1fnO1eFTNpH7bL2aioLRS5GqIiIiaTvRw4+vri8TERHz//ff485//jPDwcPz444+t9vqRkZEoLCzULRkZGa322uYspFdn9HWXo6xKg6+/Txe7HCIioiYTPdxYW1vDx8cHgYGBiIqKQkBAAD7++ON627q5uSEnJ0dvXU5ODtzc3Bp8fZlMprsaq3ahxkkkEiweXTOpX/S5VFRUa0SuiIiIqGlEDze/p9VqUVFRUe9zwcHBOHbsmN66mJiYBsfo0IN5fIAH3OQ2uFNUgb2JWWKXQ0RE1CSihpvIyEicOnUKqampSEpKQmRkJGJjYzFnzhwAQFhYGCIjI3Xtly1bhkOHDuGDDz7AjRs38MYbb+DixYtYunSpWF0wa1YWUiwY4Q0A+PJ0CozswjoiIqJ6iRpucnNzERYWBl9fX4wbNw4XLlzA4cOHMWHCBABAeno6srOzde1DQkKwZcsWfPHFFwgICMDOnTuxZ88e+Pv7i9UFszdrWHfYW1vgZk4RTv18V+xyiIiIGmV089y0Nc5z03wrv/sR68+mYKRPF2xeNFzscoiIqAMyyXluyHgtGOENC6kEZ5Lv4scszhNERETGjeGGGqV0tsMk/5or0tad4Q01iYjIuDHcUJPUTur3v8QsqArLRa6GiIioYQw31CQBSkcM6+GMaq2A6HOpYpdDRETUIIYbarLaozdbvk9DSUW1yNUQERHVj+GGmmycnwt6drGHurwa2y/yNhZERGScGG6oyaRSCRaOqrklw5dnUlCt0YpcERERUV0MN9Qs0wd7wtneGpn5ZTh8LafxDYiIiNoZww01i42VBeYGeQEAvjh9i7dkICIio8NwQ80WFuwFa0sprmQU4GJavtjlEBER6WG4oWbr0kmG6YO7AQDWnuKkfkREZFwYbqhFFo6suSw85noOUu6WiFwNERHRrxhuqEV8XDrhET8XCALwJW/JQERERoThhlqsdlK/nQmZyC+pFLkaIiKiGgw31GJBPZ3h302O8iotNseniV0OERERAIYbegASiUR39OaruFSUV2lEroiIiIjhhh7Q5P7u8FDY4G5xJfYm3ha7HCIiIoYbejBWFlIsGFFzS4a1p1Og1XJSPyIiEhfDDT2wmcOU6CSzRHJuMU7+dEfscoiIqINjuKEHJrexwuxhSgDA2tO8LJyIiMTFcEOtYv6IHrCQSnDul3u4llUodjlERNSBMdxQq+jmaIsp/d0BAOtOp4hcDRERdWQMN9Rqai8L/+5KFrILy0SuhoiIOiqGG2o1/T0VCOrpjGqtgOizqWKXQ0REHRTDDbWq2qM3W75PR1F5lcjVEBFRR8RwQ63qYV8X9Opqj6KKamy7kCF2OURE1AEx3FCrkkolWHT/6M2Gs6mo1mhFroiIiDoahhtqdU8M6obO9ta4XVCGg1dVYpdDREQdDMMNtTobKwvMC/YCUDOpnyDwlgxERNR+GG6oTcwL8oLMUoofMgtxPiVP7HKIiKgDYbihNtG5kwzTAz0B1NxQk4iIqL2IGm6ioqIwdOhQODg4wMXFBaGhobh586bBbaKjoyGRSPQWGxubdqqYmmPhyJq7hR+9noNf7hSLXA0REXUUooabkydPIiIiAvHx8YiJiUFVVRUeffRRlJSUGNxOLpcjOztbt6SlpbVTxdQcvbp2wvg+rgCAL8/w6A0REbUPSzHf/NChQ3qPo6Oj4eLigoSEBIwePbrB7SQSCdzc3Nq6PGoFi0f1wNHrOdiVkIkXJzyEzp1kYpdERERmrkVHbjIyMpCZmal7fP78eSxfvhxffPHFAxVTWFhzN2lnZ2eD7YqLi+Hl5QWlUolp06bh2rVrD/S+1HaG9XDGAE8FKqq12BTPI2xERNT2WhRunn76aZw4cQIAoFKpMGHCBJw/fx6vvPIKVq5c2aJCtFotli9fjhEjRsDf37/Bdr6+vli/fj327t2LzZs3Q6vVIiQkRC9s/VZFRQXUarXeQu1HIpHobsmwKS4N5VUakSsiIiJz16Jwc/XqVQwbNgwAsH37dvj7++PcuXP4+uuvER0d3aJCIiIicPXqVWzdutVgu+DgYISFhWHgwIEYM2YMvv32W3Tt2hVr1qypt31UVBQUCoVuUSqVLaqPWm6Svxu6OdriXkkldl++LXY5RERk5loUbqqqqiCT1YydOHr0KP7whz8AAPz8/JCdnd3s11u6dCn27duHEydOwNPTs1nbWllZYdCgQUhOTq73+cjISBQWFuqWjAze76i9WVpIsWCEN4CaSf20Wk7qR0REbadF4aZfv35YvXo1Tp8+jZiYGDz22GMAgKysLHTu3LnJryMIApYuXYrdu3fj+PHj6NGjR7Nr0Wg0SEpKgru7e73Py2QyyOVyvYXa36xh3eFgY4lbd0pw4mau2OUQEZEZa1G4effdd7FmzRqMHTsWs2fPRkBAAADgf//7n+50VVNERERg8+bN2LJlCxwcHKBSqaBSqVBWVqZrExYWhsjISN3jlStX4siRI7h16xYuXbqEuXPnIi0tDYsWLWpJV6iddJJZ4ulh3QHUHL0hIiJqKy26FHzs2LG4e/cu1Go1nJycdOuXLFkCOzu7Jr/OqlWrdK/3Wxs2bMD8+fMBAOnp6ZBKf81g+fn5WLx4MVQqFZycnBAYGIhz586hb9++LekKtaP5I7zx5ZkUxN/KQ1JmIfp7KsQuiYiIzJBEaMFdDcvKyiAIgi7IpKWlYffu3ejTpw8mTpzY6kW2JrVaDYVCgcLCQp6iEsFftyVi9+Xb+EOABz6ZPUjscoiIyEQ05/u7Raelpk2bho0bNwIACgoKMHz4cHzwwQcIDQ3VHY0hqs+iUTXjqvYnZeN2QVkjrYmIiJqvReHm0qVLGDVqFABg586dcHV1RVpaGjZu3IhPPvmkVQsk89LPQ4GQXp2h0QqIPstbMhARUetrUbgpLS2Fg4MDAODIkSN48sknIZVKERQUxPs8UaMWj66Z1O+b8xlQl1eJXA0REZmbFoUbHx8f7NmzBxkZGTh8+DAeffRRAEBubi7HsVCjxj7UFb1dOqG4ohrbznPeISIial0tCjevvfYaXnrpJXh7e2PYsGEIDg4GUHMUZ9AgDhIlwyQSiW7szYazKajSaEWuiIiIzEmLws1TTz2F9PR0XLx4EYcPH9atHzduHD788MNWK47M17SB3dClkzWyCstxIKn5s1oTERE1pEXhBgDc3NwwaNAgZGVl6W5aOWzYMPj5+bVacWS+bKwsEB7sDaBmUr8WzEhARERUrxaFG61Wi5UrV0KhUMDLywteXl5wdHTEW2+9Ba2WpxioaeYGecHGSoqrt9WIv5UndjlERGQmWhRuXnnlFXz22Wd45513cPnyZVy+fBn/+te/8Omnn+LVV19t7RrJTDnZW+OpwJobpfKWDERE1FpaNEOxh4cHVq9erbsbeK29e/fiueeew+3bt1utwNbGGYqNS8rdEjzyQSwEATj6wmj4uDiIXRIRERmhNp+hOC8vr96xNX5+fsjL4+kFaroeXewxoY8rAODLM5zUj4iIHlyLwk1AQAA+++yzOus/++wzDBgw4IGLoo6ldlK/XZdu405RhcjVEBGRqWvRXcHfe+89TJkyBUePHtXNcRMXF4eMjAwcOHCgVQsk8zfEywkDlY5IzCjApvg0vDDhIbFLIiIiE9aiIzdjxozBTz/9hCeeeAIFBQUoKCjAk08+iWvXrmHTpk2tXSOZOYlEgsWjao7ebI5PQ3mVRuSKiIjIlLVoQHFDrly5gsGDB0OjMd4vJw4oNk7VGi3G/jsWmfllePsJf8wZ7iV2SUREZETafEAxUWuztJBi4ciaWzJ8eToFWi0n9SMiopZhuCGjMWOIEnIbS9y6W4JjN3LFLoeIiEwUww0ZDXuZJZ6+fzqKk/oREVFLNetqqSeffNLg8wUFBQ9SCxHmh3jjyzO3cD4lD1cyChCgdBS7JCIiMjHNOnKjUCgMLl5eXggLC2urWqkDcFPYYGqABwAevSEiopZp1pGbDRs2tFUdRDqLRvbEt5du4+BVFTLySqF0thO7JCIiMiEcc0NGp6+HHKN6d4FGK2DD2VSxyyEiIhPDcENGadH9Sf22XUhHYVmVyNUQEZEpYbghozS6dxf4ujqgpFKDrefTxS6HiIhMCMMNGSWJRIKFo2om9dtwNhWV1VqRKyIiIlPBcENGa9pAD3R1kEGlLsf+pCyxyyEiIhPBcENGS2Zpgfkh3gCAtadS0Iq3QSMiIjPGcENGbc7w7rC1ssCP2Wqc++We2OUQEZEJYLgho+ZoZ40ZQzwBcFI/IiJqGoYbMnrPjOwBiQSIvXkHP+UUiV0OEREZOYYbMnpene0xsa8bAGAdj94QEVEjGG7IJCweXTOp357LWcgtKhe5GiIiMmaihpuoqCgMHToUDg4OcHFxQWhoKG7evNnodjt27ICfnx9sbGzQv39/HDhwoB2qJTEFejlhcHdHVGq02BSXJnY5RERkxEQNNydPnkRERATi4+MRExODqqoqPProoygpKWlwm3PnzmH27NlYuHAhLl++jNDQUISGhuLq1avtWDmJYfH9WzJsik9DWaVG5GqIiMhYSQQjmjzkzp07cHFxwcmTJzF69Oh628ycORMlJSXYt2+fbl1QUBAGDhyI1atXN/oearUaCoUChYWFkMvlrVY7tT2NVsDD/45Fel4p3prWD/OCvcUuiYiI2klzvr+NasxNYWEhAMDZ2bnBNnFxcRg/frzeuokTJyIuLq7e9hUVFVCr1XoLmSYLqQQLR9bckuHLMynQaI0mlxMRkRExmnCj1WqxfPlyjBgxAv7+/g22U6lUcHV11Vvn6uoKlUpVb/uoqCgoFArdolQqW7Vual9/HOIJha0VUu+V4uj1HLHLISIiI2Q04SYiIgJXr17F1q1bW/V1IyMjUVhYqFsyMjJa9fWpfdlZW2JuUHcAwNpTvCyciIjqMopws3TpUuzbtw8nTpyAp6enwbZubm7IydH/iz0nJwdubm71tpfJZJDL5XoLmbbwYG9YW0hxMS0fl9PzxS6HiIiMjKjhRhAELF26FLt378bx48fRo0ePRrcJDg7GsWPH9NbFxMQgODi4rcokI+Mit8EfBnoAANadThG5GiIiMjaihpuIiAhs3rwZW7ZsgYODA1QqFVQqFcrKynRtwsLCEBkZqXu8bNkyHDp0CB988AFu3LiBN954AxcvXsTSpUvF6AKJZNGomiB88Go2MvJKRa6GiIiMiajhZtWqVSgsLMTYsWPh7u6uW7Zt26Zrk56ejuzsbN3jkJAQbNmyBV988QUCAgKwc+dO7Nmzx+AgZDI/fm5yjH6oK7RCzZVTREREtYxqnpv2wHluzMfpn+9g3pfnYWdtgbgV46CwsxK7JCIiaiMmO88NUXOM9OkCPzcHlFZq8PV53pKBiIhqMNyQyZJIJLpbMnx1LhWV1VqRKyIiImPAcEMmbWqAB1zlMuSoK/DdlSyxyyEiIiPAcEMmzdpSivAQbwDA2tO30MGGkBERUT0YbsjkzRnmBTtrC9xQFeFM8l2xyyEiIpEx3JDJU9hZYcaQmnuGreWkfkREHR7DDZmFhSN7QCoBTv10BzdVRWKXQ0REImK4IbOgdLbDJH93ADVjb4iIqONiuCGzUXtLhr2Jt5GrLhe5GiIiEgvDDZmNQd2dMMTLCVUaAV/FpYpdDhERiYThhszK4tE1k/ptjk9HaWW1yNUQEZEYGG7IrIzv4wrvznYoLKvCjouZYpdDREQiYLghs2IhlWDhyJqxN1+eSYFGy0n9iIg6GoYbMjtPBSrhZGeF9LxSHLmmErscIiJqZww3ZHZsrS0wN8gLAC8LJyLqiBhuyCzNC/aCtYUUl9ILkJCWJ3Y5RETUjhhuyCy5ONggdJAHAGDtKd6SgYioI2G4IbO1aFTNZeGHf1Qh7V6JyNUQEVF7Ybghs/WQqwPG+naFIADrz/DoDRFRR8FwQ2Zt8f2jN9svZqKgtFLkaoiIqD0w3JBZC+nVGX3d5Sir0uDr79PFLoeIiNoBww2ZNYlEgsWjayb1iz6XiopqjcgVERFRW2O4IbP3+AAPuMltcKeoAnsTs8Quh4iI2hjDDZk9KwspFozwBgB8eToFgsBbMhARmTOGG+oQZg3rDntrC9zMKcKpn++KXQ4REbUhhhvqEBS2Vpg5tDsAYB1vyUBEZNYYbqjDWDDCGxZSCU7/fBc/ZqnFLoeIiNoIww11GEpnO0zydwMArDvDozdEROaK4YY6lNpJ/b67kgVVYbnI1RARUVtguKEOJUDpiGE9nFGlERB9LlXscoiIqA0w3FCHU3v0Zsv3aSipqBa5GiIiam2ihptTp05h6tSp8PDwgEQiwZ49ewy2j42NhUQiqbOoVKr2KZjMwjg/F/TsYg91eTW2X8wQuxwiImplooabkpISBAQE4PPPP2/Wdjdv3kR2drZucXFxaaMKyRxJpRIsHFVzS4Yvz6SgWqMVuSIiImpNlmK++aRJkzBp0qRmb+fi4gJHR8fWL4g6jOmDPfHBkZ+QmV+Gw9dyMGWAu9glERFRKzHJMTcDBw6Eu7s7JkyYgLNnz4pdDpkgGysLzA3yAgB8cfoWb8lARGRGTCrcuLu7Y/Xq1di1axd27doFpVKJsWPH4tKlSw1uU1FRAbVarbcQAUBYsBesLaW4klGAi2n5YpdDREStxKTCja+vL/70pz8hMDAQISEhWL9+PUJCQvDhhx82uE1UVBQUCoVuUSqV7VgxGbMunWSYPrgbAGDtKU7qR0RkLkwq3NRn2LBhSE5ObvD5yMhIFBYW6paMDF4dQ79aOLLmsvCY6zlIuVsicjVERNQaTD7cJCYmwt294cGgMpkMcrlcbyGq5ePSCY/4uUAQgC95SwYiIrMg6tVSxcXFekddUlJSkJiYCGdnZ3Tv3h2RkZG4ffs2Nm7cCAD46KOP0KNHD/Tr1w/l5eVYt24djh8/jiNHjojVBTIDi0f1xPEbudiZkIkXJ/jCyd5a7JKIiOgBiHrk5uLFixg0aBAGDRoEAHjhhRcwaNAgvPbaawCA7OxspKen69pXVlbixRdfRP/+/TFmzBhcuXIFR48exbhx40Spn8xDUE9n+HeTo7xKi83xaWKXQ0RED0gidLBrYNVqNRQKBQoLC3mKinT2Jt7Gsq2J6NJJhjN/fxg2VhZil0RERL/RnO9vkx9zQ9QaJvd3h4fCBneLK/DK7qtI5eBiIiKTxXBDBMDKQoqIR3wAALsuZWLsv2Mxf8N5HL+RA622Qx3cJCIyeaIOKCYyJnOGe8HD0RYbz6Ui9qc7iL1Zs3R3tsO8IC/8cYgnHO042JiIyNhxzA1RPVLvlmBzfBq2X8yAurwaACCzlCJ0YDfMC/aCfzeFyBUSEXUszfn+ZrghMqCsUoO9ibexMS4NP2b/euuOQC8nhAV7YZK/O6wteXaXiKitMdwYwHBDLSEIAhLS8rExLg0HkrJRfX8cTpdOMswepsTTw7vDXWErcpVEROaL4cYAhht6ULlF5dh6PgNff5+GHHUFAMBCKsHEfq6YF+SNoJ7OkEgkIldJRGReGG4MYLih1lKl0eLItRxsjEvF9yl5uvUPuXbCvGBvPDmoG+xlHLNPRNQaGG4MYLihtnBDpcamuDTsvnwbpZUaAICDzBLTAz0xL9gLvbp2ErlCIiLTxnBjAMMNtSV1eRV2JWRiU1wabv1mIsBRvbtgXpAXxvVxhYWUp6yIiJqL4cYAhhtqD1qtgLO/3MVX59Jw7EYOaj9l3RxtMSeoO2YN7Q5n3qCTiKjJGG4MYLih9paRV4qvv0/HtgvpyC+tAgBYW0oxdYAHwoK9EKB0FLdAIiITwHBjAMMNiaW8SoPvrmRhY1wakm4X6tYHKB0RFuSFKQPcecNOIqIGMNwYwHBDYhMEAYkZBdgUl4Z9P2SjUqMFADjbW2PmUCXmDO8OTyc7kaskIjIuDDcGMNyQMblbXIFtFzLwdXwasgrLAQBSCTCujyvCg70xwqcz58whIgLDjUEMN2SMqjVaHLuRi41xqTibfE+3vldXe8wL8sL0QE842FiJWCERkbgYbgxguCFjl5xbhE1xadiZkImS+3Pm2Ftb4InB3RAW7I2HXB1ErpCIqP0x3BjAcEOmoriiGrsvZeKruDQk5xbr1gf37IywYC9M6OsKSwvetJOIOgaGGwMYbsjUCIKAuFv3sPFcGmKu50Bz/6ad7gobPD2sO2YN646uDjKRqyQialsMNwYw3JApyyoow5bv0/HN+XTcK6kEAFhZSDC5vzvCgr0xuLsjByATkVliuDGA4YbMQUW1BgeTVPgqLhWX0wt06/27yREW5I0/DPTgnDlEZFYYbgxguCFzk5RZiI1xqfjflSxUVNfMmeNoZ4UZQ5SYO9wL3TtzzhwiMn0MNwYw3JC5yi+pxPaLGdgUn4bM/DIAgEQCPOzrgrBgL4zu3RVS3rSTiEwUw40BDDdk7jRaAbE3c/FVXBpO/XRHt967sx3mBnnhj4FKKOw4Zw4RmRaGGwMYbqgjuXWnGJvj07EjIQNF5dUAAFsrC4QO8sC8IG/09eBngIhMA8ONAQw31BGVVlZjz+UsbIxLxQ1VkW79MG9nzAv2wmP+brDinDlEZMQYbgxguKGOTBAEXEjNx1dxqTh8VYXq+3PmuDjIMHtYdzw9vDtc5TYiV0lEVBfDjQEMN0Q1ctTl2PJ9OracT8edogoAgKVUgon+bggP9sZQbyfOmUNERoPhxgCGGyJ9ldVaHL6mwsa4VFxIzdet93NzQFiwN0IHecDO2lLEComIGG4MYrghatiPWWpsik/FnstZKKuquWmng40l/hioxLxgL/ToYi9yhUTUUTHcGMBwQ9S4wtIq7EjIwOb4NKTeK9WtH/1QV4QHe2GsrwssOGcOEbUjhhsDGG6Imk6rFXDq5zvYGJeGEzdzUfuvhdLZFnOHe2HGECWc7K3FLZKIOoTmfH+Leu3nqVOnMHXqVHh4eEAikWDPnj2NbhMbG4vBgwdDJpPBx8cH0dHRbV4nUUcllUow1tcF6+cPxcmXHsaS0T2hsLVCRl4Zog7eQFDUMfxtxxVcvV0odqlERDqihpuSkhIEBATg888/b1L7lJQUTJkyBQ8//DASExOxfPlyLFq0CIcPH27jSomoe2c7/GNyH8RHjsN70wegn4ccFdVa7EjIxOOfnsET/z2LPZdvo6JaI3apRNTBGc1pKYlEgt27dyM0NLTBNn//+9+xf/9+XL16Vbdu1qxZKCgowKFDh5r0PjwtRdQ6BEHApfQCbIpLxf6kbFRpav4p6dLJGrOG1syZ4+FoK3KVRGQuTOa0VHPFxcVh/PjxeusmTpyIuLi4BrepqKiAWq3WW4jowUkkEgR6OeGjWYNwbsU4vDjhIbjJbXC3uBKfnUjGqPdO4NlNCTj3y10Yyd9QRNRBmFS4UalUcHV11Vvn6uoKtVqNsrKyereJioqCQqHQLUqlsj1KJepQujrI8Py43jjz94exas5gBPV0hkYr4NA1FZ5e+z0e/fAUNsWloriiWuxSiagDMKlw0xKRkZEoLCzULRkZGWKXRGS2LC2kmNTfHVuXBOPIX0djblB32Flb4OfcYry69xqC/nUMr++9iuTcYrFLJSIzZlLTjrq5uSEnJ0dvXU5ODuRyOWxt6z+3L5PJIJPJ2qM8IvqNh1wd8M/Q/nj5MT98m5CJjfFpuHWnBF/FpeGruDSM8OmMsGBvjPNzgSVv2klErcikwk1wcDAOHDigty4mJgbBwcEiVUREjZHbWGH+iB4ID/HG2eR72BiXiqPXc3A2+R7OJt+Dh8IGc4K8MGuoEp078Q8RInpwol4tVVxcjOTkZADAoEGD8J///AcPP/wwnJ2d0b17d0RGRuL27dvYuHEjgJpLwf39/REREYFnnnkGx48fx1/+8hfs378fEydObNJ78mopIvFl5pfi6+/Tse1CBvJKKgEA1hZSPD7AHWEh3hiodBS3QCIyOiYzQ3FsbCwefvjhOuvDw8MRHR2N+fPnIzU1FbGxsXrb/PWvf8WPP/4IT09PvPrqq5g/f36T35Phhsh4lFdpsP+HbGyMT8OVjALd+gGeCoQFe+PxAe6wsbIQr0AiMhomE27EwHBDZJwSMwqwMS4V+37IRmW1FgDgZGeFmUO7Y87w7lA624lcIRGJieHGAIYbIuOWV1KJbRdqbtp5u6BmigepBHjEzxXhIV4Y0asLpLxpJ1GHw3BjAMMNkWnQaAUcu56DTfFpOP3zXd36nl3sMS/YC9MDPSG3sRKxQiJqTww3BjDcEJme5NxibI5Pw86ETN1EgHbWFnhiUDeEBXvD181B5AqJqK0x3BjAcENkuoorqrH78m1sikvFTzm/TgQ4vIczwkO8MaGvK6w4Zw6RWWK4MYDhhsj0CYKA+Ft52BSfisPXcqDR1vwz5iqXYeYQJYJ6doa/p4KnrYjMCMONAQw3ROYlu7AM33yfji3n03G3uFLvuV5d7RGgdMRApSMCPB3h5+4AmSUvLScyRQw3BjDcEJmnimoNDl1V4ciPOfghswAZeXVvpmttIUUfDzkGeioQoHREgNIRPTrb8+orIhPAcGMAww1Rx3CvuAI/ZBYiMaMAVzILcCWjAPmlVXXaOcgsMUCpQICno+4oj6vcRoSKicgQhhsDGG6IOiZBEJCZX1YTdu4HnqTbhSiv0tZp6ya3QYCy5ujOQE9Hjt8hMgIMNwYw3BBRrWqNFj/lFOuO7CRmFOCnnCJo6/lXkeN3iMTFcGMAww0RGVJaWY1rWWpd2LnC8TtERoHhxgCGGyJqLo7fIRIfw40BDDdE9KAEQUBGXhkSMwvwA8fvELULhhsDGG6IqC1w/A5R22K4MYDhhojaS2llNa7eVuuuzuL4HaKWY7gxgOGGiMTE8TtELcNwYwDDDREZE47fIWoahhsDGG6IyNhx/A5RXQw3BjDcEJEp4vgd6ugYbgxguCEic8HxO9SRMNwYwHBDROaK43fInDHcGMBwQ0QdCcfvkLlguDGA4YaIOjqO3yFTxHBjAMMNEVFdHL9Dxo7hxgCGGyKixnH8DhkbhhsDGG6IiFqG43dITAw3BjDcEBG1nt+O30nMLMAPHL9DbYThxgCGGyKitsXxO9QWGG4MYLghImpfvx2/cyWjZrma1fD4nT7uDlA620HpZFfzX2dbKJ3tOI6ng2O4MYDhhohIfM0Zv1NLYWuF7rVhx8kOns52NY+dbNHNyZZjeswcw40BDDdERMapdvzOL3eKkZFXivS8UmTklyEzrxT3SioNbiuRAK4ONujubAfP++FHWRt+nG3h6mDD8T0mzuTCzeeff473338fKpUKAQEB+PTTTzFs2LB620ZHR2PBggV662QyGcrLy5v0Xgw3RESmp6SiGhn5pcjIK0NGXun9/7//OL8UpZUag9tbW0jRzcn2/ukuW91pr9rwo7C1gkTC8GPMmvP9bdlONTVo27ZteOGFF7B69WoMHz4cH330ESZOnIibN2/CxcWl3m3kcjlu3rype8xfSCIi82Yvs4Sfmxx+bnW/1ARBQF5Jpe5IT0ZeKTLz7x/5yStDVkEZKjVapNwtQcrdknpf30Fmef80l63eWJ/uznbwdLKDjRVPeZkS0Y/cDB8+HEOHDsVnn30GANBqtVAqlXj++eexYsWKOu2jo6OxfPlyFBQUtOj9eOSGiKhjqdZooVKXIz2vFJn3j/T89rTXnaKKRl+jq4MMSifb+0d67O6P+akJQu4KG1haSNuhJx2byRy5qaysREJCAiIjI3XrpFIpxo8fj7i4uAa3Ky4uhpeXF7RaLQYPHox//etf6NevX71tKyoqUFHx6y+uWq1uvQ4QEZHRs7SQwtOp5ggMetV9vrxKo3ekp/a0V3pezXifoopq3CmqwJ2iClxKL6j7+lIJPBxtdQOdlboAVHP6q7O9Nc8wtDNRw83du3eh0Wjg6uqqt97V1RU3btyodxtfX1+sX78eAwYMQGFhIf79738jJCQE165dg6enZ532UVFRePPNN9ukfiIiMn02VhbwcXGAj4tDnecEQUBhWRUy8sruH+m5P9bn/umv2/k1p7zS7x8JAu7VeQ07a4v7occWnrpxPr9e9WUvE32EiNkR9bRUVlYWunXrhnPnziE4OFi3/uWXX8bJkyfx/fffN/oaVVVV6NOnD2bPno233nqrzvP1HblRKpU8LUVERA9MqxWQU1T+a/i5H4Ay7z/OKSpHY9+yne2t4fmbIz3ddXP82MLD0RZWPOUFwIROS3Xp0gUWFhbIycnRW5+TkwM3N7cmvYaVlRUGDRqE5OTkep+XyWSQyWQPXCsREdHvSaUSuCts4a6wxbAeznWer6jW4HZ+GTLyy+6P+SnVXfWVnleKwrIq3CupxL2SSlzJKKj7+hLAXaF/yuu3c/10dZDxlFc9RA031tbWCAwMxLFjxxAaGgqgZkDxsWPHsHTp0ia9hkajQVJSEiZPntyGlRIRETWfzNICPbt2Qs+unep9Xl1e9esl7fWc9qqo1uJ2QRluF5QhHnn1vL60zuXtnNXZCC4Ff+GFFxAeHo4hQ4Zg2LBh+Oijj1BSUqKbyyYsLAzdunVDVFQUAGDlypUICgqCj48PCgoK8P777yMtLQ2LFi0SsxtERETNJrexQj8PBfp5KOo8p9UKuFtcoXekJ+M3R36yC8tQUa1Fcm4xknOL6339jjqrs+jhZubMmbhz5w5ee+01qFQqDBw4EIcOHdINMk5PT4dU+uv5xvz8fCxevBgqlQpOTk4IDAzEuXPn0LdvX7G6QERE1OqkUglc5DZwkdsg0Kvu81UaLbIKynQTGf4afn6d1bmwrApJtwuRdLuwzvbmPKuz6PPctDfOc0NERB3B72d1Tr8/uaGpzupsMgOKiYiIqG00NqvzvZJKvfE9vz3lZeqzOvPIDREREemp1miRXViud1n7bwc7Nzars49LJxx9YUyr1sQjN0RERNRilhZS3UzL9c3qXFZZM6tz3cHONeN9lE627V/0bzDcEBERUbPYWlugt6sDervWP6tzRbVWhKp+xWkPiYiIqNVIJBLR76LOcENERERmheGGiIiIzArDDREREZkVhhsiIiIyKww3REREZFYYboiIiMisMNwQERGRWWG4ISIiIrPCcENERERmheGGiIiIzArDDREREZkVhhsiIiIyKww3REREZFYsxS6gvQmCAABQq9UiV0JERERNVfu9Xfs9bkiHCzdFRUUAAKVSKXIlRERE1FxFRUVQKBQG20iEpkQgM6LVapGVlQUHBwdIJJJWfW21Wg2lUomMjAzI5fJWfW1jYO79A8y/j+yf6TP3PrJ/pq+t+igIAoqKiuDh4QGp1PComg535EYqlcLT07NN30Mul5vtLy1g/v0DzL+P7J/pM/c+sn+mry362NgRm1ocUExERERmheGGiIiIzArDTSuSyWR4/fXXIZPJxC6lTZh7/wDz7yP7Z/rMvY/sn+kzhj52uAHFREREZN545IaIiIjMCsMNERERmRWGGyIiIjIrDDdERERkVhhumunzzz+Ht7c3bGxsMHz4cJw/f95g+x07dsDPzw82Njbo378/Dhw40E6Vtkxz+hcdHQ2JRKK32NjYtGO1zXPq1ClMnToVHh4ekEgk2LNnT6PbxMbGYvDgwZDJZPDx8UF0dHSb19lSze1fbGxsnf0nkUigUqnap+BmioqKwtChQ+Hg4AAXFxeEhobi5s2bjW5nSp/BlvTRlD6Hq1atwoABA3STuwUHB+PgwYMGtzGl/dfc/pnSvqvPO++8A4lEguXLlxtsJ8Y+ZLhphm3btuGFF17A66+/jkuXLiEgIAATJ05Ebm5uve3PnTuH2bNnY+HChbh8+TJCQ0MRGhqKq1evtnPlTdPc/gE1M1BmZ2frlrS0tHasuHlKSkoQEBCAzz//vEntU1JSMGXKFDz88MNITEzE8uXLsWjRIhw+fLiNK22Z5vav1s2bN/X2oYuLSxtV+GBOnjyJiIgIxMfHIyYmBlVVVXj00UdRUlLS4Dam9hlsSR8B0/kcenp64p133kFCQgIuXryIRx55BNOmTcO1a9fqbW9q+6+5/QNMZ9/93oULF7BmzRoMGDDAYDvR9qFATTZs2DAhIiJC91ij0QgeHh5CVFRUve1nzJghTJkyRW/d8OHDhT/96U9tWmdLNbd/GzZsEBQKRTtV17oACLt37zbY5uWXXxb69eunt27mzJnCxIkT27Cy1tGU/p04cUIAIOTn57dLTa0tNzdXACCcPHmywTam9hn8vab00ZQ/h4IgCE5OTsK6devqfc7U958gGO6fqe67oqIioXfv3kJMTIwwZswYYdmyZQ22FWsf8shNE1VWViIhIQHjx4/XrZNKpRg/fjzi4uLq3SYuLk6vPQBMnDixwfZiakn/AKC4uBheXl5QKpWN/oViakxp/z2IgQMHwt3dHRMmTMDZs2fFLqfJCgsLAQDOzs4NtjH1fdiUPgKm+TnUaDTYunUrSkpKEBwcXG8bU95/TekfYJr7LiIiAlOmTKmzb+oj1j5kuGmiu3fvQqPRwNXVVW+9q6trg2MUVCpVs9qLqSX98/X1xfr167F3715s3rwZWq0WISEhyMzMbI+S21xD+0+tVqOsrEykqlqPu7s7Vq9ejV27dmHXrl1QKpUYO3YsLl26JHZpjdJqtVi+fDlGjBgBf3//BtuZ0mfw95raR1P7HCYlJaFTp06QyWR49tlnsXv3bvTt27fetqa4/5rTP1PbdwCwdetWXLp0CVFRUU1qL9Y+7HB3BafWExwcrPcXSUhICPr06YM1a9bgrbfeErEyagpfX1/4+vrqHoeEhOCXX37Bhx9+iE2bNolYWeMiIiJw9epVnDlzRuxS2kxT+2hqn0NfX18kJiaisLAQO3fuRHh4OE6ePNlgADA1zemfqe27jIwMLFu2DDExMUY/8Jnhpom6dOkCCwsL5OTk6K3PycmBm5tbvdu4ubk1q72YWtK/37OyssKgQYOQnJzcFiW2u4b2n1wuh62trUhVta1hw4YZfWBYunQp9u3bh1OnTsHT09NgW1P6DP5Wc/r4e8b+ObS2toaPjw8AIDAwEBcuXMDHH3+MNWvW1GlrivuvOf37PWPfdwkJCcjNzcXgwYN16zQaDU6dOoXPPvsMFRUVsLCw0NtGrH3I01JNZG1tjcDAQBw7dky3TqvV4tixYw2eTw0ODtZrDwAxMTEGz7+KpSX9+z2NRoOkpCS4u7u3VZntypT2X2tJTEw02v0nCAKWLl2K3bt34/jx4+jRo0ej25jaPmxJH3/P1D6HWq0WFRUV9T5navuvPob693vGvu/GjRuHpKQkJCYm6pYhQ4Zgzpw5SExMrBNsABH3YZsOVzYzW7duFWQymRAdHS38+OOPwpIlSwRHR0dBpVIJgiAI8+bNE1asWKFrf/bsWcHS0lL497//LVy/fl14/fXXBSsrKyEpKUmsLhjU3P69+eabwuHDh4VffvlFSEhIEGbNmiXY2NgI165dE6sLBhUVFQmXL18WLl++LAAQ/vOf/wiXL18W0tLSBEEQhBUrVgjz5s3Ttb9165ZgZ2cn/O1vfxOuX78ufP7554KFhYVw6NAhsbpgUHP79+GHHwp79uwRfv75ZyEpKUlYtmyZIJVKhaNHj4rVBYP+/Oc/CwqFQoiNjRWys7N1S2lpqa6NqX8GW9JHU/ocrlixQjh58qSQkpIi/PDDD8KKFSsEiUQiHDlyRBAE099/ze2fKe27hvz+ailj2YcMN8306aefCt27dxesra2FYcOGCfHx8brnxowZI4SHh+u13759u/DQQw8J1tbWQr9+/YT9+/e3c8XN05z+LV++XNfW1dVVmDx5snDp0iURqm6a2kuff7/U9ik8PFwYM2ZMnW0GDhwoWFtbCz179hQ2bNjQ7nU3VXP79+677wq9evUSbGxsBGdnZ2Hs2LHC8ePHxSm+CerrGwC9fWLqn8GW9NGUPofPPPOM4OXlJVhbWwtdu3YVxo0bp/viFwTT33/N7Z8p7buG/D7cGMs+lAiCILTtsSEiIiKi9sMxN0RERGRWGG6IiIjIrDDcEBERkVlhuCEiIiKzwnBDREREZoXhhoiIiMwKww0RERGZFYYbIurwJBIJ9uzZI3YZRNRKGG6ISFTz58+HRCKpszz22GNil0ZEJop3BSci0T322GPYsGGD3jqZTCZSNURk6njkhohEJ5PJ4Obmprc4OTkBqDlltGrVKkyaNAm2trbo2bMndu7cqbd9UlISHnnkEdja2qJz585YsmQJiouL9dqsX78e/fr1g0wmg7u7O5YuXar3/N27d/HEE0/Azs4OvXv3xv/+97+27TQRtRmGGyIyeq+++iqmT5+OK1euYM6cOZg1axauX78OACgpKcHEiRPh5OSECxcuYMeOHTh69KheeFm1ahUiIiKwZMkSJCUl4X//+x98fHz03uPNN9/EjBkz8MMPP2Dy5MmYM2cO8vLy2rWfRNRK2vzWnEREBoSHhwsWFhaCvb293vL2228LglBzp+xnn31Wb5vhw4cLf/7znwVBEIQvvvhCcHJyEoqLi3XP79+/X5BKpYJKpRIEQRA8PDyEV155pcEaAAj/93//p3tcXFwsABAOHjzYav0kovbDMTdEJLqHH34Yq1at0lvn7Oys+//g4GC954KDg5GYmAgAuH79OgICAmBvb697fsSIEdBqtbh58yYkEgmysrIwbtw4gzUMGDBA9//29vaQy+XIzc1taZeISEQMN0QkOnt7+zqniVqLra1tk9pZWVnpPZZIJNBqtW1REhG1MY65ISKjFx8fX+dxnz59AAB9+vTBlStXUFJSonv+7NmzkEql8PX1hYODA7y9vXHs2LF2rZmIxMMjN0QkuoqKCqhUKr11lpaW6NKlCwBgx44dGDJkCEaOHImvv/4a58+fx5dffgkAmDNnDl5//XWEh4fjjTfewJ07d/D8889j3rx5cHV1BQC88cYbePbZZ+Hi4oJJkyahqKgIZ8+exfPPP9++HSWidsFwQ0SiO3ToENzd3fXW+fr64saNGwBqrmTaunUrnnvuObi7u+Obb75B3759AQB2dnY4fPgwli1bhqFDh8LOzg7Tp0/Hf/7zH91rhYeHo7y8HB9++CFeeukldOnSBU899VT7dZCI2pVEEARB7CKIiBoikUiwe/duhIaGil0KEZkIjrkhIiIis8JwQ0RERGaFY26IyKjxzDkRNReP3BAREZFZYbghIiIis8JwQ0RERGaF4YaIiIjMCsMNERERmRWGGyIiIjIrDDdERERkVhhuiIiIyKww3BAREZFZ+X9611boj/snDQAAAABJRU5ErkJggg==\n" + }, + "metadata": {} + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tn-jgVULyBXq" + }, + "source": [ + "Note that the model began to grasp our intent more effectively from Epoch #3 onwards.\n", + "\n", + "To compare and contrast, it was utlized the \"Write a poem\" prompt. Interestingly, in Epoch #5, the model began to generate Portuguese in response to that prompt. This shift indicates a strong influence of our training dataset on the model's behavior. However, depending on your application, such a significat change might not be desirable. In such cases, Epoch #4 would be a more suitable choice." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "P-tVAKmda2Zt" + }, + "source": [ + "## Load LoRA\n", + "\n", + "Use the code below if you shared LoRA weights. It's much more lightweight than the model files themselves - for instance, a LoRA rank 4 weights file for a 10gb model might only be on the order of a few megabytes, easily shared over email." + ] + }, + { + "cell_type": "code", + "source": [ + "# Example Code for Load LoRA\n", + "\n", + "# from peft import PeftModel\n", + "\n", + "# # Load pre-trained LoRA weights (assuming the weights are saved in Hugging Face format)\n", + "# # Load the pre-trained LoRA weights\n", + "# lora_weights_path = f\"./{lora_name}_{lora_rank}_epoch{train_epoch}.lora.pt\"\n", + "\n", + "# # Load the LoRA adapter into the model using PeftModel\n", + "# gemma_lm = PeftModel.from_pretrained(gemma_lm, lora_weights_path)" + ], + "metadata": { + "id": "kVe4vjgCngsd" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ipg1u_wEKTxG" + }, + "source": [ + "## Try a different sampler\n", + "\n", + "The top-K algorithm randomly picks the next token from the tokens of top K probability." + ] + }, + { + "cell_type": "code", + "source": [ + "import torch\n", + "\n", + "def text_gen_with_top_k(prompt, token_limit=100, top_k=50): # You can set your token limit and top_k\n", + " tick()\n", + "\n", + " # Format input, same as your original code\n", + " input_text = f\"user\\n{prompt}\\nmodel\\n\"\n", + "\n", + " # Tokenize input\n", + " inputs = tokenizer(input_text, return_tensors=\"pt\")\n", + "\n", + " # Generate text using the model with Top-K sampling\n", + " output_tokens = gemma_lm.generate(\n", + " inputs[\"input_ids\"],\n", + " max_length=token_limit,\n", + " do_sample=True, # Enable sampling\n", + " top_k=top_k, # Set Top-K sampling strategy\n", + " pad_token_id=tokenizer.eos_token_id # Prevent errors if the input length exceeds the model's limit\n", + " )\n", + "\n", + " # Decode the generated tokens back to text\n", + " output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)\n", + "\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + "\n", + "\n", + "# Generate text 5 times using the top_k sampling strategy\n", + "text_gen_with_top_k(\"Write a title\", token_limit=100, top_k=50)\n" + ], + "metadata": { + "id": "K2JUE2IilwNi", + "outputId": "1ffdd099-1e0f-4a66-a7c6-81b43cc643e3", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "Capitรฃes da Areia\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3m1XaCrlMu3Y" + }, + "source": [ + "Try a slight different prompts" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qC-MLxYWM1HU", + "outputId": "f26faf29-ce26-4a5c-e6ff-f2125148249b", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "\n", + "Gemma output:\n", + "user\n", + "Write a music title\n", + "model\n", + "A Sibila\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a poem title\n", + "model\n", + "O V alienรญgena\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a blog title\n", + "model\n", + "Mar Secreto do Palmar\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a movie title\n", + "model\n", + "A Hora da Estrela\n", + "\n", + "Gemma output:\n", + "user\n", + "Write a novel title\n", + "model\n", + "Os Maias\n" + ] + } + ], + "source": [ + "text_gen_with_top_k(\"Write a music title\")\n", + "text_gen_with_top_k(\"Write a poem title\")\n", + "text_gen_with_top_k(\"Write a blog title\")\n", + "text_gen_with_top_k(\"Write a movie title\")\n", + "text_gen_with_top_k(\"Write a novel title\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aEptDCED9tVp" + }, + "source": [ + "## Publish your model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "T3Qhrlyy5ReL" + }, + "source": [ + "Lets save our model. It takes some time (~11 minutes) as it is a very large file" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4TcvzBH995FE", + "outputId": "26a4be25-9a51-41f9-ad28-ef1708bf440f", + "colab": { + "base_uri": "https://localhost:8080/" + } + }, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "('./my_gemma2_lt_pt/tokenizer_config.json',\n", + " './my_gemma2_lt_pt/special_tokens_map.json',\n", + " './my_gemma2_lt_pt/tokenizer.model',\n", + " './my_gemma2_lt_pt/added_tokens.json',\n", + " './my_gemma2_lt_pt/tokenizer.json')" + ] + }, + "metadata": {}, + "execution_count": 18 + } + ], + "source": [ + "# Define the model name (used for both model and tokenizer)\n", + "my_model_name = \"my_gemma2_lt_pt\"\n", + "\n", + "# Save the fine-tuned model to the specified directory\n", + "gemma_lm.save_pretrained(f\"./{my_model_name}\", token=access_token)\n", + "\n", + "# # Save the tokenizer to the same directory\n", + "tokenizer.save_pretrained(f\"./{my_model_name}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xQ4de1a79zy0" + }, + "source": [ + "## Publishing on Hugging Face\n", + "\n", + "To publish your model on Hugging Face, you'll need your hugging face user (`HF_USER`) and an access token with write permission (`HF_TOKEN`) to the your secret keys." + ] + }, + { + "cell_type": "code", + "source": [ + "# Upload the model to Hugging Face Hub\n", + "my_model_name = \"my_gemma2_pt\"\n", + "writeToken = userdata.get(\"HF_WRITE_TOKEN\")\n", + "hf_repo_id = f\"{my_hf_username}/{my_model_name}\" # Correct format\n", + "gemma_lm.push_to_hub(hf_repo_id, token=writeToken)" + ], + "metadata": { + "id": "AK31-LuXpwen", + "outputId": "e96865e3-4e8c-4acf-e256-c5ec31b6f931", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 136, + "referenced_widgets": [ + "84cd1a70535f4272a0206041373ff281", + "325cf823a7244a4b8e25462ae4912ea3", + "325f38ff3fef43cbbd248ff3ff714c43", + "de4fae4a6e3e420d86414a572846a9e0", + "4d729ba11c9442528af79d6e377a292c", + "49a42d4bcb8b4684a69656ee8b0449df", + "50b06db300ce432c9235071f48a18432", + "a31850a948e344848d43caae277f3200", + "fe5291b55e7147b682b671dad99d16e6", + "b485e94c1509478da6dd7a04c41e055d", + "98e4874bfa904312a4a9b00ca2ea8577", + "a163fe0a8b594568bf102caf54856484", + "dd53c0e07382441fbad818b0c3852006", + "3c316ae120e445c8b4bbc8260cf14ab4", + "9387a006080845ce9ef9aeb41fbd6c6e", + "c49cd450c3d24511a1efe272820c5f0a", + "99a5ad5c04b4456ba65118367cd2ea5a", + "d74745e493704920b56855e0cb73a39e", + "bd53b02c9808436ead8b213886e76bed", + "3f1bd9e04dc64bca88b99998339df788", + "d082ae773106445aa6ec56167b2a5cb9", + "030123f151674251ac0f0be0f8f713dc" + ] + } + }, + "execution_count": null, + "outputs": [ + { + "output_type": "display_data", + "data": { + "text/plain": [ + "README.md: 0%| | 0.00/5.17k [00:00 Date: Mon, 21 Oct 2024 22:49:45 +0300 Subject: [PATCH 14/43] Rename Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb to Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb Fix Folder. --- ...Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb => Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb (99%) diff --git a/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb similarity index 99% rename from Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb rename to Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb index 71fb523..bcb0600 100644 --- a/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb +++ b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb @@ -6099,4 +6099,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +} From 353bb010ed2c1c821a9e8c95b1f3ac731eb34b43 Mon Sep 17 00:00:00 2001 From: Juyeong Ji Date: Tue, 22 Oct 2024 09:36:40 +0900 Subject: [PATCH 15/43] fix lint --- ...ne_tuning_Gemma_Transformers_Edition.ipynb | 24 +++++++------------ 1 file changed, 9 insertions(+), 15 deletions(-) diff --git a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb index bcb0600..5a2ed1e 100644 --- a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb +++ b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb @@ -1,20 +1,9 @@ { "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "view-in-github", - "colab_type": "text" - }, - "source": [ - "\"Open" - ] - }, { "cell_type": "code", "execution_count": null, "metadata": { - "cellView": "form", "id": "i1PHqD-ZY4-c" }, "outputs": [], @@ -40,7 +29,13 @@ "source": [ "# Workshop: How to Fine-tuning Gemma - Transformers Edition\n", "\n", - "To illustrate fine-tuning the model for a specific task, You'll learn how to condition a Gemma model to answer in a specific language. Let's consider the example of generating a random Portuguese title based on a user's instruction such as \"Write a title\". To make this possible, you will curate a manageable dataset that can be manually processed. This approach is feasible because Gemma 2 has prior knowledge of general Portuguese language patterns, enabling it to adapt to this specific task effectively." + "To illustrate fine-tuning the model for a specific task, You'll learn how to condition a Gemma model to answer in a specific language. Let's consider the example of generating a random Portuguese title based on a user's instruction such as \"Write a title\". To make this possible, you will curate a manageable dataset that can be manually processed. This approach is feasible because Gemma 2 has prior knowledge of general Portuguese language patterns, enabling it to adapt to this specific task effectively.\n", + "\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" ] }, { @@ -1639,8 +1634,7 @@ "metadata": { "accelerator": "GPU", "colab": { - "provenance": [], - "include_colab_link": true + "provenance": [] }, "kernelspec": { "display_name": "Python 3", @@ -6099,4 +6093,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} +} \ No newline at end of file From 983916db97b54336fb9bd6105b1224a4146c05e8 Mon Sep 17 00:00:00 2001 From: Juyeong Ji Date: Tue, 22 Oct 2024 09:37:43 +0900 Subject: [PATCH 16/43] fix nbfmt --- ...ne_tuning_Gemma_Transformers_Edition.ipynb | 5307 +---------------- 1 file changed, 330 insertions(+), 4977 deletions(-) diff --git a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb index 5a2ed1e..02caf27 100644 --- a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb +++ b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb @@ -4,6 +4,7 @@ "cell_type": "code", "execution_count": null, "metadata": { + "cellView": "form", "id": "i1PHqD-ZY4-c" }, "outputs": [], @@ -92,30 +93,14 @@ }, { "cell_type": "code", - "source": [ - "import os\n", - "from google.colab import userdata, drive\n", - "from huggingface_hub import login\n", - "\n", - "login(userdata.get(\"HUGGING_FACE\"))\n", - "\n", - "access_token = userdata.get(\"HUGGING_FACE\")\n", - "my_hf_username = userdata.get(\"HUGGING_FACE_UN\")\n", - "os.environ[\"HF_USER\"] = my_hf_username\n", - "os.environ[\"HF_TOKEN\"] = userdata.get(\"HUGGING_FACE\")" - ], + "execution_count": null, "metadata": { - "id": "TT7GexJnZZCj", - "outputId": "5f5cc93f-d9c8-4bb4-d828-b670ed352b35", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "TT7GexJnZZCj" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.\n", "Token is valid (permission: read).\n", @@ -123,6 +108,18 @@ "Login successful\n" ] } + ], + "source": [ + "import os\n", + "from google.colab import userdata, drive\n", + "from huggingface_hub import login\n", + "\n", + "login(userdata.get(\"HUGGING_FACE\"))\n", + "\n", + "access_token = userdata.get(\"HUGGING_FACE\")\n", + "my_hf_username = userdata.get(\"HUGGING_FACE_UN\")\n", + "os.environ[\"HF_USER\"] = my_hf_username\n", + "os.environ[\"HF_TOKEN\"] = userdata.get(\"HUGGING_FACE\")" ] }, { @@ -138,34 +135,14 @@ }, { "cell_type": "code", - "source": [ - "!pip install transformers torch\n", - "# Set the backbend before importing Keras\n", - "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n", - "# Avoid memory fragmentation on JAX backend.\n", - "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n", - "\n", - "# Training Configurations\n", - "token_limit = 128\n", - "num_data_limit = 100\n", - "lora_name = \"my_lora\"\n", - "lora_rank = 4\n", - "lr_value = 1e-3\n", - "train_epoch = 5\n", - "model_id = \"google/gemma-2-2b-it\"" - ], + "execution_count": null, "metadata": { - "id": "WNn86PiiXTNf", - "outputId": "5ac7a598-0773-4596-a393-ba53cbfd2470", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "WNn86PiiXTNf" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.44.2)\n", "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.4.1+cu121)\n", @@ -192,6 +169,22 @@ "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch) (1.3.0)\n" ] } + ], + "source": [ + "!pip install transformers torch\n", + "# Set the backbend before importing Keras\n", + "os.environ[\"KERAS_BACKEND\"] = \"jax\"\n", + "# Avoid memory fragmentation on JAX backend.\n", + "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"1.00\"\n", + "\n", + "# Training Configurations\n", + "token_limit = 128\n", + "num_data_limit = 100\n", + "lora_name = \"my_lora\"\n", + "lora_rank = 4\n", + "lr_value = 1e-3\n", + "train_epoch = 5\n", + "model_id = \"google/gemma-2-2b-it\"" ] }, { @@ -217,182 +210,10 @@ }, { "cell_type": "code", - "source": [ - "from transformers import AutoModelForCausalLM, AutoTokenizer\n", - "import time\n", - "\n", - "# Load a pretrained model and tokenizer from Hugging Face\n", - "gemma_lm = AutoModelForCausalLM.from_pretrained(model_id, token=access_token)\n", - "tokenizer = AutoTokenizer.from_pretrained(model_id, token=access_token)\n", - "\n", - "# Summarize the model\n", - "print(gemma_lm)\n", - "\n", - "tick_start = 0\n", - "\n", - "def tick():\n", - " global tick_start\n", - " tick_start = time.time()\n", - "\n", - "def tock():\n", - " print(f\"TOTAL TIME ELAPSED: {time.time() - tick_start:.2f}s\")\n", - "\n", - "def text_gen(prompt, token_limit=100): # You can set your token limit\n", - " tick()\n", - "\n", - " # Format input, same as your original code\n", - " input_text = f\"user\\n{prompt}\\nmodel\\n\"\n", - "\n", - " # Tokenize input\n", - " inputs = tokenizer(input_text, return_tensors=\"pt\")\n", - "\n", - " # Generate text using the model\n", - " output_tokens = gemma_lm.generate(\n", - " inputs[\"input_ids\"],\n", - " max_length=token_limit,\n", - " pad_token_id=tokenizer.eos_token_id # Prevent errors if the input length exceeds the model's limit\n", - " )\n", - "\n", - " # Decode the generated tokens back to text\n", - " output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)\n", - "\n", - " print(\"\\nGemma output:\")\n", - " print(output)\n", - "\n", - " tock()" - ], + "execution_count": null, "metadata": { - "id": "ywcDWVhAXb_9", - "outputId": "d9af537b-6c7a-4018-ada6-7b5dc58328d8", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 880, - "referenced_widgets": [ - "5dcc5fb3ad9e41eeb877fdd04f09ba7d", - "eeda0cc4233f41c7a1bb077b335922af", - "7786f72663544d3b964f8706ee4c87db", - "16489924a7194b6780f51cb3fa808c5e", - "8a87e9b8ba9d41ce8b44f2fef3244a81", - "e84b2d9e1e9e46c28e788385d64bc7c6", - "60c77f65806f4ff688453439fe7c9d83", - "de9354062f994aaabb104890709fc34a", - "9ae9adefe60b4511a90cdb2b3ef6e03d", - "3a29897fd3b3454ca752f63f45007d7f", - "1ef2aa04eaba4620a49c748ca4380d99", - "f4188acdf47f4cce86c02cc0d2bf46c7", - "54c456ec68684e57aa57aa7715921bfb", - "4ad2c31035574f26bf6118ef0d1413a9", - "a12cf321b8ae4075a4c3b0677d7bdf45", - "f8234ec2555b4e5684816aed66b63651", - "d816729082e94c41988d467654ef7841", - "b2d33d479a634d5c807e2a1860f4bc8c", - "b542562fcd6c466c9aa7f12b1b881207", - "2e8af71a4d85433a8e68f5c3737d7fb9", - "cdf16eb2e8ca487b9e9c7de16381c5c8", - "15332ef69c9244ccaaf3030dc9cb113a", - "eee1dd11b43e435cb226ee6df594291d", - "cc3c5ae5ebb74b2dbc81333533cc78cf", - "2f79a5c2a23b4808a181904593072dc5", - "a1c592e1355844adaa9ea80718499e4c", - "9805d9bec3f14c41a7a87fced47bb6ca", - "aed3fca1af5648e3b681d0c44f9c4f86", - "5960f282615e4932af3d9b7f69de2efa", - "151d7e95158d4d8aa2faf2886c8e0604", - "b5e8f1d688ea4694ac83a1bf89e27ae0", - "02ddb4d4c134415fa89a98984ae45c5f", - "7ba48bd1bbb148ff9dc7a2c33b835a35", - "9dc263e2dbfa4d9d9a56d62a69a57a8f", - "6a3171fe96744527805d8dc9eb159006", - "0bd0296c84a248259b5c48ac4e6bb760", - "20bc2639f53d4701aeafdb30e60f47da", - "372294f99f90486391b77296724916cc", - "2d9816c05a3544c0afc5a41c74ad6179", - "52132d841f1f48dd8c1d87ee0d308764", - "2fd5a5bfe878469c8f6b710c672fda9c", - "3a0c665acdbb4c83918c0ac469114235", - "eebc1808719c45b3bc7ad525798ce120", - "5d5a094d9cb545788f75992cbd859257", - "ecb1c6f7effa46b296e3574cddcb88de", - "cf5ec90af2944a63a0f83d5520307b82", - "28c7d5e1e84848c29b3d533930ccc198", - "b2831f93a9d148f4871c435382c6302e", - "9a827d1a0a424d6381b3e1eaac796b1e", - "afda84f0f4a445ee9dee2ccf047ce5ce", - "fad3963f838c44d98af0fdebb787741c", - "2dbc25a62598422b9c0768e109f8fd92", - "f97ae90efc3f44498db9203ef90d232e", - "707955f6553648fbb75ea65f2df2eaf6", - "c2fd92147e044b96ace6530ae67ca181", - "64324fda5c814874aaa2e97fa78ab3f0", - "abc4f6598a1f4c57a74a55f854ddc66d", - "6bfe81a71ff54a1390ff1d90d7d78864", - "25c94e5770e84ecea0e0f68f5bd64905", - "d15f3939e8b143a5b2deeb82435f9d26", - "a006f1ee609b47dda37389b5478a803d", - "0254a7b1482545cba26b5708a8920b34", - "977e2dc71b224a05823aa8120fa472bd", - "f6bdb5a59a044f4bb7332068179adce1", - "ecb124db56ea4925b83e91206c53f89b", - "e29c06ee25f84198bbea49aa0b19bb65", - "43bccde2f2804b089e4e40081058855f", - "f96a8568f4dc409090a123ff63766e58", - "163aee402b454733b1e74d43834849fb", - "8ebf2a32eac047cf9367f615ec01f0a8", - "760de7e3cef94be8b82cf8fc99385844", - "22d61c177e4e4d09a62ac08785682151", - "7ad0cea5c87e40da9b1dd1966b222f5a", - "a1e5eef9348f4bfeb456e5f096301778", - "4d94e337748343bb8e26d7ad7d361906", - "bc1dcd6dcd16462999359b579146c856", - "7c79134c76334c8680e81b58b9f75819", - "488f3d23e6ee4370b0e3750c4610acc1", - "5b1b61f335594a4487667b9d60395218", - "350ac704ca744a859527627fa92f9aa7", - "5a25dc05639c4cb282b1a2c4bbfdb572", - "4aaae0e141b14af78d9207b5f99a4516", - "182f4c00d1e94a4081057f34462704b7", - "62bb5bafdf2e47628a3ff1b114bac45a", - "79fac9dcae554d6f885608bd29fb904e", - "3b43ab0605644de9a852218edf3516fb", - "67261f6d01b14c50a4d7413ae8a4d113", - "110e4a8116e847dc9e4cea9a9df9e2b5", - "9ac1990111d64f919644a292b4ea2d36", - "494574a90f844172b5dce66aba248564", - "6b30ac7eab3346e8978274b8bcb03371", - "bdee91edaed5469a98ed808735bb30b0", - "eb7cfbc69c4848b3b4274203a70b6f1e", - "adb76c55b90d4f5aadd92d51b526fc27", - "5dd424f1051645abb00a3a3aa7d93990", - "018cb6becc764afb8bc6d9e9f2196fb9", - "bdd09cce6f4d4cf5b555678537c029c9", - "41ffd80053b6402f8aba20e1413e30cd", - "015fabd8f4934029900eec53ef02bdb8", - "b91c257be15341179ce8a262b3962ef0", - "e207f4774be44704a804dfacd628ad5a", - "559c60d219f24907b2b3b8e7c78deed8", - "88189b7d8769412685e197c8c8b7ec5c", - "2c8f45c2c1624865a698eb8c5527b0e4", - "d55c5f99d58942c2ba0506090f9755c2", - "9b5c6869f25945fa9424593255af07fc", - "435dfab7d6c94fb7b8b1a9dbb81bdbe7", - "9c0fa484394142c3beee7e272f626b08", - "3ad6acbc511b4417b5ca4e56a8688828", - "2d693baa6ea04b58965141813bd0ceca", - "379696127cc841a08cb6ca5000180032", - "9d03f9f959254badae3c94b71e84f2cc", - "65d1d113a0a8446c83563d9aefd8b112", - "9f61fd282f104094bc821e8a4b9f27ce", - "fd0fc02ef4b545c0a459527aca8bc81e", - "01cf0405f63a4d38ad8d1fe31721806f", - "ae20add968524517a261a81be45cde48", - "dc76053f6ae847469aab55949ad182be", - "4d98195ba52f48a29a237777eeef35cf", - "ffeafe7d69484d64af7484311e32703d", - "b66e7b11b2404e3daeaed1272994adb5" - ] - } + "id": "ywcDWVhAXb_9" }, - "execution_count": null, "outputs": [ { "data": { @@ -479,78 +300,78 @@ "output_type": "display_data" }, { - "output_type": "display_data", "data": { - "text/plain": [ - "generation_config.json: 0%| | 0.00/187 [00:00user\\n{prompt}\\nmodel\\n\"\n", + "\n", + " # Tokenize input\n", + " inputs = tokenizer(input_text, return_tensors=\"pt\")\n", + "\n", + " # Generate text using the model\n", + " output_tokens = gemma_lm.generate(\n", + " inputs[\"input_ids\"],\n", + " max_length=token_limit,\n", + " pad_token_id=tokenizer.eos_token_id # Prevent errors if the input length exceeds the model's limit\n", + " )\n", + "\n", + " # Decode the generated tokens back to text\n", + " output = tokenizer.decode(output_tokens[0], skip_special_tokens=True)\n", + "\n", + " print(\"\\nGemma output:\")\n", + " print(output)\n", + "\n", + " tock()" ] }, { @@ -631,16 +496,12 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "xR1riO7NURFF", - "outputId": "f9277cf5-9ea4-4509-a50b-260941c24ae3", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "xR1riO7NURFF" }, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "[2, 177383, 235265, 6235, 3004, 2190, 26809, 235260, 235442, 235290, 545, 235265, 687, 11030, 5365, 14693, 12318, 43897, 235265]\n", " 2 -> \n", @@ -728,16 +589,12 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "ZiS-KU9osh_N", - "outputId": "dedf9024-ff52-4a36-e586-6fea9139a53f", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "ZiS-KU9osh_N" }, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "15\n", "user\n", @@ -807,61 +664,25 @@ }, { "cell_type": "code", - "source": [ - "!pip install datasets" - ], + "execution_count": null, "metadata": { "id": "ZS9zT92tiKHu" }, - "execution_count": null, - "outputs": [] + "outputs": [], + "source": [ + "!pip install datasets" + ] }, { "cell_type": "code", - "source": [ - "# from datasets import load_dataset\n", - "\n", - "# # Load the dataset\n", - "# ds = load_dataset(\"bebechien/korean_cake_boss\", split=\"train\")\n", - "# print(ds)\n", - "\n", - "# # Prepare the dataset for tokenization\n", - "# train = []\n", - "\n", - "# # Iterate through the dataset and format the prompts\n", - "# for x in ds:\n", - "# # Create the formatted input-output text\n", - "# item = f\"user\\n๋‹ค์Œ์— ๋Œ€ํ•œ ์ด๋ฉ”์ผ ๋‹ต์žฅ์„ ์ž‘์„ฑํ•ด์ค˜.\\n\\\"{x['input']}\\\"\\nmodel\\n{x['output']}\"\n", - "\n", - "# # Tokenize the item and get its length\n", - "# length = len(tokenizer(item)[\"input_ids\"])\n", - "# print(length)\n", - "# # Skip if the tokenized item is longer than the token limit\n", - "# if length < token_limit:\n", - "# train.append(item)\n", - "\n", - "# # Stop if we have reached the desired data limit\n", - "# if len(train) >= num_data_limit:\n", - "# break\n", - "\n", - "# # Output the results\n", - "# print(f\"Number of training examples: {len(train)}\")\n", - "# print(f\"First example: {train[0]}\")\n", - "# print(f\"Second example: {train[1]}\")\n", - "# print(f\"Third example: {train[2]}\")" - ], + "execution_count": null, "metadata": { - "id": "Vh9s8m_PgoAH", - "outputId": "ccdf0c19-c3b5-40d5-d7ae-dc7a2c59fe0b", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "Vh9s8m_PgoAH" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "Dataset({\n", " features: ['input', 'output'],\n", @@ -923,6 +744,38 @@ "[๊ฐ€๊ฒŒ ์ด๋ฆ„] ๋“œ๋ฆผ\n" ] } + ], + "source": [ + "# from datasets import load_dataset\n", + "\n", + "# # Load the dataset\n", + "# ds = load_dataset(\"bebechien/korean_cake_boss\", split=\"train\")\n", + "# print(ds)\n", + "\n", + "# # Prepare the dataset for tokenization\n", + "# train = []\n", + "\n", + "# # Iterate through the dataset and format the prompts\n", + "# for x in ds:\n", + "# # Create the formatted input-output text\n", + "# item = f\"user\\n๋‹ค์Œ์— ๋Œ€ํ•œ ์ด๋ฉ”์ผ ๋‹ต์žฅ์„ ์ž‘์„ฑํ•ด์ค˜.\\n\\\"{x['input']}\\\"\\nmodel\\n{x['output']}\"\n", + "\n", + "# # Tokenize the item and get its length\n", + "# length = len(tokenizer(item)[\"input_ids\"])\n", + "# print(length)\n", + "# # Skip if the tokenized item is longer than the token limit\n", + "# if length < token_limit:\n", + "# train.append(item)\n", + "\n", + "# # Stop if we have reached the desired data limit\n", + "# if len(train) >= num_data_limit:\n", + "# break\n", + "\n", + "# # Output the results\n", + "# print(f\"Number of training examples: {len(train)}\")\n", + "# print(f\"First example: {train[0]}\")\n", + "# print(f\"Second example: {train[1]}\")\n", + "# print(f\"Third example: {train[2]}\")" ] }, { @@ -973,53 +826,14 @@ }, { "cell_type": "code", - "source": [ - "!pip install peft\n", - "from peft import get_peft_model, LoraConfig, TaskType\n", - "import torch.nn as nn\n", - "\n", - "lora_config = LoraConfig(\n", - " task_type=TaskType.CAUSAL_LM,\n", - " r=lora_rank, # Using your predefined lora_rank\n", - " lora_alpha=32,\n", - " lora_dropout=0.1\n", - ")\n", - "gemma_lm = get_peft_model(gemma_lm, lora_config) # Enable LoRA for the model\n", - "\n", - "print(gemma_lm) # Hugging Face models don't have a summary method; use print() instead\n", - "\n", - "tokenizer.model_max_length = token_limit # Set token limit in the tokenizer\n", - "\n", - "from transformers import AdamW\n", - "\n", - "optimizer_grouped_parameters = [\n", - " {'params': [p for n, p in gemma_lm.named_parameters() if not any(nd in n for nd in [\"bias\", \"LayerNorm.weight\"])], 'weight_decay': 0.01},\n", - " {'params': [p for n, p in gemma_lm.named_parameters() if any(nd in n for nd in [\"bias\", \"LayerNorm.weight\"])], 'weight_decay': 0.0}\n", - "]\n", - "optimizer = AdamW(optimizer_grouped_parameters, lr=lr_value) # Use AdamW optimizer\n", - "\n", - "\n", - "loss_fn = nn.CrossEntropyLoss() # Define the loss function\n", - "\n", - "def forward_pass(input_text):\n", - " inputs = tokenizer(input_text, return_tensors=\"pt\", max_length=token_limit, truncation=True)\n", - " outputs = gemma_lm(**inputs, labels=inputs[\"input_ids\"])\n", - " loss = outputs.loss\n", - " return loss\n", - "\n" - ], + "execution_count": null, "metadata": { - "id": "YQiQxLFKfyzx", - "outputId": "5a5b1398-61cd-48ba-d354-71e4eee37213", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "YQiQxLFKfyzx" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "Collecting peft\n", " Downloading peft-0.13.2-py3-none-any.whl.metadata (13 kB)\n", @@ -1115,13 +929,47 @@ ] }, { - "output_type": "stream", "name": "stderr", + "output_type": "stream", "text": [ "/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:591: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n", " warnings.warn(\n" ] } + ], + "source": [ + "!pip install peft\n", + "from peft import get_peft_model, LoraConfig, TaskType\n", + "import torch.nn as nn\n", + "\n", + "lora_config = LoraConfig(\n", + " task_type=TaskType.CAUSAL_LM,\n", + " r=lora_rank, # Using your predefined lora_rank\n", + " lora_alpha=32,\n", + " lora_dropout=0.1\n", + ")\n", + "gemma_lm = get_peft_model(gemma_lm, lora_config) # Enable LoRA for the model\n", + "\n", + "print(gemma_lm) # Hugging Face models don't have a summary method; use print() instead\n", + "\n", + "tokenizer.model_max_length = token_limit # Set token limit in the tokenizer\n", + "\n", + "from transformers import AdamW\n", + "\n", + "optimizer_grouped_parameters = [\n", + " {'params': [p for n, p in gemma_lm.named_parameters() if not any(nd in n for nd in [\"bias\", \"LayerNorm.weight\"])], 'weight_decay': 0.01},\n", + " {'params': [p for n, p in gemma_lm.named_parameters() if any(nd in n for nd in [\"bias\", \"LayerNorm.weight\"])], 'weight_decay': 0.0}\n", + "]\n", + "optimizer = AdamW(optimizer_grouped_parameters, lr=lr_value) # Use AdamW optimizer\n", + "\n", + "\n", + "loss_fn = nn.CrossEntropyLoss() # Define the loss function\n", + "\n", + "def forward_pass(input_text):\n", + " inputs = tokenizer(input_text, return_tensors=\"pt\", max_length=token_limit, truncation=True)\n", + " outputs = gemma_lm(**inputs, labels=inputs[\"input_ids\"])\n", + " loss = outputs.loss\n", + " return loss\n" ] }, { @@ -1139,81 +987,21 @@ }, { "cell_type": "code", - "source": [ - "import torch\n", - "import os\n", - "import matplotlib.pyplot as plt\n", - "\n", - "# Define a custom callback-like function to handle actions at the end of each epoch\n", - "class CustomCallback:\n", - " def __init__(self, model, lora_name, lora_rank, text_gen):\n", - " self.model = model\n", - " self.lora_name = lora_name\n", - " self.lora_rank = lora_rank\n", - " self.text_gen = text_gen # text_gen function for evaluation\n", - "\n", - " def on_epoch_end(self, epoch):\n", - " # Save LoRA weights at the end of each epoch\n", - " model_name = f\"./{self.lora_name}_{self.lora_rank}_epoch{epoch+1}.lora.pt\"\n", - " self.model.save_pretrained(model_name, token=access_token) # Save model with LoRA weights locally\n", - "\n", - " # Evaluate the model using text generation\n", - " print(f\"Epoch {epoch + 1} finished. Running evaluation:\")\n", - " self.text_gen(\"Write a title\")\n", - " self.text_gen(\"Write a poem\")\n", - "\n", - "# Assuming train is your DataLoader and gemma_lm is your model\n", - "callback = CustomCallback(gemma_lm, lora_name, lora_rank, text_gen)\n", - "\n", - "# Training loop with callback-like behavior\n", - "losses = []\n", - "for epoch in range(train_epoch):\n", - " epoch_loss = 0\n", - " for batch in train: # Assuming `train` is a DataLoader or similar iterable\n", - " optimizer.zero_grad()\n", - "\n", - " inputs = tokenizer(batch, return_tensors=\"pt\", max_length=token_limit, truncation=True, padding=True)\n", - " labels = inputs[\"input_ids\"]\n", - " outputs = gemma_lm(**inputs, labels=labels)\n", - " loss = outputs.loss\n", - "\n", - " loss.backward()\n", - " optimizer.step()\n", - "\n", - " epoch_loss += loss.item()\n", - "\n", - " losses.append(epoch_loss / len(train)) # Store average loss per epoch\n", - "\n", - " # Run custom callback at the end of each epoch\n", - " callback.on_epoch_end(epoch)\n", - "\n", - "# Plot training loss over epochs\n", - "plt.plot(losses)\n", - "plt.xlabel(\"Epoch\")\n", - "plt.ylabel(\"Loss\")\n", - "plt.title(\"Training Loss Over Epochs\")\n", - "plt.show()" - ], + "execution_count": null, "metadata": { - "id": "YKpmDIfXh1Kx", - "outputId": "2422fc27-260d-4e7d-ce4b-bfa4234bec7d", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - } + "id": "YKpmDIfXh1Kx" }, - "execution_count": null, "outputs": [ { - "output_type": "stream", "name": "stderr", + "output_type": "stream", "text": [ "The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.\n" ] }, { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "Epoch 1 finished. Running evaluation:\n", "\n", @@ -1293,15 +1081,70 @@ ] }, { - "output_type": "display_data", "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTlUlEQVR4nO3deVwTZ/4H8E/CEQ5JAJVLIqhYUFFUPADPVq1V60pr16MqaD22W+zqtt2ubH+97Hbpsd3eq1arVK31rLr1xgNPqIpi0aotlVMIqBzhPpL5/YGkTYFwCEwSPu/Xa15tJs8k34ch5sPMM89IBEEQQERERGQmpGIXQERERNSaGG6IiIjIrDDcEBERkVlhuCEiIiKzwnBDREREZoXhhoiIiMwKww0RERGZFYYbIiIiMisMN0RERGRWGG6IjMT8+fPh7e3dom3feOMNSCSS1i2IqBG1v3d3794VuxQiPQw3RI2QSCRNWmJjY8UuVRTz589Hp06dxC6jSQRBwKZNmzB69Gg4OjrCzs4O/fv3x8qVK1FSUiJ2eXXUhoeGFpVKJXaJREbJUuwCiIzdpk2b9B5v3LgRMTExddb36dPngd5n7dq10Gq1Ldr2//7v/7BixYoHen9zp9Fo8PTTT2P79u0YNWoU3njjDdjZ2eH06dN48803sWPHDhw9ehSurq5il1rHqlWr6g2Qjo6O7V8MkQlguCFqxNy5c/Uex8fHIyYmps763ystLYWdnV2T38fKyqpF9QGApaUlLC35cTbkvffew/bt2/HSSy/h/fff161fsmQJZsyYgdDQUMyfPx8HDx5s17qa8nvy1FNPoUuXLu1UEZHp42kpolYwduxY+Pv7IyEhAaNHj4adnR3+8Y9/AAD27t2LKVOmwMPDAzKZDL169cJbb70FjUaj9xq/H3OTmpoKiUSCf//73/jiiy/Qq1cvyGQyDB06FBcuXNDbtr4xNxKJBEuXLsWePXvg7+8PmUyGfv364dChQ3Xqj42NxZAhQ2BjY4NevXphzZo1rT6OZ8eOHQgMDIStrS26dOmCuXPn4vbt23ptVCoVFixYAE9PT8hkMri7u2PatGlITU3Vtbl48SImTpyILl26wNbWFj169MAzzzxj8L3Lysrw/vvv46GHHkJUVFSd56dOnYrw8HAcOnQI8fHxAIDHH38cPXv2rPf1goODMWTIEL11mzdv1vXP2dkZs2bNQkZGhl4bQ78nDyI2NhYSiQTbtm3DP/7xD7i5ucHe3h5/+MMf6tQANG1fAMCNGzcwY8YMdO3aFba2tvD19cUrr7xSp11BQQHmz58PR0dHKBQKLFiwAKWlpXptYmJiMHLkSDg6OqJTp07w9fVtlb4T1Yd/6hG1knv37mHSpEmYNWsW5s6dqzu9ER0djU6dOuGFF15Ap06dcPz4cbz22mtQq9V6RxAasmXLFhQVFeFPf/oTJBIJ3nvvPTz55JO4detWo0d7zpw5g2+//RbPPfccHBwc8Mknn2D69OlIT09H586dAQCXL1/GY489Bnd3d7z55pvQaDRYuXIlunbt+uA/lPuio6OxYMECDB06FFFRUcjJycHHH3+Ms2fP4vLly7rTK9OnT8e1a9fw/PPPw9vbG7m5uYiJiUF6erru8aOPPoquXbtixYoVcHR0RGpqKr799ttGfw75+flYtmxZg0e4wsLCsGHDBuzbtw9BQUGYOXMmwsLCcOHCBQwdOlTXLi0tDfHx8Xr77u2338arr76KGTNmYNGiRbhz5w4+/fRTjB49Wq9/QMO/J4bk5eXVWWdpaVnntNTbb78NiUSCv//978jNzcVHH32E8ePHIzExEba2tgCavi9++OEHjBo1ClZWVliyZAm8vb3xyy+/4LvvvsPbb7+t974zZsxAjx49EBUVhUuXLmHdunVwcXHBu+++CwC4du0aHn/8cQwYMAArV66ETCZDcnIyzp4922jfiVpEIKJmiYiIEH7/0RkzZowAQFi9enWd9qWlpXXW/elPfxLs7OyE8vJy3brw8HDBy8tL9zglJUUAIHTu3FnIy8vTrd+7d68AQPjuu+90615//fU6NQEQrK2theTkZN26K1euCACETz/9VLdu6tSpgp2dnXD79m3dup9//lmwtLSs85r1CQ8PF+zt7Rt8vrKyUnBxcRH8/f2FsrIy3fp9+/YJAITXXntNEARByM/PFwAI77//foOvtXv3bgGAcOHChUbr+q2PPvpIACDs3r27wTZ5eXkCAOHJJ58UBEEQCgsLBZlMJrz44ot67d577z1BIpEIaWlpgiAIQmpqqmBhYSG8/fbbeu2SkpIES0tLvfWGfk/qU7tf61t8fX117U6cOCEAELp16yao1Wrd+u3btwsAhI8//lgQhKbvC0EQhNGjRwsODg66ftbSarV16nvmmWf02jzxxBNC586ddY8//PBDAYBw586dJvWb6EHxtBRRK5HJZFiwYEGd9bV/MQNAUVER7t69i1GjRqG0tBQ3btxo9HVnzpwJJycn3eNRo0YBAG7dutXotuPHj0evXr10jwcMGAC5XK7bVqPR4OjRowgNDYWHh4eunY+PDyZNmtTo6zfFxYsXkZubi+eeew42Nja69VOmTIGfnx/2798PoObnZG1tjdjYWOTn59f7WrVHFfbt24eqqqom11BUVAQAcHBwaLBN7XNqtRoAIJfLMWnSJGzfvh2CIOjabdu2DUFBQejevTsA4Ntvv4VWq8WMGTNw9+5d3eLm5obevXvjxIkTeu/T0O+JIbt27UJMTIzesmHDhjrtwsLC9Pr41FNPwd3dHQcOHADQ9H1x584dnDp1Cs8884yun7XqO1X57LPP6j0eNWoU7t27p/tZ1u63vXv3tnjQPFFzMNwQtZJu3brB2tq6zvpr167hiSeegEKhgFwuR9euXXWDkQsLCxt93d9/udQGnYYCgKFta7ev3TY3NxdlZWXw8fGp066+dS2RlpYGAPD19a3znJ+fn+55mUyGd999FwcPHoSrqytGjx6N9957T+9y5zFjxmD69Ol488030aVLF0ybNg0bNmxARUWFwRpqv/BrQ0596gtAM2fOREZGBuLi4gAAv/zyCxISEjBz5kxdm59//hmCIKB3797o2rWr3nL9+nXk5ubqvU9DvyeGjB49GuPHj9dbgoOD67Tr3bu33mOJRAIfHx/dmKWm7ova8Ovv79+k+hr7HZ05cyZGjBiBRYsWwdXVFbNmzcL27dsZdKjNMNwQtZLfHqGpVVBQgDFjxuDKlStYuXIlvvvuO8TExOjGIjTlH3cLC4t61//2aEJbbCuG5cuX46effkJUVBRsbGzw6quvok+fPrh8+TKAmi/rnTt3Ii4uDkuXLsXt27fxzDPPIDAwEMXFxQ2+bu1l+j/88EODbWqf69u3r27d1KlTYWdnh+3btwMAtm/fDqlUij/+8Y+6NlqtFhKJBIcOHapzdCUmJgZr1qzRe5/6fk9MXWO/Z7a2tjh16hSOHj2KefPm4YcffsDMmTMxYcKEOgPriVoDww1RG4qNjcW9e/cQHR2NZcuW4fHHH8f48eP1TjOJycXFBTY2NkhOTq7zXH3rWsLLywsAcPPmzTrP3bx5U/d8rV69euHFF1/EkSNHcPXqVVRWVuKDDz7QaxMUFIS3334bFy9exNdff41r165h69atDdZQe5XOli1bGvwy3bhxI4Caq6Rq2dvb4/HHH8eOHTug1Wqxbds2jBo1Su8UXq9evSAIAnr06FHn6Mr48eMRFBTUyE+o9fz88896jwVBQHJysu4qvKbui9qrxK5evdpqtUmlUowbNw7/+c9/8OOPP+Ltt9/G8ePH65y2I2oNDDdEbaj2L9rfHimprKzEf//7X7FK0mNhYYHx48djz549yMrK0q1PTk5utflehgwZAhcXF6xevVrv9NHBgwdx/fp1TJkyBUDNfC/l5eV62/bq1QsODg667fLz8+scdRo4cCAAGDw1ZWdnh5deegk3b96s91Lm/fv3Izo6GhMnTqwTRmbOnImsrCysW7cOV65c0TslBQBPPvkkLCws8Oabb9apTRAE3Lt3r8G6WtvGjRv1Tr3t3LkT2dnZuvFTTd0XXbt2xejRo7F+/Xqkp6frvUdLjvrVd7VXU/YbUUvxUnCiNhQSEgInJyeEh4fjL3/5CyQSCTZt2mRUp4XeeOMNHDlyBCNGjMCf//xnaDQafPbZZ/D390diYmKTXqOqqgr//Oc/66x3dnbGc889h3fffRcLFizAmDFjMHv2bN3lx97e3vjrX/8KAPjpp58wbtw4zJgxA3379oWlpSV2796NnJwczJo1CwDw1Vdf4b///S+eeOIJ9OrVC0VFRVi7di3kcjkmT55ssMYVK1bg8uXLePfddxEXF4fp06fD1tYWZ86cwebNm9GnTx989dVXdbabPHkyHBwc8NJLL8HCwgLTp0/Xe75Xr1745z//icjISKSmpiI0NBQODg5ISUnB7t27sWTJErz00ktN+jk2ZOfOnfXOUDxhwgS9S8mdnZ0xcuRILFiwADk5Ofjoo4/g4+ODxYsXA6iZKLIp+wIAPvnkE4wcORKDBw/GkiVL0KNHD6SmpmL//v1N/r2otXLlSpw6dQpTpkyBl5cXcnNz8d///heenp4YOXJky34oRIaIco0WkQlr6FLwfv361dv+7NmzQlBQkGBrayt4eHgIL7/8snD48GEBgHDixAldu4YuBa/v0mgAwuuvv6573NCl4BEREXW29fLyEsLDw/XWHTt2TBg0aJBgbW0t9OrVS1i3bp3w4osvCjY2Ng38FH4VHh7e4OXKvXr10rXbtm2bMGjQIEEmkwnOzs7CnDlzhMzMTN3zd+/eFSIiIgQ/Pz/B3t5eUCgUwvDhw4Xt27fr2ly6dEmYPXu20L17d0EmkwkuLi7C448/Lly8eLHROgVBEDQajbBhwwZhxIgRglwuF2xsbIR+/foJb775plBcXNzgdnPmzBEACOPHj2+wza5du4SRI0cK9vb2gr29veDn5ydEREQIN2/e1LUx9HtSH0OXgv/296f2UvBvvvlGiIyMFFxcXARbW1thypQpdS7lFoTG90Wtq1evCk888YTg6Ogo2NjYCL6+vsKrr75ap77fX+K9YcMGAYCQkpIiCELN79e0adMEDw8PwdraWvDw8BBmz54t/PTTT03+WRA1h0QQjOhPSCIyGqGhobh27VqdcRxkfGJjY/Hwww9jx44deOqpp8Quh0h0HHNDRCgrK9N7/PPPP+PAgQMYO3asOAURET0AjrkhIvTs2RPz589Hz549kZaWhlWrVsHa2hovv/yy2KURETUbww0R4bHHHsM333wDlUoFmUyG4OBg/Otf/6ozKRwRkSngmBsiIiIyKxxzQ0RERGaF4YaIiIjMSocbc6PVapGVlQUHB4d6725LRERExkcQBBQVFcHDwwNSqeFjMx0u3GRlZUGpVIpdBhEREbVARkYGPD09DbYxmnDzzjvvIDIyEsuWLcNHH31Ub5vo6GgsWLBAb51MJqtzPxpDHBwcANT8cORyeYvrJSIiovajVquhVCp13+OGGEW4uXDhAtasWYMBAwY02lYul+vd0ba5p5Zq28vlcoYbIiIiE9OU733RBxQXFxdjzpw5WLt2LZycnBptL5FI4Obmplt+e9M4IiIiItHDTUREBKZMmYLx48c3qX1xcTG8vLygVCoxbdo0XLt2zWD7iooKqNVqvYWIiIjMl6jhZuvWrbh06RKioqKa1N7X1xfr16/H3r17sXnzZmi1WoSEhCAzM7PBbaKioqBQKHQLBxMTERGZN9FmKM7IyMCQIUMQExOjG2szduxYDBw4sMEBxb9XVVWFPn36YPbs2XjrrbfqbVNRUYGKigrd49oBSYWFhRxzQ0REZCLUajUUCkWTvr9FG1CckJCA3NxcDB48WLdOo9Hg1KlT+Oyzz1BRUQELCwuDr2FlZYVBgwYhOTm5wTYymQwymazV6iYiIiLjJlq4GTduHJKSkvTWLViwAH5+fvj73//eaLABasJQUlISJk+e3FZlEhERkYkRLdw4ODjA399fb529vT06d+6sWx8WFoZu3brpxuSsXLkSQUFB8PHxQUFBAd5//32kpaVh0aJF7V4/ERERGSejmOemIenp6XpTLOfn52Px4sVQqVRwcnJCYGAgzp07h759+4pYJRERERkT0QYUi6U5A5KIiIjIODTn+1v0eW6IiIiIWhPDDREREZkVhhsiIiIyKww3rUhVWI7r2by9AxERkZgYblrJwaRsjHrvOF7ZndR4YyIiImozDDetJNDbCRJIcCm9AAlpeWKXQ0RE1GEx3LQSFwcbhA7yAACsPZUicjVEREQdF8NNK1o0qicA4PCPKqTdKxG5GiIioo6J4aYVPeTqgLG+XSEIwPozPHpDREQkBoabVrb4/tGb7RczUVBaKXI1REREHQ/DTSsL6dUZfd3lKKvS4Ovv08Uuh4iIqMNhuGllEokEi0f3AABEn0tFRbVG5IqIiIg6FoabNvD4AA+4yW1wp6gCexOzxC6HiIioQ2G4aQNWFlIsGOENAPjydAo62I3XiYiIRMVw00ZmDesOe2sL3Mwpwqmf74pdDhERUYfBcNNGFLZWmDm0OwBg7albIldDRETUcTDctKEFI7xhIZXgTPJd/JjFG2oSERG1B4abNqR0tsMkfzcAwLozPHpDRETUHhhu2ljtpH7/S8yCqrBc5GqIiIjMH8NNGwtQOmJYD2dUawVEn0sVuxwiIiKzx3DTDmqP3mz5Pg3FFdUiV0NERGTeGG7awTg/F/TsYg91eTW2X8gQuxwiIiKzxnDTDqRSCRaOqrklw/qzKajWaEWuiIiIyHwx3LST6YM94Wxvjcz8Mhy+liN2OURERGaL4aad2FhZYG6QFwDgi9O3eEsGIiKiNsJw047Cgr1gbSnFlYwCXEzLF7scIiIis8Rw0466dJJh+uBuAHhLBiIiorbCcNPOFo6suSw85noOUu6WiFwNERGR+WG4aWc+Lp3wiJ8LBAH4krdkICIianUMNyKondRvZ0Im8ksqRa6GiIjIvDDciCCopzP8u8lRXqXF5vg0scshIiIyK0YTbt555x1IJBIsX77cYLsdO3bAz88PNjY26N+/Pw4cONA+BbYiiUSiO3rzVVwqyqs0IldERERkPowi3Fy4cAFr1qzBgAEDDLY7d+4cZs+ejYULF+Ly5csIDQ1FaGgorl692k6Vtp7J/d3hobDB3eJK7E28LXY5REREZkP0cFNcXIw5c+Zg7dq1cHJyMtj2448/xmOPPYa//e1v6NOnD9566y0MHjwYn332WTtV23qsLKRYMKLmlgxrT6dAq+WkfkRERK1B9HATERGBKVOmYPz48Y22jYuLq9Nu4sSJiIuLa3CbiooKqNVqvcVYzBymRCeZJZJzi3Hypztil0NERGQWRA03W7duxaVLlxAVFdWk9iqVCq6urnrrXF1doVKpGtwmKioKCoVCtyiVygequTXJbawwe1hNPWtP87JwIiKi1iBauMnIyMCyZcvw9ddfw8bGps3eJzIyEoWFhbolIyOjzd6rJeaP6AELqQTnfrmHq7cLxS6HiIjI5IkWbhISEpCbm4vBgwfD0tISlpaWOHnyJD755BNYWlpCo6l7BZGbmxtycvTvqJ2TkwM3N7cG30cmk0Eul+stxqSboy2m9HcHAKzj0RsiIqIHJlq4GTduHJKSkpCYmKhbhgwZgjlz5iAxMREWFhZ1tgkODsaxY8f01sXExCA4OLi9ym4TtZeF7/shG9mFZSJXQ0REZNosxXpjBwcH+Pv7662zt7dH586ddevDwsLQrVs33ZicZcuWYcyYMfjggw8wZcoUbN26FRcvXsQXX3zR7vW3pv6eCgT1dEb8rTxEn01F5OQ+YpdERERkskS/WsqQ9PR0ZGdn6x6HhIRgy5Yt+OKLLxAQEICdO3diz549dUKSKao9erPl+3QUlVeJXA0REZHpkgiC0KEmWFGr1VAoFCgsLDSq8TdarYAJH57EL3dK8H9T+mDR/bBDREREzfv+NuojNx2JVCrRBZoNZ1NRrdGKXBEREZFpYrgxIk8M6obO9ta4XVCGg1cbnruHiIiIGsZwY0RsrCwwL9gLQM2kfh3sjCEREVGrYLgxMvOCvCCzlOKHzEKcT8kTuxwiIiKTw3BjZDp3kmF6oCeAmhtqEhERUfMw3BihhSNr7hZ+9HoOfrlTLHI1REREpoXhxgj16toJ4/vU3CD0yzM8ekNERNQcDDdGavGomqM3uxIyca+4QuRqiIiITAfDjZEa1sMZAzwVqKjWYlN8mtjlEBERmQyGGyMlkUh0t2TYFJeG8qq6d0knIiKiuhhujNgkfzd0c7TFvZJK7L58W+xyiIiITALDjRGztJBiwQhvADWT+mm1nNSPiIioMQw3Rm7WsO5wsLHErTslOHEzV+xyiIiIjB7DjZHrJLPE08O6A6g5ekNERESGMdyYgPkjvGEplSD+Vh6SMgvFLoeIiMioMdyYAHeFLaYGeADg0RsiIqLGMNyYiEX3J/Xbn5SN2wVlIldDRERkvBhuTEQ/DwVCenWGRisg+ixvyUBERNQQhhsTsnh0zaR+35zPgLq8SuRqiIiIjBPDjQkZ+1BX9HbphOKKamw7nyF2OUREREaJ4caESCQS3dibDWdTUKXRilwRERGR8WG4MTHTBnZDl07WyCosx4GkbLHLISIiMjoMNybGxsoC4cHeAGouCxcE3pKBiIjotxhuTNDcIC/YWElx9bYa8bfyxC6HiIjIqDDcmCAne2s8FegJgJP6ERER/R7DjYlaOLInJBLg+I1cJOcWiV0OERGR0WC4MVE9uthjQh9XAMCXZzipHxERUS2GGxNWO6nfrku3caeoQuRqiIiIjAPDjQkb4uWEgUpHVFZrsSk+TexyiIiIjALDjQmTSCRYPKrm6M3m+DSUVWpEroiIiEh8DDcmbmI/V3g62SKvpBK7LmWKXQ4REZHoGG5MnKWFFAtH1tySYf2ZFGi1nNSPiIg6NlHDzapVqzBgwADI5XLI5XIEBwfj4MGDDbaPjo6GRCLRW2xsbNqxYuM0Y4gSchtL3LpbgmM3csUuh4iISFSihhtPT0+88847SEhIwMWLF/HII49g2rRpuHbtWoPbyOVyZGdn65a0NA6ktZdZ4unhXgCAtac4qR8REXVsooabqVOnYvLkyejduzceeughvP322+jUqRPi4+Mb3EYikcDNzU23uLq6tmPFxmt+iDesLCQ4n5qHKxkFYpdDREQkGqMZc6PRaLB161aUlJQgODi4wXbFxcXw8vKCUqls9CgPAFRUVECtVust5shNYYOpAR4AeEsGIiLq2EQPN0lJSejUqRNkMhmeffZZ7N69G3379q23ra+vL9avX4+9e/di8+bN0Gq1CAkJQWZmw1cJRUVFQaFQ6BalUtlWXRHdopE1l4UfvKpCRl6pyNUQERGJQyIIgqiX11RWViI9PR2FhYXYuXMn1q1bh5MnTzYYcH6rqqoKffr0wezZs/HWW2/V26aiogIVFb/O3qtWq6FUKlFYWAi5XN5q/TAW8778Hqd/votnRvTAa1Mb/xkSERGZArVaDYVC0aTvb9GP3FhbW8PHxweBgYGIiopCQEAAPv744yZta2VlhUGDBiE5ObnBNjKZTHc1Vu1izhbdn9Rv24V0FJZViVwNERFR+xM93PyeVqvVO9JiiEajQVJSEtzd3du4KtMxuncX+Lo6oKRSg63n08Uuh4iIqN2JGm4iIyNx6tQppKamIikpCZGRkYiNjcWcOXMAAGFhYYiMjNS1X7lyJY4cOYJbt27h0qVLmDt3LtLS0rBo0SKxumB0JBIJFo6qmdRvw9lUVFZrRa6IiIiofVmK+ea5ubkICwtDdnY2FAoFBgwYgMOHD2PChAkAgPT0dEilv+av/Px8LF68GCqVCk5OTggMDMS5c+eaND6nI5k20APvH74Jlboc+5Oy8MQgT7FLIiIiajeiDyhub80ZkGTKPj+RjPcP30Rfdzn2/2UkJBKJ2CURERG1mEkNKKa2MWd4d9haWeDHbDXO/XJP7HKIiIjaDcONmXK0s8aMITWnozipHxERdSQMN2bsmZE9IJEAsTfv4KecIrHLISIiahcMN2bMq7M9JvZ1AwCs49EbIiLqIBhuzNzi0TWT+u25nIXconKRqyEiImp7DDdmLtDLCYO7O6JSo8WmuDSxyyEiImpzDDcdwOL7t2TYFJ+G0spqkashIiJqWww3HcCj/dzQ3dkOBaVV2JXQ8B3UiYiIzAHDTQdgIZVg4ciaWzJ8eSYFGm2HmreRiIg6GIabDuKPQzyhsLVC6r1SHL2eI3Y5REREbYbhpoOws7bE3KDuAIC1p3hZOBERmS+Gmw4kPNgb1hZSXEzLx+X0fLHLISIiahMMNx2Ii9wGfxjoAQBYdzpF5GqIiIjaBsNNB1N7WfjBq9nIyCsVuRoiIqLWx3DTwfi6OWD0Q12hFWqunCIiIjI3DDcd0OJRNZeFb7+YgcLSKpGrISIial0MNx3QSJ8u8HNzQGmlBl+f5y0ZiIjIvDDcdEASiUQ39uarc6morNaKXBEREVHrYbjpoKYGeMBVLkOOugLfXckSuxwiIqJWw3DTQVlbShEe4g0AWHv6FgSBt2QgIiLzwHDTgc0Z5gU7awvcUBXhTPJdscshIiJqFQw3HZjCzgozhigBAGs5qR8REZkJhpsObuHIHpBKgFM/3cENlVrscoiIiB4Yw00Hp3S2wyR/dwC8JQMREZkHhhvCovuT+u1NvI1cdbnI1RARET0YhhvCoO5OGOLlhCqNgOhzqWKXQ0RE9EAYbggAsHh0zaR+X3+fjtLKapGrISIiajmGGwIAjO/jCu/Odigsq8KOi5lil0NERNRiDDcEALCQSrBwZM3Ymy/PpECj5aR+RERkmhhuSOepQCWc7KyQnleKI9dUYpdDRETUIgw3pGNrbYG5QV4Aam7JQEREZIpEDTerVq3CgAEDIJfLIZfLERwcjIMHDxrcZseOHfDz84ONjQ369++PAwcOtFO1HcO8YC9YW0hxKb0ACWl5YpdDRETUbKKGG09PT7zzzjtISEjAxYsX8cgjj2DatGm4du1ave3PnTuH2bNnY+HChbh8+TJCQ0MRGhqKq1evtnPl5svFwQahgzwAAGtPcVI/IiIyPRLByG4H7ezsjPfffx8LFy6s89zMmTNRUlKCffv26dYFBQVh4MCBWL16dZNeX61WQ6FQoLCwEHK5vNXqNic/5RTh0Q9PQSIBYl8aC6/O9mKXREREHVxzvr+NZsyNRqPB1q1bUVJSguDg4HrbxMXFYfz48XrrJk6ciLi4uAZft6KiAmq1Wm8hwx5ydcBY364QBGD9GR69ISIi0yJ6uElKSkKnTp0gk8nw7LPPYvfu3ejbt2+9bVUqFVxdXfXWubq6QqVq+MqeqKgoKBQK3aJUKlu1fnO1eFTNpH7bL2aioLRS5GqIiIiaTvRw4+vri8TERHz//ff485//jPDwcPz444+t9vqRkZEoLCzULRkZGa322uYspFdn9HWXo6xKg6+/Txe7HCIioiYTPdxYW1vDx8cHgYGBiIqKQkBAAD7++ON627q5uSEnJ0dvXU5ODtzc3Bp8fZlMprsaq3ahxkkkEiweXTOpX/S5VFRUa0SuiIiIqGlEDze/p9VqUVFRUe9zwcHBOHbsmN66mJiYBsfo0IN5fIAH3OQ2uFNUgb2JWWKXQ0RE1CSihpvIyEicOnUKqampSEpKQmRkJGJjYzFnzhwAQFhYGCIjI3Xtly1bhkOHDuGDDz7AjRs38MYbb+DixYtYunSpWF0wa1YWUiwY4Q0A+PJ0CozswjoiIqJ6iRpucnNzERYWBl9fX4wbNw4XLlzA4cOHMWHCBABAeno6srOzde1DQkKwZcsWfPHFFwgICMDOnTuxZ88e+Pv7i9UFszdrWHfYW1vgZk4RTv18V+xyiIiIGmV089y0Nc5z03wrv/sR68+mYKRPF2xeNFzscoiIqAMyyXluyHgtGOENC6kEZ5Lv4scszhNERETGjeGGGqV0tsMk/5or0tad4Q01iYjIuDHcUJPUTur3v8QsqArLRa6GiIioYQw31CQBSkcM6+GMaq2A6HOpYpdDRETUIIYbarLaozdbvk9DSUW1yNUQERHVj+GGmmycnwt6drGHurwa2y/yNhZERGScGG6oyaRSCRaOqrklw5dnUlCt0YpcERERUV0MN9Qs0wd7wtneGpn5ZTh8LafxDYiIiNoZww01i42VBeYGeQEAvjh9i7dkICIio8NwQ80WFuwFa0sprmQU4GJavtjlEBER6WG4oWbr0kmG6YO7AQDWnuKkfkREZFwYbqhFFo6suSw85noOUu6WiFwNERHRrxhuqEV8XDrhET8XCALwJW/JQERERoThhlqsdlK/nQmZyC+pFLkaIiKiGgw31GJBPZ3h302O8iotNseniV0OERERAIYbegASiUR39OaruFSUV2lEroiIiIjhhh7Q5P7u8FDY4G5xJfYm3ha7HCIiIoYbejBWFlIsGFFzS4a1p1Og1XJSPyIiEhfDDT2wmcOU6CSzRHJuMU7+dEfscoiIqINjuKEHJrexwuxhSgDA2tO8LJyIiMTFcEOtYv6IHrCQSnDul3u4llUodjlERNSBMdxQq+jmaIsp/d0BAOtOp4hcDRERdWQMN9Rqai8L/+5KFrILy0SuhoiIOiqGG2o1/T0VCOrpjGqtgOizqWKXQ0REHRTDDbWq2qM3W75PR1F5lcjVEBFRR8RwQ63qYV8X9Opqj6KKamy7kCF2OURE1AEx3FCrkkolWHT/6M2Gs6mo1mhFroiIiDoahhtqdU8M6obO9ta4XVCGg1dVYpdDREQdDMMNtTobKwvMC/YCUDOpnyDwlgxERNR+GG6oTcwL8oLMUoofMgtxPiVP7HKIiKgDYbihNtG5kwzTAz0B1NxQk4iIqL2IGm6ioqIwdOhQODg4wMXFBaGhobh586bBbaKjoyGRSPQWGxubdqqYmmPhyJq7hR+9noNf7hSLXA0REXUUooabkydPIiIiAvHx8YiJiUFVVRUeffRRlJSUGNxOLpcjOztbt6SlpbVTxdQcvbp2wvg+rgCAL8/w6A0REbUPSzHf/NChQ3qPo6Oj4eLigoSEBIwePbrB7SQSCdzc3Nq6PGoFi0f1wNHrOdiVkIkXJzyEzp1kYpdERERmrkVHbjIyMpCZmal7fP78eSxfvhxffPHFAxVTWFhzN2lnZ2eD7YqLi+Hl5QWlUolp06bh2rVrD/S+1HaG9XDGAE8FKqq12BTPI2xERNT2WhRunn76aZw4cQIAoFKpMGHCBJw/fx6vvPIKVq5c2aJCtFotli9fjhEjRsDf37/Bdr6+vli/fj327t2LzZs3Q6vVIiQkRC9s/VZFRQXUarXeQu1HIpHobsmwKS4N5VUakSsiIiJz16Jwc/XqVQwbNgwAsH37dvj7++PcuXP4+uuvER0d3aJCIiIicPXqVWzdutVgu+DgYISFhWHgwIEYM2YMvv32W3Tt2hVr1qypt31UVBQUCoVuUSqVLaqPWm6Svxu6OdriXkkldl++LXY5RERk5loUbqqqqiCT1YydOHr0KP7whz8AAPz8/JCdnd3s11u6dCn27duHEydOwNPTs1nbWllZYdCgQUhOTq73+cjISBQWFuqWjAze76i9WVpIsWCEN4CaSf20Wk7qR0REbadF4aZfv35YvXo1Tp8+jZiYGDz22GMAgKysLHTu3LnJryMIApYuXYrdu3fj+PHj6NGjR7Nr0Wg0SEpKgru7e73Py2QyyOVyvYXa36xh3eFgY4lbd0pw4mau2OUQEZEZa1G4effdd7FmzRqMHTsWs2fPRkBAAADgf//7n+50VVNERERg8+bN2LJlCxwcHKBSqaBSqVBWVqZrExYWhsjISN3jlStX4siRI7h16xYuXbqEuXPnIi0tDYsWLWpJV6iddJJZ4ulh3QHUHL0hIiJqKy26FHzs2LG4e/cu1Go1nJycdOuXLFkCOzu7Jr/OqlWrdK/3Wxs2bMD8+fMBAOnp6ZBKf81g+fn5WLx4MVQqFZycnBAYGIhz586hb9++LekKtaP5I7zx5ZkUxN/KQ1JmIfp7KsQuiYiIzJBEaMFdDcvKyiAIgi7IpKWlYffu3ejTpw8mTpzY6kW2JrVaDYVCgcLCQp6iEsFftyVi9+Xb+EOABz6ZPUjscoiIyEQ05/u7Raelpk2bho0bNwIACgoKMHz4cHzwwQcIDQ3VHY0hqs+iUTXjqvYnZeN2QVkjrYmIiJqvReHm0qVLGDVqFABg586dcHV1RVpaGjZu3IhPPvmkVQsk89LPQ4GQXp2h0QqIPstbMhARUetrUbgpLS2Fg4MDAODIkSN48sknIZVKERQUxPs8UaMWj66Z1O+b8xlQl1eJXA0REZmbFoUbHx8f7NmzBxkZGTh8+DAeffRRAEBubi7HsVCjxj7UFb1dOqG4ohrbznPeISIial0tCjevvfYaXnrpJXh7e2PYsGEIDg4GUHMUZ9AgDhIlwyQSiW7szYazKajSaEWuiIiIzEmLws1TTz2F9PR0XLx4EYcPH9atHzduHD788MNWK47M17SB3dClkzWyCstxIKn5s1oTERE1pEXhBgDc3NwwaNAgZGVl6W5aOWzYMPj5+bVacWS+bKwsEB7sDaBmUr8WzEhARERUrxaFG61Wi5UrV0KhUMDLywteXl5wdHTEW2+9Ba2WpxioaeYGecHGSoqrt9WIv5UndjlERGQmWhRuXnnlFXz22Wd45513cPnyZVy+fBn/+te/8Omnn+LVV19t7RrJTDnZW+OpwJobpfKWDERE1FpaNEOxh4cHVq9erbsbeK29e/fiueeew+3bt1utwNbGGYqNS8rdEjzyQSwEATj6wmj4uDiIXRIRERmhNp+hOC8vr96xNX5+fsjL4+kFaroeXewxoY8rAODLM5zUj4iIHlyLwk1AQAA+++yzOus/++wzDBgw4IGLoo6ldlK/XZdu405RhcjVEBGRqWvRXcHfe+89TJkyBUePHtXNcRMXF4eMjAwcOHCgVQsk8zfEywkDlY5IzCjApvg0vDDhIbFLIiIiE9aiIzdjxozBTz/9hCeeeAIFBQUoKCjAk08+iWvXrmHTpk2tXSOZOYlEgsWjao7ebI5PQ3mVRuSKiIjIlLVoQHFDrly5gsGDB0OjMd4vJw4oNk7VGi3G/jsWmfllePsJf8wZ7iV2SUREZETafEAxUWuztJBi4ciaWzJ8eToFWi0n9SMiopZhuCGjMWOIEnIbS9y6W4JjN3LFLoeIiEwUww0ZDXuZJZ6+fzqKk/oREVFLNetqqSeffNLg8wUFBQ9SCxHmh3jjyzO3cD4lD1cyChCgdBS7JCIiMjHNOnKjUCgMLl5eXggLC2urWqkDcFPYYGqABwAevSEiopZp1pGbDRs2tFUdRDqLRvbEt5du4+BVFTLySqF0thO7JCIiMiEcc0NGp6+HHKN6d4FGK2DD2VSxyyEiIhPDcENGadH9Sf22XUhHYVmVyNUQEZEpYbghozS6dxf4ujqgpFKDrefTxS6HiIhMCMMNGSWJRIKFo2om9dtwNhWV1VqRKyIiIlPBcENGa9pAD3R1kEGlLsf+pCyxyyEiIhPBcENGS2Zpgfkh3gCAtadS0Iq3QSMiIjPGcENGbc7w7rC1ssCP2Wqc++We2OUQEZEJYLgho+ZoZ40ZQzwBcFI/IiJqGoYbMnrPjOwBiQSIvXkHP+UUiV0OEREZOYYbMnpene0xsa8bAGAdj94QEVEjGG7IJCweXTOp357LWcgtKhe5GiIiMmaihpuoqCgMHToUDg4OcHFxQWhoKG7evNnodjt27ICfnx9sbGzQv39/HDhwoB2qJTEFejlhcHdHVGq02BSXJnY5RERkxEQNNydPnkRERATi4+MRExODqqoqPProoygpKWlwm3PnzmH27NlYuHAhLl++jNDQUISGhuLq1avtWDmJYfH9WzJsik9DWaVG5GqIiMhYSQQjmjzkzp07cHFxwcmTJzF69Oh628ycORMlJSXYt2+fbl1QUBAGDhyI1atXN/oearUaCoUChYWFkMvlrVY7tT2NVsDD/45Fel4p3prWD/OCvcUuiYiI2klzvr+NasxNYWEhAMDZ2bnBNnFxcRg/frzeuokTJyIuLq7e9hUVFVCr1XoLmSYLqQQLR9bckuHLMynQaI0mlxMRkRExmnCj1WqxfPlyjBgxAv7+/g22U6lUcHV11Vvn6uoKlUpVb/uoqCgoFArdolQqW7Vual9/HOIJha0VUu+V4uj1HLHLISIiI2Q04SYiIgJXr17F1q1bW/V1IyMjUVhYqFsyMjJa9fWpfdlZW2JuUHcAwNpTvCyciIjqMopws3TpUuzbtw8nTpyAp6enwbZubm7IydH/iz0nJwdubm71tpfJZJDL5XoLmbbwYG9YW0hxMS0fl9PzxS6HiIiMjKjhRhAELF26FLt378bx48fRo0ePRrcJDg7GsWPH9NbFxMQgODi4rcokI+Mit8EfBnoAANadThG5GiIiMjaihpuIiAhs3rwZW7ZsgYODA1QqFVQqFcrKynRtwsLCEBkZqXu8bNkyHDp0CB988AFu3LiBN954AxcvXsTSpUvF6AKJZNGomiB88Go2MvJKRa6GiIiMiajhZtWqVSgsLMTYsWPh7u6uW7Zt26Zrk56ejuzsbN3jkJAQbNmyBV988QUCAgKwc+dO7Nmzx+AgZDI/fm5yjH6oK7RCzZVTREREtYxqnpv2wHluzMfpn+9g3pfnYWdtgbgV46CwsxK7JCIiaiMmO88NUXOM9OkCPzcHlFZq8PV53pKBiIhqMNyQyZJIJLpbMnx1LhWV1VqRKyIiImPAcEMmbWqAB1zlMuSoK/DdlSyxyyEiIiPAcEMmzdpSivAQbwDA2tO30MGGkBERUT0YbsjkzRnmBTtrC9xQFeFM8l2xyyEiIpEx3JDJU9hZYcaQmnuGreWkfkREHR7DDZmFhSN7QCoBTv10BzdVRWKXQ0REImK4IbOgdLbDJH93ADVjb4iIqONiuCGzUXtLhr2Jt5GrLhe5GiIiEgvDDZmNQd2dMMTLCVUaAV/FpYpdDhERiYThhszK4tE1k/ptjk9HaWW1yNUQEZEYGG7IrIzv4wrvznYoLKvCjouZYpdDREQiYLghs2IhlWDhyJqxN1+eSYFGy0n9iIg6GoYbMjtPBSrhZGeF9LxSHLmmErscIiJqZww3ZHZsrS0wN8gLAC8LJyLqiBhuyCzNC/aCtYUUl9ILkJCWJ3Y5RETUjhhuyCy5ONggdJAHAGDtKd6SgYioI2G4IbO1aFTNZeGHf1Qh7V6JyNUQEVF7Ybghs/WQqwPG+naFIADrz/DoDRFRR8FwQ2Zt8f2jN9svZqKgtFLkaoiIqD0w3JBZC+nVGX3d5Sir0uDr79PFLoeIiNoBww2ZNYlEgsWjayb1iz6XiopqjcgVERFRW2O4IbP3+AAPuMltcKeoAnsTs8Quh4iI2hjDDZk9KwspFozwBgB8eToFgsBbMhARmTOGG+oQZg3rDntrC9zMKcKpn++KXQ4REbUhhhvqEBS2Vpg5tDsAYB1vyUBEZNYYbqjDWDDCGxZSCU7/fBc/ZqnFLoeIiNoIww11GEpnO0zydwMArDvDozdEROaK4YY6lNpJ/b67kgVVYbnI1RARUVtguKEOJUDpiGE9nFGlERB9LlXscoiIqA0w3FCHU3v0Zsv3aSipqBa5GiIiam2ihptTp05h6tSp8PDwgEQiwZ49ewy2j42NhUQiqbOoVKr2KZjMwjg/F/TsYg91eTW2X8wQuxwiImplooabkpISBAQE4PPPP2/Wdjdv3kR2drZucXFxaaMKyRxJpRIsHFVzS4Yvz6SgWqMVuSIiImpNlmK++aRJkzBp0qRmb+fi4gJHR8fWL4g6jOmDPfHBkZ+QmV+Gw9dyMGWAu9glERFRKzHJMTcDBw6Eu7s7JkyYgLNnz4pdDpkgGysLzA3yAgB8cfoWb8lARGRGTCrcuLu7Y/Xq1di1axd27doFpVKJsWPH4tKlSw1uU1FRAbVarbcQAUBYsBesLaW4klGAi2n5YpdDREStxKTCja+vL/70pz8hMDAQISEhWL9+PUJCQvDhhx82uE1UVBQUCoVuUSqV7VgxGbMunWSYPrgbAGDtKU7qR0RkLkwq3NRn2LBhSE5ObvD5yMhIFBYW6paMDF4dQ79aOLLmsvCY6zlIuVsicjVERNQaTD7cJCYmwt294cGgMpkMcrlcbyGq5ePSCY/4uUAQgC95SwYiIrMg6tVSxcXFekddUlJSkJiYCGdnZ3Tv3h2RkZG4ffs2Nm7cCAD46KOP0KNHD/Tr1w/l5eVYt24djh8/jiNHjojVBTIDi0f1xPEbudiZkIkXJ/jCyd5a7JKIiOgBiHrk5uLFixg0aBAGDRoEAHjhhRcwaNAgvPbaawCA7OxspKen69pXVlbixRdfRP/+/TFmzBhcuXIFR48exbhx40Spn8xDUE9n+HeTo7xKi83xaWKXQ0RED0gidLBrYNVqNRQKBQoLC3mKinT2Jt7Gsq2J6NJJhjN/fxg2VhZil0RERL/RnO9vkx9zQ9QaJvd3h4fCBneLK/DK7qtI5eBiIiKTxXBDBMDKQoqIR3wAALsuZWLsv2Mxf8N5HL+RA622Qx3cJCIyeaIOKCYyJnOGe8HD0RYbz6Ui9qc7iL1Zs3R3tsO8IC/8cYgnHO042JiIyNhxzA1RPVLvlmBzfBq2X8yAurwaACCzlCJ0YDfMC/aCfzeFyBUSEXUszfn+ZrghMqCsUoO9ibexMS4NP2b/euuOQC8nhAV7YZK/O6wteXaXiKitMdwYwHBDLSEIAhLS8rExLg0HkrJRfX8cTpdOMswepsTTw7vDXWErcpVEROaL4cYAhht6ULlF5dh6PgNff5+GHHUFAMBCKsHEfq6YF+SNoJ7OkEgkIldJRGReGG4MYLih1lKl0eLItRxsjEvF9yl5uvUPuXbCvGBvPDmoG+xlHLNPRNQaGG4MYLihtnBDpcamuDTsvnwbpZUaAICDzBLTAz0xL9gLvbp2ErlCIiLTxnBjAMMNtSV1eRV2JWRiU1wabv1mIsBRvbtgXpAXxvVxhYWUp6yIiJqL4cYAhhtqD1qtgLO/3MVX59Jw7EYOaj9l3RxtMSeoO2YN7Q5n3qCTiKjJGG4MYLih9paRV4qvv0/HtgvpyC+tAgBYW0oxdYAHwoK9EKB0FLdAIiITwHBjAMMNiaW8SoPvrmRhY1wakm4X6tYHKB0RFuSFKQPcecNOIqIGMNwYwHBDYhMEAYkZBdgUl4Z9P2SjUqMFADjbW2PmUCXmDO8OTyc7kaskIjIuDDcGMNyQMblbXIFtFzLwdXwasgrLAQBSCTCujyvCg70xwqcz58whIgLDjUEMN2SMqjVaHLuRi41xqTibfE+3vldXe8wL8sL0QE842FiJWCERkbgYbgxguCFjl5xbhE1xadiZkImS+3Pm2Ftb4InB3RAW7I2HXB1ErpCIqP0x3BjAcEOmoriiGrsvZeKruDQk5xbr1gf37IywYC9M6OsKSwvetJOIOgaGGwMYbsjUCIKAuFv3sPFcGmKu50Bz/6ad7gobPD2sO2YN646uDjKRqyQialsMNwYw3JApyyoow5bv0/HN+XTcK6kEAFhZSDC5vzvCgr0xuLsjByATkVliuDGA4YbMQUW1BgeTVPgqLhWX0wt06/27yREW5I0/DPTgnDlEZFYYbgxguCFzk5RZiI1xqfjflSxUVNfMmeNoZ4UZQ5SYO9wL3TtzzhwiMn0MNwYw3JC5yi+pxPaLGdgUn4bM/DIAgEQCPOzrgrBgL4zu3RVS3rSTiEwUw40BDDdk7jRaAbE3c/FVXBpO/XRHt967sx3mBnnhj4FKKOw4Zw4RmRaGGwMYbqgjuXWnGJvj07EjIQNF5dUAAFsrC4QO8sC8IG/09eBngIhMA8ONAQw31BGVVlZjz+UsbIxLxQ1VkW79MG9nzAv2wmP+brDinDlEZMQYbgxguKGOTBAEXEjNx1dxqTh8VYXq+3PmuDjIMHtYdzw9vDtc5TYiV0lEVBfDjQEMN0Q1ctTl2PJ9OracT8edogoAgKVUgon+bggP9sZQbyfOmUNERoPhxgCGGyJ9ldVaHL6mwsa4VFxIzdet93NzQFiwN0IHecDO2lLEComIGG4MYrghatiPWWpsik/FnstZKKuquWmng40l/hioxLxgL/ToYi9yhUTUUTHcGMBwQ9S4wtIq7EjIwOb4NKTeK9WtH/1QV4QHe2GsrwssOGcOEbUjhhsDGG6Imk6rFXDq5zvYGJeGEzdzUfuvhdLZFnOHe2HGECWc7K3FLZKIOoTmfH+Leu3nqVOnMHXqVHh4eEAikWDPnj2NbhMbG4vBgwdDJpPBx8cH0dHRbV4nUUcllUow1tcF6+cPxcmXHsaS0T2hsLVCRl4Zog7eQFDUMfxtxxVcvV0odqlERDqihpuSkhIEBATg888/b1L7lJQUTJkyBQ8//DASExOxfPlyLFq0CIcPH27jSomoe2c7/GNyH8RHjsN70wegn4ccFdVa7EjIxOOfnsET/z2LPZdvo6JaI3apRNTBGc1pKYlEgt27dyM0NLTBNn//+9+xf/9+XL16Vbdu1qxZKCgowKFDh5r0PjwtRdQ6BEHApfQCbIpLxf6kbFRpav4p6dLJGrOG1syZ4+FoK3KVRGQuTOa0VHPFxcVh/PjxeusmTpyIuLi4BrepqKiAWq3WW4jowUkkEgR6OeGjWYNwbsU4vDjhIbjJbXC3uBKfnUjGqPdO4NlNCTj3y10Yyd9QRNRBmFS4UalUcHV11Vvn6uoKtVqNsrKyereJioqCQqHQLUqlsj1KJepQujrI8Py43jjz94exas5gBPV0hkYr4NA1FZ5e+z0e/fAUNsWloriiWuxSiagDMKlw0xKRkZEoLCzULRkZGWKXRGS2LC2kmNTfHVuXBOPIX0djblB32Flb4OfcYry69xqC/nUMr++9iuTcYrFLJSIzZlLTjrq5uSEnJ0dvXU5ODuRyOWxt6z+3L5PJIJPJ2qM8IvqNh1wd8M/Q/nj5MT98m5CJjfFpuHWnBF/FpeGruDSM8OmMsGBvjPNzgSVv2klErcikwk1wcDAOHDigty4mJgbBwcEiVUREjZHbWGH+iB4ID/HG2eR72BiXiqPXc3A2+R7OJt+Dh8IGc4K8MGuoEp078Q8RInpwol4tVVxcjOTkZADAoEGD8J///AcPP/wwnJ2d0b17d0RGRuL27dvYuHEjgJpLwf39/REREYFnnnkGx48fx1/+8hfs378fEydObNJ78mopIvFl5pfi6+/Tse1CBvJKKgEA1hZSPD7AHWEh3hiodBS3QCIyOiYzQ3FsbCwefvjhOuvDw8MRHR2N+fPnIzU1FbGxsXrb/PWvf8WPP/4IT09PvPrqq5g/f36T35Phhsh4lFdpsP+HbGyMT8OVjALd+gGeCoQFe+PxAe6wsbIQr0AiMhomE27EwHBDZJwSMwqwMS4V+37IRmW1FgDgZGeFmUO7Y87w7lA624lcIRGJieHGAIYbIuOWV1KJbRdqbtp5u6BmigepBHjEzxXhIV4Y0asLpLxpJ1GHw3BjAMMNkWnQaAUcu56DTfFpOP3zXd36nl3sMS/YC9MDPSG3sRKxQiJqTww3BjDcEJme5NxibI5Pw86ETN1EgHbWFnhiUDeEBXvD181B5AqJqK0x3BjAcENkuoorqrH78m1sikvFTzm/TgQ4vIczwkO8MaGvK6w4Zw6RWWK4MYDhhsj0CYKA+Ft52BSfisPXcqDR1vwz5iqXYeYQJYJ6doa/p4KnrYjMCMONAQw3ROYlu7AM33yfji3n03G3uFLvuV5d7RGgdMRApSMCPB3h5+4AmSUvLScyRQw3BjDcEJmnimoNDl1V4ciPOfghswAZeXVvpmttIUUfDzkGeioQoHREgNIRPTrb8+orIhPAcGMAww1Rx3CvuAI/ZBYiMaMAVzILcCWjAPmlVXXaOcgsMUCpQICno+4oj6vcRoSKicgQhhsDGG6IOiZBEJCZX1YTdu4HnqTbhSiv0tZp6ya3QYCy5ujOQE9Hjt8hMgIMNwYw3BBRrWqNFj/lFOuO7CRmFOCnnCJo6/lXkeN3iMTFcGMAww0RGVJaWY1rWWpd2LnC8TtERoHhxgCGGyJqLo7fIRIfw40BDDdE9KAEQUBGXhkSMwvwA8fvELULhhsDGG6IqC1w/A5R22K4MYDhhojaS2llNa7eVuuuzuL4HaKWY7gxgOGGiMTE8TtELcNwYwDDDREZE47fIWoahhsDGG6IyNhx/A5RXQw3BjDcEJEp4vgd6ugYbgxguCEic8HxO9SRMNwYwHBDROaK43fInDHcGMBwQ0QdCcfvkLlguDGA4YaIOjqO3yFTxHBjAMMNEVFdHL9Dxo7hxgCGGyKixnH8DhkbhhsDGG6IiFqG43dITAw3BjDcEBG1nt+O30nMLMAPHL9DbYThxgCGGyKitsXxO9QWGG4MYLghImpfvx2/cyWjZrma1fD4nT7uDlA620HpZFfzX2dbKJ3tOI6ng2O4MYDhhohIfM0Zv1NLYWuF7rVhx8kOns52NY+dbNHNyZZjeswcw40BDDdERMapdvzOL3eKkZFXivS8UmTklyEzrxT3SioNbiuRAK4ONujubAfP++FHWRt+nG3h6mDD8T0mzuTCzeeff473338fKpUKAQEB+PTTTzFs2LB620ZHR2PBggV662QyGcrLy5v0Xgw3RESmp6SiGhn5pcjIK0NGXun9/7//OL8UpZUag9tbW0jRzcn2/ukuW91pr9rwo7C1gkTC8GPMmvP9bdlONTVo27ZteOGFF7B69WoMHz4cH330ESZOnIibN2/CxcWl3m3kcjlu3rype8xfSCIi82Yvs4Sfmxx+bnW/1ARBQF5Jpe5IT0ZeKTLz7x/5yStDVkEZKjVapNwtQcrdknpf30Fmef80l63eWJ/uznbwdLKDjRVPeZkS0Y/cDB8+HEOHDsVnn30GANBqtVAqlXj++eexYsWKOu2jo6OxfPlyFBQUtOj9eOSGiKhjqdZooVKXIz2vFJn3j/T89rTXnaKKRl+jq4MMSifb+0d67O6P+akJQu4KG1haSNuhJx2byRy5qaysREJCAiIjI3XrpFIpxo8fj7i4uAa3Ky4uhpeXF7RaLQYPHox//etf6NevX71tKyoqUFHx6y+uWq1uvQ4QEZHRs7SQwtOp5ggMetV9vrxKo3ekp/a0V3pezXifoopq3CmqwJ2iClxKL6j7+lIJPBxtdQOdlboAVHP6q7O9Nc8wtDNRw83du3eh0Wjg6uqqt97V1RU3btyodxtfX1+sX78eAwYMQGFhIf79738jJCQE165dg6enZ532UVFRePPNN9ukfiIiMn02VhbwcXGAj4tDnecEQUBhWRUy8sruH+m5P9bn/umv2/k1p7zS7x8JAu7VeQ07a4v7occWnrpxPr9e9WUvE32EiNkR9bRUVlYWunXrhnPnziE4OFi3/uWXX8bJkyfx/fffN/oaVVVV6NOnD2bPno233nqrzvP1HblRKpU8LUVERA9MqxWQU1T+a/i5H4Ay7z/OKSpHY9+yne2t4fmbIz3ddXP82MLD0RZWPOUFwIROS3Xp0gUWFhbIycnRW5+TkwM3N7cmvYaVlRUGDRqE5OTkep+XyWSQyWQPXCsREdHvSaUSuCts4a6wxbAeznWer6jW4HZ+GTLyy+6P+SnVXfWVnleKwrIq3CupxL2SSlzJKKj7+hLAXaF/yuu3c/10dZDxlFc9RA031tbWCAwMxLFjxxAaGgqgZkDxsWPHsHTp0ia9hkajQVJSEiZPntyGlRIRETWfzNICPbt2Qs+unep9Xl1e9esl7fWc9qqo1uJ2QRluF5QhHnn1vL60zuXtnNXZCC4Ff+GFFxAeHo4hQ4Zg2LBh+Oijj1BSUqKbyyYsLAzdunVDVFQUAGDlypUICgqCj48PCgoK8P777yMtLQ2LFi0SsxtERETNJrexQj8PBfp5KOo8p9UKuFtcoXekJ+M3R36yC8tQUa1Fcm4xknOL6339jjqrs+jhZubMmbhz5w5ee+01qFQqDBw4EIcOHdINMk5PT4dU+uv5xvz8fCxevBgqlQpOTk4IDAzEuXPn0LdvX7G6QERE1OqkUglc5DZwkdsg0Kvu81UaLbIKynQTGf4afn6d1bmwrApJtwuRdLuwzvbmPKuz6PPctDfOc0NERB3B72d1Tr8/uaGpzupsMgOKiYiIqG00NqvzvZJKvfE9vz3lZeqzOvPIDREREemp1miRXViud1n7bwc7Nzars49LJxx9YUyr1sQjN0RERNRilhZS3UzL9c3qXFZZM6tz3cHONeN9lE627V/0bzDcEBERUbPYWlugt6sDervWP6tzRbVWhKp+xWkPiYiIqNVIJBLR76LOcENERERmheGGiIiIzArDDREREZkVhhsiIiIyKww3REREZFYYboiIiMisMNwQERGRWWG4ISIiIrPCcENERERmheGGiIiIzArDDREREZkVhhsiIiIyKww3REREZFYsxS6gvQmCAABQq9UiV0JERERNVfu9Xfs9bkiHCzdFRUUAAKVSKXIlRERE1FxFRUVQKBQG20iEpkQgM6LVapGVlQUHBwdIJJJWfW21Wg2lUomMjAzI5fJWfW1jYO79A8y/j+yf6TP3PrJ/pq+t+igIAoqKiuDh4QGp1PComg535EYqlcLT07NN30Mul5vtLy1g/v0DzL+P7J/pM/c+sn+mry362NgRm1ocUExERERmheGGiIiIzArDTSuSyWR4/fXXIZPJxC6lTZh7/wDz7yP7Z/rMvY/sn+kzhj52uAHFREREZN545IaIiIjMCsMNERERmRWGGyIiIjIrDDdERERkVhhumunzzz+Ht7c3bGxsMHz4cJw/f95g+x07dsDPzw82Njbo378/Dhw40E6Vtkxz+hcdHQ2JRKK32NjYtGO1zXPq1ClMnToVHh4ekEgk2LNnT6PbxMbGYvDgwZDJZPDx8UF0dHSb19lSze1fbGxsnf0nkUigUqnap+BmioqKwtChQ+Hg4AAXFxeEhobi5s2bjW5nSp/BlvTRlD6Hq1atwoABA3STuwUHB+PgwYMGtzGl/dfc/pnSvqvPO++8A4lEguXLlxtsJ8Y+ZLhphm3btuGFF17A66+/jkuXLiEgIAATJ05Ebm5uve3PnTuH2bNnY+HChbh8+TJCQ0MRGhqKq1evtnPlTdPc/gE1M1BmZ2frlrS0tHasuHlKSkoQEBCAzz//vEntU1JSMGXKFDz88MNITEzE8uXLsWjRIhw+fLiNK22Z5vav1s2bN/X2oYuLSxtV+GBOnjyJiIgIxMfHIyYmBlVVVXj00UdRUlLS4Dam9hlsSR8B0/kcenp64p133kFCQgIuXryIRx55BNOmTcO1a9fqbW9q+6+5/QNMZ9/93oULF7BmzRoMGDDAYDvR9qFATTZs2DAhIiJC91ij0QgeHh5CVFRUve1nzJghTJkyRW/d8OHDhT/96U9tWmdLNbd/GzZsEBQKRTtV17oACLt37zbY5uWXXxb69eunt27mzJnCxIkT27Cy1tGU/p04cUIAIOTn57dLTa0tNzdXACCcPHmywTam9hn8vab00ZQ/h4IgCE5OTsK6devqfc7U958gGO6fqe67oqIioXfv3kJMTIwwZswYYdmyZQ22FWsf8shNE1VWViIhIQHjx4/XrZNKpRg/fjzi4uLq3SYuLk6vPQBMnDixwfZiakn/AKC4uBheXl5QKpWN/oViakxp/z2IgQMHwt3dHRMmTMDZs2fFLqfJCgsLAQDOzs4NtjH1fdiUPgKm+TnUaDTYunUrSkpKEBwcXG8bU95/TekfYJr7LiIiAlOmTKmzb+oj1j5kuGmiu3fvQqPRwNXVVW+9q6trg2MUVCpVs9qLqSX98/X1xfr167F3715s3rwZWq0WISEhyMzMbI+S21xD+0+tVqOsrEykqlqPu7s7Vq9ejV27dmHXrl1QKpUYO3YsLl26JHZpjdJqtVi+fDlGjBgBf3//BtuZ0mfw95raR1P7HCYlJaFTp06QyWR49tlnsXv3bvTt27fetqa4/5rTP1PbdwCwdetWXLp0CVFRUU1qL9Y+7HB3BafWExwcrPcXSUhICPr06YM1a9bgrbfeErEyagpfX1/4+vrqHoeEhOCXX37Bhx9+iE2bNolYWeMiIiJw9epVnDlzRuxS2kxT+2hqn0NfX18kJiaisLAQO3fuRHh4OE6ePNlgADA1zemfqe27jIwMLFu2DDExMUY/8Jnhpom6dOkCCwsL5OTk6K3PycmBm5tbvdu4ubk1q72YWtK/37OyssKgQYOQnJzcFiW2u4b2n1wuh62trUhVta1hw4YZfWBYunQp9u3bh1OnTsHT09NgW1P6DP5Wc/r4e8b+ObS2toaPjw8AIDAwEBcuXMDHH3+MNWvW1GlrivuvOf37PWPfdwkJCcjNzcXgwYN16zQaDU6dOoXPPvsMFRUVsLCw0NtGrH3I01JNZG1tjcDAQBw7dky3TqvV4tixYw2eTw0ODtZrDwAxMTEGz7+KpSX9+z2NRoOkpCS4u7u3VZntypT2X2tJTEw02v0nCAKWLl2K3bt34/jx4+jRo0ej25jaPmxJH3/P1D6HWq0WFRUV9T5navuvPob693vGvu/GjRuHpKQkJCYm6pYhQ4Zgzpw5SExMrBNsABH3YZsOVzYzW7duFWQymRAdHS38+OOPwpIlSwRHR0dBpVIJgiAI8+bNE1asWKFrf/bsWcHS0lL497//LVy/fl14/fXXBSsrKyEpKUmsLhjU3P69+eabwuHDh4VffvlFSEhIEGbNmiXY2NgI165dE6sLBhUVFQmXL18WLl++LAAQ/vOf/wiXL18W0tLSBEEQhBUrVgjz5s3Ttb9165ZgZ2cn/O1vfxOuX78ufP7554KFhYVw6NAhsbpgUHP79+GHHwp79uwRfv75ZyEpKUlYtmyZIJVKhaNHj4rVBYP+/Oc/CwqFQoiNjRWys7N1S2lpqa6NqX8GW9JHU/ocrlixQjh58qSQkpIi/PDDD8KKFSsEiUQiHDlyRBAE099/ze2fKe27hvz+ailj2YcMN8306aefCt27dxesra2FYcOGCfHx8brnxowZI4SHh+u13759u/DQQw8J1tbWQr9+/YT9+/e3c8XN05z+LV++XNfW1dVVmDx5snDp0iURqm6a2kuff7/U9ik8PFwYM2ZMnW0GDhwoWFtbCz179hQ2bNjQ7nU3VXP79+677wq9evUSbGxsBGdnZ2Hs2LHC8ePHxSm+CerrGwC9fWLqn8GW9NGUPofPPPOM4OXlJVhbWwtdu3YVxo0bp/viFwTT33/N7Z8p7buG/D7cGMs+lAiCILTtsSEiIiKi9sMxN0RERGRWGG6IiIjIrDDcEBERkVlhuCEiIiKzwnBDREREZoXhhoiIiMwKww0RERGZFYYbIurwJBIJ9uzZI3YZRNRKGG6ISFTz58+HRCKpszz22GNil0ZEJop3BSci0T322GPYsGGD3jqZTCZSNURk6njkhohEJ5PJ4Obmprc4OTkBqDlltGrVKkyaNAm2trbo2bMndu7cqbd9UlISHnnkEdja2qJz585YsmQJiouL9dqsX78e/fr1g0wmg7u7O5YuXar3/N27d/HEE0/Azs4OvXv3xv/+97+27TQRtRmGGyIyeq+++iqmT5+OK1euYM6cOZg1axauX78OACgpKcHEiRPh5OSECxcuYMeOHTh69KheeFm1ahUiIiKwZMkSJCUl4X//+x98fHz03uPNN9/EjBkz8MMPP2Dy5MmYM2cO8vLy2rWfRNRK2vzWnEREBoSHhwsWFhaCvb293vL2228LglBzp+xnn31Wb5vhw4cLf/7znwVBEIQvvvhCcHJyEoqLi3XP79+/X5BKpYJKpRIEQRA8PDyEV155pcEaAAj/93//p3tcXFwsABAOHjzYav0kovbDMTdEJLqHH34Yq1at0lvn7Oys+//g4GC954KDg5GYmAgAuH79OgICAmBvb697fsSIEdBqtbh58yYkEgmysrIwbtw4gzUMGDBA9//29vaQy+XIzc1taZeISEQMN0QkOnt7+zqniVqLra1tk9pZWVnpPZZIJNBqtW1REhG1MY65ISKjFx8fX+dxnz59AAB9+vTBlStXUFJSonv+7NmzkEql8PX1hYODA7y9vXHs2LF2rZmIxMMjN0QkuoqKCqhUKr11lpaW6NKlCwBgx44dGDJkCEaOHImvv/4a58+fx5dffgkAmDNnDl5//XWEh4fjjTfewJ07d/D8889j3rx5cHV1BQC88cYbePbZZ+Hi4oJJkyahqKgIZ8+exfPPP9++HSWidsFwQ0SiO3ToENzd3fXW+fr64saNGwBqrmTaunUrnnvuObi7u+Obb75B3759AQB2dnY4fPgwli1bhqFDh8LOzg7Tp0/Hf/7zH91rhYeHo7y8HB9++CFeeukldOnSBU899VT7dZCI2pVEEARB7CKIiBoikUiwe/duhIaGil0KEZkIjrkhIiIis8JwQ0RERGaFY26IyKjxzDkRNReP3BAREZFZYbghIiIis8JwQ0RERGaF4YaIiIjMCsMNERERmRWGGyIiIjIrDDdERERkVhhuiIiIyKww3BAREZFZ+X9611boj/snDQAAAABJRU5ErkJggg==\n", "text/plain": [ "
" - ], - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTlUlEQVR4nO3deVwTZ/4H8E/CEQ5JAJVLIqhYUFFUPADPVq1V60pr16MqaD22W+zqtt2ubH+97Hbpsd3eq1arVK31rLr1xgNPqIpi0aotlVMIqBzhPpL5/YGkTYFwCEwSPu/Xa15tJs8k34ch5sPMM89IBEEQQERERGQmpGIXQERERNSaGG6IiIjIrDDcEBERkVlhuCEiIiKzwnBDREREZoXhhoiIiMwKww0RERGZFYYbIiIiMisMN0RERGRWGG6IjMT8+fPh7e3dom3feOMNSCSS1i2IqBG1v3d3794VuxQiPQw3RI2QSCRNWmJjY8UuVRTz589Hp06dxC6jSQRBwKZNmzB69Gg4OjrCzs4O/fv3x8qVK1FSUiJ2eXXUhoeGFpVKJXaJREbJUuwCiIzdpk2b9B5v3LgRMTExddb36dPngd5n7dq10Gq1Ldr2//7v/7BixYoHen9zp9Fo8PTTT2P79u0YNWoU3njjDdjZ2eH06dN48803sWPHDhw9ehSurq5il1rHqlWr6g2Qjo6O7V8MkQlguCFqxNy5c/Uex8fHIyYmps763ystLYWdnV2T38fKyqpF9QGApaUlLC35cTbkvffew/bt2/HSSy/h/fff161fsmQJZsyYgdDQUMyfPx8HDx5s17qa8nvy1FNPoUuXLu1UEZHp42kpolYwduxY+Pv7IyEhAaNHj4adnR3+8Y9/AAD27t2LKVOmwMPDAzKZDL169cJbb70FjUaj9xq/H3OTmpoKiUSCf//73/jiiy/Qq1cvyGQyDB06FBcuXNDbtr4xNxKJBEuXLsWePXvg7+8PmUyGfv364dChQ3Xqj42NxZAhQ2BjY4NevXphzZo1rT6OZ8eOHQgMDIStrS26dOmCuXPn4vbt23ptVCoVFixYAE9PT8hkMri7u2PatGlITU3Vtbl48SImTpyILl26wNbWFj169MAzzzxj8L3Lysrw/vvv46GHHkJUVFSd56dOnYrw8HAcOnQI8fHxAIDHH38cPXv2rPf1goODMWTIEL11mzdv1vXP2dkZs2bNQkZGhl4bQ78nDyI2NhYSiQTbtm3DP/7xD7i5ucHe3h5/+MMf6tQANG1fAMCNGzcwY8YMdO3aFba2tvD19cUrr7xSp11BQQHmz58PR0dHKBQKLFiwAKWlpXptYmJiMHLkSDg6OqJTp07w9fVtlb4T1Yd/6hG1knv37mHSpEmYNWsW5s6dqzu9ER0djU6dOuGFF15Ap06dcPz4cbz22mtQq9V6RxAasmXLFhQVFeFPf/oTJBIJ3nvvPTz55JO4detWo0d7zpw5g2+//RbPPfccHBwc8Mknn2D69OlIT09H586dAQCXL1/GY489Bnd3d7z55pvQaDRYuXIlunbt+uA/lPuio6OxYMECDB06FFFRUcjJycHHH3+Ms2fP4vLly7rTK9OnT8e1a9fw/PPPw9vbG7m5uYiJiUF6erru8aOPPoquXbtixYoVcHR0RGpqKr799ttGfw75+flYtmxZg0e4wsLCsGHDBuzbtw9BQUGYOXMmwsLCcOHCBQwdOlTXLi0tDfHx8Xr77u2338arr76KGTNmYNGiRbhz5w4+/fRTjB49Wq9/QMO/J4bk5eXVWWdpaVnntNTbb78NiUSCv//978jNzcVHH32E8ePHIzExEba2tgCavi9++OEHjBo1ClZWVliyZAm8vb3xyy+/4LvvvsPbb7+t974zZsxAjx49EBUVhUuXLmHdunVwcXHBu+++CwC4du0aHn/8cQwYMAArV66ETCZDcnIyzp4922jfiVpEIKJmiYiIEH7/0RkzZowAQFi9enWd9qWlpXXW/elPfxLs7OyE8vJy3brw8HDBy8tL9zglJUUAIHTu3FnIy8vTrd+7d68AQPjuu+90615//fU6NQEQrK2theTkZN26K1euCACETz/9VLdu6tSpgp2dnXD79m3dup9//lmwtLSs85r1CQ8PF+zt7Rt8vrKyUnBxcRH8/f2FsrIy3fp9+/YJAITXXntNEARByM/PFwAI77//foOvtXv3bgGAcOHChUbr+q2PPvpIACDs3r27wTZ5eXkCAOHJJ58UBEEQCgsLBZlMJrz44ot67d577z1BIpEIaWlpgiAIQmpqqmBhYSG8/fbbeu2SkpIES0tLvfWGfk/qU7tf61t8fX117U6cOCEAELp16yao1Wrd+u3btwsAhI8//lgQhKbvC0EQhNGjRwsODg66ftbSarV16nvmmWf02jzxxBNC586ddY8//PBDAYBw586dJvWb6EHxtBRRK5HJZFiwYEGd9bV/MQNAUVER7t69i1GjRqG0tBQ3btxo9HVnzpwJJycn3eNRo0YBAG7dutXotuPHj0evXr10jwcMGAC5XK7bVqPR4OjRowgNDYWHh4eunY+PDyZNmtTo6zfFxYsXkZubi+eeew42Nja69VOmTIGfnx/2798PoObnZG1tjdjYWOTn59f7WrVHFfbt24eqqqom11BUVAQAcHBwaLBN7XNqtRoAIJfLMWnSJGzfvh2CIOjabdu2DUFBQejevTsA4Ntvv4VWq8WMGTNw9+5d3eLm5obevXvjxIkTeu/T0O+JIbt27UJMTIzesmHDhjrtwsLC9Pr41FNPwd3dHQcOHADQ9H1x584dnDp1Cs8884yun7XqO1X57LPP6j0eNWoU7t27p/tZ1u63vXv3tnjQPFFzMNwQtZJu3brB2tq6zvpr167hiSeegEKhgFwuR9euXXWDkQsLCxt93d9/udQGnYYCgKFta7ev3TY3NxdlZWXw8fGp066+dS2RlpYGAPD19a3znJ+fn+55mUyGd999FwcPHoSrqytGjx6N9957T+9y5zFjxmD69Ol488030aVLF0ybNg0bNmxARUWFwRpqv/BrQ0596gtAM2fOREZGBuLi4gAAv/zyCxISEjBz5kxdm59//hmCIKB3797o2rWr3nL9+nXk5ubqvU9DvyeGjB49GuPHj9dbgoOD67Tr3bu33mOJRAIfHx/dmKWm7ova8Ovv79+k+hr7HZ05cyZGjBiBRYsWwdXVFbNmzcL27dsZdKjNMNwQtZLfHqGpVVBQgDFjxuDKlStYuXIlvvvuO8TExOjGIjTlH3cLC4t61//2aEJbbCuG5cuX46effkJUVBRsbGzw6quvok+fPrh8+TKAmi/rnTt3Ii4uDkuXLsXt27fxzDPPIDAwEMXFxQ2+bu1l+j/88EODbWqf69u3r27d1KlTYWdnh+3btwMAtm/fDqlUij/+8Y+6NlqtFhKJBIcOHapzdCUmJgZr1qzRe5/6fk9MXWO/Z7a2tjh16hSOHj2KefPm4YcffsDMmTMxYcKEOgPriVoDww1RG4qNjcW9e/cQHR2NZcuW4fHHH8f48eP1TjOJycXFBTY2NkhOTq7zXH3rWsLLywsAcPPmzTrP3bx5U/d8rV69euHFF1/EkSNHcPXqVVRWVuKDDz7QaxMUFIS3334bFy9exNdff41r165h69atDdZQe5XOli1bGvwy3bhxI4Caq6Rq2dvb4/HHH8eOHTug1Wqxbds2jBo1Su8UXq9evSAIAnr06FHn6Mr48eMRFBTUyE+o9fz88896jwVBQHJysu4qvKbui9qrxK5evdpqtUmlUowbNw7/+c9/8OOPP+Ltt9/G8ePH65y2I2oNDDdEbaj2L9rfHimprKzEf//7X7FK0mNhYYHx48djz549yMrK0q1PTk5utflehgwZAhcXF6xevVrv9NHBgwdx/fp1TJkyBUDNfC/l5eV62/bq1QsODg667fLz8+scdRo4cCAAGDw1ZWdnh5deegk3b96s91Lm/fv3Izo6GhMnTqwTRmbOnImsrCysW7cOV65c0TslBQBPPvkkLCws8Oabb9apTRAE3Lt3r8G6WtvGjRv1Tr3t3LkT2dnZuvFTTd0XXbt2xejRo7F+/Xqkp6frvUdLjvrVd7VXU/YbUUvxUnCiNhQSEgInJyeEh4fjL3/5CyQSCTZt2mRUp4XeeOMNHDlyBCNGjMCf//xnaDQafPbZZ/D390diYmKTXqOqqgr//Oc/66x3dnbGc889h3fffRcLFizAmDFjMHv2bN3lx97e3vjrX/8KAPjpp58wbtw4zJgxA3379oWlpSV2796NnJwczJo1CwDw1Vdf4b///S+eeOIJ9OrVC0VFRVi7di3kcjkmT55ssMYVK1bg8uXLePfddxEXF4fp06fD1tYWZ86cwebNm9GnTx989dVXdbabPHkyHBwc8NJLL8HCwgLTp0/Xe75Xr1745z//icjISKSmpiI0NBQODg5ISUnB7t27sWTJErz00ktN+jk2ZOfOnfXOUDxhwgS9S8mdnZ0xcuRILFiwADk5Ofjoo4/g4+ODxYsXA6iZKLIp+wIAPvnkE4wcORKDBw/GkiVL0KNHD6SmpmL//v1N/r2otXLlSpw6dQpTpkyBl5cXcnNz8d///heenp4YOXJky34oRIaIco0WkQlr6FLwfv361dv+7NmzQlBQkGBrayt4eHgIL7/8snD48GEBgHDixAldu4YuBa/v0mgAwuuvv6573NCl4BEREXW29fLyEsLDw/XWHTt2TBg0aJBgbW0t9OrVS1i3bp3w4osvCjY2Ng38FH4VHh7e4OXKvXr10rXbtm2bMGjQIEEmkwnOzs7CnDlzhMzMTN3zd+/eFSIiIgQ/Pz/B3t5eUCgUwvDhw4Xt27fr2ly6dEmYPXu20L17d0EmkwkuLi7C448/Lly8eLHROgVBEDQajbBhwwZhxIgRglwuF2xsbIR+/foJb775plBcXNzgdnPmzBEACOPHj2+wza5du4SRI0cK9vb2gr29veDn5ydEREQIN2/e1LUx9HtSH0OXgv/296f2UvBvvvlGiIyMFFxcXARbW1thypQpdS7lFoTG90Wtq1evCk888YTg6Ogo2NjYCL6+vsKrr75ap77fX+K9YcMGAYCQkpIiCELN79e0adMEDw8PwdraWvDw8BBmz54t/PTTT03+WRA1h0QQjOhPSCIyGqGhobh27VqdcRxkfGJjY/Hwww9jx44deOqpp8Quh0h0HHNDRCgrK9N7/PPPP+PAgQMYO3asOAURET0AjrkhIvTs2RPz589Hz549kZaWhlWrVsHa2hovv/yy2KURETUbww0R4bHHHsM333wDlUoFmUyG4OBg/Otf/6ozKRwRkSngmBsiIiIyKxxzQ0RERGaF4YaIiIjMSocbc6PVapGVlQUHB4d6725LRERExkcQBBQVFcHDwwNSqeFjMx0u3GRlZUGpVIpdBhEREbVARkYGPD09DbYxmnDzzjvvIDIyEsuWLcNHH31Ub5vo6GgsWLBAb51MJqtzPxpDHBwcANT8cORyeYvrJSIiovajVquhVCp13+OGGEW4uXDhAtasWYMBAwY02lYul+vd0ba5p5Zq28vlcoYbIiIiE9OU733RBxQXFxdjzpw5WLt2LZycnBptL5FI4Obmplt+e9M4IiIiItHDTUREBKZMmYLx48c3qX1xcTG8vLygVCoxbdo0XLt2zWD7iooKqNVqvYWIiIjMl6jhZuvWrbh06RKioqKa1N7X1xfr16/H3r17sXnzZmi1WoSEhCAzM7PBbaKioqBQKHQLBxMTERGZN9FmKM7IyMCQIUMQExOjG2szduxYDBw4sMEBxb9XVVWFPn36YPbs2XjrrbfqbVNRUYGKigrd49oBSYWFhRxzQ0REZCLUajUUCkWTvr9FG1CckJCA3NxcDB48WLdOo9Hg1KlT+Oyzz1BRUQELCwuDr2FlZYVBgwYhOTm5wTYymQwymazV6iYiIiLjJlq4GTduHJKSkvTWLViwAH5+fvj73//eaLABasJQUlISJk+e3FZlEhERkYkRLdw4ODjA399fb529vT06d+6sWx8WFoZu3brpxuSsXLkSQUFB8PHxQUFBAd5//32kpaVh0aJF7V4/ERERGSejmOemIenp6XpTLOfn52Px4sVQqVRwcnJCYGAgzp07h759+4pYJRERERkT0QYUi6U5A5KIiIjIODTn+1v0eW6IiIiIWhPDDREREZkVhhsiIiIyKww3rUhVWI7r2by9AxERkZgYblrJwaRsjHrvOF7ZndR4YyIiImozDDetJNDbCRJIcCm9AAlpeWKXQ0RE1GEx3LQSFwcbhA7yAACsPZUicjVEREQdF8NNK1o0qicA4PCPKqTdKxG5GiIioo6J4aYVPeTqgLG+XSEIwPozPHpDREQkBoabVrb4/tGb7RczUVBaKXI1REREHQ/DTSsL6dUZfd3lKKvS4Ovv08Uuh4iIqMNhuGllEokEi0f3AABEn0tFRbVG5IqIiIg6FoabNvD4AA+4yW1wp6gCexOzxC6HiIioQ2G4aQNWFlIsGOENAPjydAo62I3XiYiIRMVw00ZmDesOe2sL3Mwpwqmf74pdDhERUYfBcNNGFLZWmDm0OwBg7albIldDRETUcTDctKEFI7xhIZXgTPJd/JjFG2oSERG1B4abNqR0tsMkfzcAwLozPHpDRETUHhhu2ljtpH7/S8yCqrBc5GqIiIjMH8NNGwtQOmJYD2dUawVEn0sVuxwiIiKzx3DTDmqP3mz5Pg3FFdUiV0NERGTeGG7awTg/F/TsYg91eTW2X8gQuxwiIiKzxnDTDqRSCRaOqrklw/qzKajWaEWuiIiIyHwx3LST6YM94Wxvjcz8Mhy+liN2OURERGaL4aad2FhZYG6QFwDgi9O3eEsGIiKiNsJw047Cgr1gbSnFlYwCXEzLF7scIiIis8Rw0466dJJh+uBuAHhLBiIiorbCcNPOFo6suSw85noOUu6WiFwNERGR+WG4aWc+Lp3wiJ8LBAH4krdkICIianUMNyKondRvZ0Im8ksqRa6GiIjIvDDciCCopzP8u8lRXqXF5vg0scshIiIyK0YTbt555x1IJBIsX77cYLsdO3bAz88PNjY26N+/Pw4cONA+BbYiiUSiO3rzVVwqyqs0IldERERkPowi3Fy4cAFr1qzBgAEDDLY7d+4cZs+ejYULF+Ly5csIDQ1FaGgorl692k6Vtp7J/d3hobDB3eJK7E28LXY5REREZkP0cFNcXIw5c+Zg7dq1cHJyMtj2448/xmOPPYa//e1v6NOnD9566y0MHjwYn332WTtV23qsLKRYMKLmlgxrT6dAq+WkfkRERK1B9HATERGBKVOmYPz48Y22jYuLq9Nu4sSJiIuLa3CbiooKqNVqvcVYzBymRCeZJZJzi3Hypztil0NERGQWRA03W7duxaVLlxAVFdWk9iqVCq6urnrrXF1doVKpGtwmKioKCoVCtyiVygequTXJbawwe1hNPWtP87JwIiKi1iBauMnIyMCyZcvw9ddfw8bGps3eJzIyEoWFhbolIyOjzd6rJeaP6AELqQTnfrmHq7cLxS6HiIjI5IkWbhISEpCbm4vBgwfD0tISlpaWOHnyJD755BNYWlpCo6l7BZGbmxtycvTvqJ2TkwM3N7cG30cmk0Eul+stxqSboy2m9HcHAKzj0RsiIqIHJlq4GTduHJKSkpCYmKhbhgwZgjlz5iAxMREWFhZ1tgkODsaxY8f01sXExCA4OLi9ym4TtZeF7/shG9mFZSJXQ0REZNosxXpjBwcH+Pv7662zt7dH586ddevDwsLQrVs33ZicZcuWYcyYMfjggw8wZcoUbN26FRcvXsQXX3zR7vW3pv6eCgT1dEb8rTxEn01F5OQ+YpdERERkskS/WsqQ9PR0ZGdn6x6HhIRgy5Yt+OKLLxAQEICdO3diz549dUKSKao9erPl+3QUlVeJXA0REZHpkgiC0KEmWFGr1VAoFCgsLDSq8TdarYAJH57EL3dK8H9T+mDR/bBDREREzfv+NuojNx2JVCrRBZoNZ1NRrdGKXBEREZFpYrgxIk8M6obO9ta4XVCGg1cbnruHiIiIGsZwY0RsrCwwL9gLQM2kfh3sjCEREVGrYLgxMvOCvCCzlOKHzEKcT8kTuxwiIiKTw3BjZDp3kmF6oCeAmhtqEhERUfMw3BihhSNr7hZ+9HoOfrlTLHI1REREpoXhxgj16toJ4/vU3CD0yzM8ekNERNQcDDdGavGomqM3uxIyca+4QuRqiIiITAfDjZEa1sMZAzwVqKjWYlN8mtjlEBERmQyGGyMlkUh0t2TYFJeG8qq6d0knIiKiuhhujNgkfzd0c7TFvZJK7L58W+xyiIiITALDjRGztJBiwQhvADWT+mm1nNSPiIioMQw3Rm7WsO5wsLHErTslOHEzV+xyiIiIjB7DjZHrJLPE08O6A6g5ekNERESGMdyYgPkjvGEplSD+Vh6SMgvFLoeIiMioMdyYAHeFLaYGeADg0RsiIqLGMNyYiEX3J/Xbn5SN2wVlIldDRERkvBhuTEQ/DwVCenWGRisg+ixvyUBERNQQhhsTsnh0zaR+35zPgLq8SuRqiIiIjBPDjQkZ+1BX9HbphOKKamw7nyF2OUREREaJ4caESCQS3dibDWdTUKXRilwRERGR8WG4MTHTBnZDl07WyCosx4GkbLHLISIiMjoMNybGxsoC4cHeAGouCxcE3pKBiIjotxhuTNDcIC/YWElx9bYa8bfyxC6HiIjIqDDcmCAne2s8FegJgJP6ERER/R7DjYlaOLInJBLg+I1cJOcWiV0OERGR0WC4MVE9uthjQh9XAMCXZzipHxERUS2GGxNWO6nfrku3caeoQuRqiIiIjAPDjQkb4uWEgUpHVFZrsSk+TexyiIiIjALDjQmTSCRYPKrm6M3m+DSUVWpEroiIiEh8DDcmbmI/V3g62SKvpBK7LmWKXQ4REZHoGG5MnKWFFAtH1tySYf2ZFGi1nNSPiIg6NlHDzapVqzBgwADI5XLI5XIEBwfj4MGDDbaPjo6GRCLRW2xsbNqxYuM0Y4gSchtL3LpbgmM3csUuh4iISFSihhtPT0+88847SEhIwMWLF/HII49g2rRpuHbtWoPbyOVyZGdn65a0NA6ktZdZ4unhXgCAtac4qR8REXVsooabqVOnYvLkyejduzceeughvP322+jUqRPi4+Mb3EYikcDNzU23uLq6tmPFxmt+iDesLCQ4n5qHKxkFYpdDREQkGqMZc6PRaLB161aUlJQgODi4wXbFxcXw8vKCUqls9CgPAFRUVECtVust5shNYYOpAR4AeEsGIiLq2EQPN0lJSejUqRNkMhmeffZZ7N69G3379q23ra+vL9avX4+9e/di8+bN0Gq1CAkJQWZmw1cJRUVFQaFQ6BalUtlWXRHdopE1l4UfvKpCRl6pyNUQERGJQyIIgqiX11RWViI9PR2FhYXYuXMn1q1bh5MnTzYYcH6rqqoKffr0wezZs/HWW2/V26aiogIVFb/O3qtWq6FUKlFYWAi5XN5q/TAW8778Hqd/votnRvTAa1Mb/xkSERGZArVaDYVC0aTvb9GP3FhbW8PHxweBgYGIiopCQEAAPv744yZta2VlhUGDBiE5ObnBNjKZTHc1Vu1izhbdn9Rv24V0FJZViVwNERFR+xM93PyeVqvVO9JiiEajQVJSEtzd3du4KtMxuncX+Lo6oKRSg63n08Uuh4iIqN2JGm4iIyNx6tQppKamIikpCZGRkYiNjcWcOXMAAGFhYYiMjNS1X7lyJY4cOYJbt27h0qVLmDt3LtLS0rBo0SKxumB0JBIJFo6qmdRvw9lUVFZrRa6IiIiofVmK+ea5ubkICwtDdnY2FAoFBgwYgMOHD2PChAkAgPT0dEilv+av/Px8LF68GCqVCk5OTggMDMS5c+eaND6nI5k20APvH74Jlboc+5Oy8MQgT7FLIiIiajeiDyhub80ZkGTKPj+RjPcP30Rfdzn2/2UkJBKJ2CURERG1mEkNKKa2MWd4d9haWeDHbDXO/XJP7HKIiIjaDcONmXK0s8aMITWnozipHxERdSQMN2bsmZE9IJEAsTfv4KecIrHLISIiahcMN2bMq7M9JvZ1AwCs49EbIiLqIBhuzNzi0TWT+u25nIXconKRqyEiImp7DDdmLtDLCYO7O6JSo8WmuDSxyyEiImpzDDcdwOL7t2TYFJ+G0spqkashIiJqWww3HcCj/dzQ3dkOBaVV2JXQ8B3UiYiIzAHDTQdgIZVg4ciaWzJ8eSYFGm2HmreRiIg6GIabDuKPQzyhsLVC6r1SHL2eI3Y5REREbYbhpoOws7bE3KDuAIC1p3hZOBERmS+Gmw4kPNgb1hZSXEzLx+X0fLHLISIiahMMNx2Ii9wGfxjoAQBYdzpF5GqIiIjaBsNNB1N7WfjBq9nIyCsVuRoiIqLWx3DTwfi6OWD0Q12hFWqunCIiIjI3DDcd0OJRNZeFb7+YgcLSKpGrISIial0MNx3QSJ8u8HNzQGmlBl+f5y0ZiIjIvDDcdEASiUQ39uarc6morNaKXBEREVHrYbjpoKYGeMBVLkOOugLfXckSuxwiIqJWw3DTQVlbShEe4g0AWHv6FgSBt2QgIiLzwHDTgc0Z5gU7awvcUBXhTPJdscshIiJqFQw3HZjCzgozhigBAGs5qR8REZkJhpsObuHIHpBKgFM/3cENlVrscoiIiB4Yw00Hp3S2wyR/dwC8JQMREZkHhhvCovuT+u1NvI1cdbnI1RARET0YhhvCoO5OGOLlhCqNgOhzqWKXQ0RE9EAYbggAsHh0zaR+X3+fjtLKapGrISIiajmGGwIAjO/jCu/Odigsq8KOi5lil0NERNRiDDcEALCQSrBwZM3Ymy/PpECj5aR+RERkmhhuSOepQCWc7KyQnleKI9dUYpdDRETUIgw3pGNrbYG5QV4Aam7JQEREZIpEDTerVq3CgAEDIJfLIZfLERwcjIMHDxrcZseOHfDz84ONjQ369++PAwcOtFO1HcO8YC9YW0hxKb0ACWl5YpdDRETUbKKGG09PT7zzzjtISEjAxYsX8cgjj2DatGm4du1ave3PnTuH2bNnY+HChbh8+TJCQ0MRGhqKq1evtnPl5svFwQahgzwAAGtPcVI/IiIyPRLByG4H7ezsjPfffx8LFy6s89zMmTNRUlKCffv26dYFBQVh4MCBWL16dZNeX61WQ6FQoLCwEHK5vNXqNic/5RTh0Q9PQSIBYl8aC6/O9mKXREREHVxzvr+NZsyNRqPB1q1bUVJSguDg4HrbxMXFYfz48XrrJk6ciLi4uAZft6KiAmq1Wm8hwx5ydcBY364QBGD9GR69ISIi0yJ6uElKSkKnTp0gk8nw7LPPYvfu3ejbt2+9bVUqFVxdXfXWubq6QqVq+MqeqKgoKBQK3aJUKlu1fnO1eFTNpH7bL2aioLRS5GqIiIiaTvRw4+vri8TERHz//ff485//jPDwcPz444+t9vqRkZEoLCzULRkZGa322uYspFdn9HWXo6xKg6+/Txe7HCIioiYTPdxYW1vDx8cHgYGBiIqKQkBAAD7++ON627q5uSEnJ0dvXU5ODtzc3Bp8fZlMprsaq3ahxkkkEiweXTOpX/S5VFRUa0SuiIiIqGlEDze/p9VqUVFRUe9zwcHBOHbsmN66mJiYBsfo0IN5fIAH3OQ2uFNUgb2JWWKXQ0RE1CSihpvIyEicOnUKqampSEpKQmRkJGJjYzFnzhwAQFhYGCIjI3Xtly1bhkOHDuGDDz7AjRs38MYbb+DixYtYunSpWF0wa1YWUiwY4Q0A+PJ0CozswjoiIqJ6iRpucnNzERYWBl9fX4wbNw4XLlzA4cOHMWHCBABAeno6srOzde1DQkKwZcsWfPHFFwgICMDOnTuxZ88e+Pv7i9UFszdrWHfYW1vgZk4RTv18V+xyiIiIGmV089y0Nc5z03wrv/sR68+mYKRPF2xeNFzscoiIqAMyyXluyHgtGOENC6kEZ5Lv4scszhNERETGjeGGGqV0tsMk/5or0tad4Q01iYjIuDHcUJPUTur3v8QsqArLRa6GiIioYQw31CQBSkcM6+GMaq2A6HOpYpdDRETUIIYbarLaozdbvk9DSUW1yNUQERHVj+GGmmycnwt6drGHurwa2y/yNhZERGScGG6oyaRSCRaOqrklw5dnUlCt0YpcERERUV0MN9Qs0wd7wtneGpn5ZTh8LafxDYiIiNoZww01i42VBeYGeQEAvjh9i7dkICIio8NwQ80WFuwFa0sprmQU4GJavtjlEBER6WG4oWbr0kmG6YO7AQDWnuKkfkREZFwYbqhFFo6suSw85noOUu6WiFwNERHRrxhuqEV8XDrhET8XCALwJW/JQERERoThhlqsdlK/nQmZyC+pFLkaIiKiGgw31GJBPZ3h302O8iotNseniV0OERERAIYbegASiUR39OaruFSUV2lEroiIiIjhhh7Q5P7u8FDY4G5xJfYm3ha7HCIiIoYbejBWFlIsGFFzS4a1p1Og1XJSPyIiEhfDDT2wmcOU6CSzRHJuMU7+dEfscoiIqINjuKEHJrexwuxhSgDA2tO8LJyIiMTFcEOtYv6IHrCQSnDul3u4llUodjlERNSBMdxQq+jmaIsp/d0BAOtOp4hcDRERdWQMN9Rqai8L/+5KFrILy0SuhoiIOiqGG2o1/T0VCOrpjGqtgOizqWKXQ0REHRTDDbWq2qM3W75PR1F5lcjVEBFRR8RwQ63qYV8X9Opqj6KKamy7kCF2OURE1AEx3FCrkkolWHT/6M2Gs6mo1mhFroiIiDoahhtqdU8M6obO9ta4XVCGg1dVYpdDREQdDMMNtTobKwvMC/YCUDOpnyDwlgxERNR+GG6oTcwL8oLMUoofMgtxPiVP7HKIiKgDYbihNtG5kwzTAz0B1NxQk4iIqL2IGm6ioqIwdOhQODg4wMXFBaGhobh586bBbaKjoyGRSPQWGxubdqqYmmPhyJq7hR+9noNf7hSLXA0REXUUooabkydPIiIiAvHx8YiJiUFVVRUeffRRlJSUGNxOLpcjOztbt6SlpbVTxdQcvbp2wvg+rgCAL8/w6A0REbUPSzHf/NChQ3qPo6Oj4eLigoSEBIwePbrB7SQSCdzc3Nq6PGoFi0f1wNHrOdiVkIkXJzyEzp1kYpdERERmrkVHbjIyMpCZmal7fP78eSxfvhxffPHFAxVTWFhzN2lnZ2eD7YqLi+Hl5QWlUolp06bh2rVrD/S+1HaG9XDGAE8FKqq12BTPI2xERNT2WhRunn76aZw4cQIAoFKpMGHCBJw/fx6vvPIKVq5c2aJCtFotli9fjhEjRsDf37/Bdr6+vli/fj327t2LzZs3Q6vVIiQkRC9s/VZFRQXUarXeQu1HIpHobsmwKS4N5VUakSsiIiJz16Jwc/XqVQwbNgwAsH37dvj7++PcuXP4+uuvER0d3aJCIiIicPXqVWzdutVgu+DgYISFhWHgwIEYM2YMvv32W3Tt2hVr1qypt31UVBQUCoVuUSqVLaqPWm6Svxu6OdriXkkldl++LXY5RERk5loUbqqqqiCT1YydOHr0KP7whz8AAPz8/JCdnd3s11u6dCn27duHEydOwNPTs1nbWllZYdCgQUhOTq73+cjISBQWFuqWjAze76i9WVpIsWCEN4CaSf20Wk7qR0REbadF4aZfv35YvXo1Tp8+jZiYGDz22GMAgKysLHTu3LnJryMIApYuXYrdu3fj+PHj6NGjR7Nr0Wg0SEpKgru7e73Py2QyyOVyvYXa36xh3eFgY4lbd0pw4mau2OUQEZEZa1G4effdd7FmzRqMHTsWs2fPRkBAAADgf//7n+50VVNERERg8+bN2LJlCxwcHKBSqaBSqVBWVqZrExYWhsjISN3jlStX4siRI7h16xYuXbqEuXPnIi0tDYsWLWpJV6iddJJZ4ulh3QHUHL0hIiJqKy26FHzs2LG4e/cu1Go1nJycdOuXLFkCOzu7Jr/OqlWrdK/3Wxs2bMD8+fMBAOnp6ZBKf81g+fn5WLx4MVQqFZycnBAYGIhz586hb9++LekKtaP5I7zx5ZkUxN/KQ1JmIfp7KsQuiYiIzJBEaMFdDcvKyiAIgi7IpKWlYffu3ejTpw8mTpzY6kW2JrVaDYVCgcLCQp6iEsFftyVi9+Xb+EOABz6ZPUjscoiIyEQ05/u7Raelpk2bho0bNwIACgoKMHz4cHzwwQcIDQ3VHY0hqs+iUTXjqvYnZeN2QVkjrYmIiJqvReHm0qVLGDVqFABg586dcHV1RVpaGjZu3IhPPvmkVQsk89LPQ4GQXp2h0QqIPstbMhARUetrUbgpLS2Fg4MDAODIkSN48sknIZVKERQUxPs8UaMWj66Z1O+b8xlQl1eJXA0REZmbFoUbHx8f7NmzBxkZGTh8+DAeffRRAEBubi7HsVCjxj7UFb1dOqG4ohrbznPeISIial0tCjevvfYaXnrpJXh7e2PYsGEIDg4GUHMUZ9AgDhIlwyQSiW7szYazKajSaEWuiIiIzEmLws1TTz2F9PR0XLx4EYcPH9atHzduHD788MNWK47M17SB3dClkzWyCstxIKn5s1oTERE1pEXhBgDc3NwwaNAgZGVl6W5aOWzYMPj5+bVacWS+bKwsEB7sDaBmUr8WzEhARERUrxaFG61Wi5UrV0KhUMDLywteXl5wdHTEW2+9Ba2WpxioaeYGecHGSoqrt9WIv5UndjlERGQmWhRuXnnlFXz22Wd45513cPnyZVy+fBn/+te/8Omnn+LVV19t7RrJTDnZW+OpwJobpfKWDERE1FpaNEOxh4cHVq9erbsbeK29e/fiueeew+3bt1utwNbGGYqNS8rdEjzyQSwEATj6wmj4uDiIXRIRERmhNp+hOC8vr96xNX5+fsjL4+kFaroeXewxoY8rAODLM5zUj4iIHlyLwk1AQAA+++yzOus/++wzDBgw4IGLoo6ldlK/XZdu405RhcjVEBGRqWvRXcHfe+89TJkyBUePHtXNcRMXF4eMjAwcOHCgVQsk8zfEywkDlY5IzCjApvg0vDDhIbFLIiIiE9aiIzdjxozBTz/9hCeeeAIFBQUoKCjAk08+iWvXrmHTpk2tXSOZOYlEgsWjao7ebI5PQ3mVRuSKiIjIlLVoQHFDrly5gsGDB0OjMd4vJw4oNk7VGi3G/jsWmfllePsJf8wZ7iV2SUREZETafEAxUWuztJBi4ciaWzJ8eToFWi0n9SMiopZhuCGjMWOIEnIbS9y6W4JjN3LFLoeIiEwUww0ZDXuZJZ6+fzqKk/oREVFLNetqqSeffNLg8wUFBQ9SCxHmh3jjyzO3cD4lD1cyChCgdBS7JCIiMjHNOnKjUCgMLl5eXggLC2urWqkDcFPYYGqABwAevSEiopZp1pGbDRs2tFUdRDqLRvbEt5du4+BVFTLySqF0thO7JCIiMiEcc0NGp6+HHKN6d4FGK2DD2VSxyyEiIhPDcENGadH9Sf22XUhHYVmVyNUQEZEpYbghozS6dxf4ujqgpFKDrefTxS6HiIhMCMMNGSWJRIKFo2om9dtwNhWV1VqRKyIiIlPBcENGa9pAD3R1kEGlLsf+pCyxyyEiIhPBcENGS2Zpgfkh3gCAtadS0Iq3QSMiIjPGcENGbc7w7rC1ssCP2Wqc++We2OUQEZEJYLgho+ZoZ40ZQzwBcFI/IiJqGoYbMnrPjOwBiQSIvXkHP+UUiV0OEREZOYYbMnpene0xsa8bAGAdj94QEVEjGG7IJCweXTOp357LWcgtKhe5GiIiMmaihpuoqCgMHToUDg4OcHFxQWhoKG7evNnodjt27ICfnx9sbGzQv39/HDhwoB2qJTEFejlhcHdHVGq02BSXJnY5RERkxEQNNydPnkRERATi4+MRExODqqoqPProoygpKWlwm3PnzmH27NlYuHAhLl++jNDQUISGhuLq1avtWDmJYfH9WzJsik9DWaVG5GqIiMhYSQQjmjzkzp07cHFxwcmTJzF69Oh628ycORMlJSXYt2+fbl1QUBAGDhyI1atXN/oearUaCoUChYWFkMvlrVY7tT2NVsDD/45Fel4p3prWD/OCvcUuiYiI2klzvr+NasxNYWEhAMDZ2bnBNnFxcRg/frzeuokTJyIuLq7e9hUVFVCr1XoLmSYLqQQLR9bckuHLMynQaI0mlxMRkRExmnCj1WqxfPlyjBgxAv7+/g22U6lUcHV11Vvn6uoKlUpVb/uoqCgoFArdolQqW7Vual9/HOIJha0VUu+V4uj1HLHLISIiI2Q04SYiIgJXr17F1q1bW/V1IyMjUVhYqFsyMjJa9fWpfdlZW2JuUHcAwNpTvCyciIjqMopws3TpUuzbtw8nTpyAp6enwbZubm7IydH/iz0nJwdubm71tpfJZJDL5XoLmbbwYG9YW0hxMS0fl9PzxS6HiIiMjKjhRhAELF26FLt378bx48fRo0ePRrcJDg7GsWPH9NbFxMQgODi4rcokI+Mit8EfBnoAANadThG5GiIiMjaihpuIiAhs3rwZW7ZsgYODA1QqFVQqFcrKynRtwsLCEBkZqXu8bNkyHDp0CB988AFu3LiBN954AxcvXsTSpUvF6AKJZNGomiB88Go2MvJKRa6GiIiMiajhZtWqVSgsLMTYsWPh7u6uW7Zt26Zrk56ejuzsbN3jkJAQbNmyBV988QUCAgKwc+dO7Nmzx+AgZDI/fm5yjH6oK7RCzZVTREREtYxqnpv2wHluzMfpn+9g3pfnYWdtgbgV46CwsxK7JCIiaiMmO88NUXOM9OkCPzcHlFZq8PV53pKBiIhqMNyQyZJIJLpbMnx1LhWV1VqRKyIiImPAcEMmbWqAB1zlMuSoK/DdlSyxyyEiIiPAcEMmzdpSivAQbwDA2tO30MGGkBERUT0YbsjkzRnmBTtrC9xQFeFM8l2xyyEiIpEx3JDJU9hZYcaQmnuGreWkfkREHR7DDZmFhSN7QCoBTv10BzdVRWKXQ0REImK4IbOgdLbDJH93ADVjb4iIqONiuCGzUXtLhr2Jt5GrLhe5GiIiEgvDDZmNQd2dMMTLCVUaAV/FpYpdDhERiYThhszK4tE1k/ptjk9HaWW1yNUQEZEYGG7IrIzv4wrvznYoLKvCjouZYpdDREQiYLghs2IhlWDhyJqxN1+eSYFGy0n9iIg6GoYbMjtPBSrhZGeF9LxSHLmmErscIiJqZww3ZHZsrS0wN8gLAC8LJyLqiBhuyCzNC/aCtYUUl9ILkJCWJ3Y5RETUjhhuyCy5ONggdJAHAGDtKd6SgYioI2G4IbO1aFTNZeGHf1Qh7V6JyNUQEVF7Ybghs/WQqwPG+naFIADrz/DoDRFRR8FwQ2Zt8f2jN9svZqKgtFLkaoiIqD0w3JBZC+nVGX3d5Sir0uDr79PFLoeIiNoBww2ZNYlEgsWjayb1iz6XiopqjcgVERFRW2O4IbP3+AAPuMltcKeoAnsTs8Quh4iI2hjDDZk9KwspFozwBgB8eToFgsBbMhARmTOGG+oQZg3rDntrC9zMKcKpn++KXQ4REbUhhhvqEBS2Vpg5tDsAYB1vyUBEZNYYbqjDWDDCGxZSCU7/fBc/ZqnFLoeIiNoIww11GEpnO0zydwMArDvDozdEROaK4YY6lNpJ/b67kgVVYbnI1RARUVtguKEOJUDpiGE9nFGlERB9LlXscoiIqA0w3FCHU3v0Zsv3aSipqBa5GiIiam2ihptTp05h6tSp8PDwgEQiwZ49ewy2j42NhUQiqbOoVKr2KZjMwjg/F/TsYg91eTW2X8wQuxwiImplooabkpISBAQE4PPPP2/Wdjdv3kR2drZucXFxaaMKyRxJpRIsHFVzS4Yvz6SgWqMVuSIiImpNlmK++aRJkzBp0qRmb+fi4gJHR8fWL4g6jOmDPfHBkZ+QmV+Gw9dyMGWAu9glERFRKzHJMTcDBw6Eu7s7JkyYgLNnz4pdDpkgGysLzA3yAgB8cfoWb8lARGRGTCrcuLu7Y/Xq1di1axd27doFpVKJsWPH4tKlSw1uU1FRAbVarbcQAUBYsBesLaW4klGAi2n5YpdDREStxKTCja+vL/70pz8hMDAQISEhWL9+PUJCQvDhhx82uE1UVBQUCoVuUSqV7VgxGbMunWSYPrgbAGDtKU7qR0RkLkwq3NRn2LBhSE5ObvD5yMhIFBYW6paMDF4dQ79aOLLmsvCY6zlIuVsicjVERNQaTD7cJCYmwt294cGgMpkMcrlcbyGq5ePSCY/4uUAQgC95SwYiIrMg6tVSxcXFekddUlJSkJiYCGdnZ3Tv3h2RkZG4ffs2Nm7cCAD46KOP0KNHD/Tr1w/l5eVYt24djh8/jiNHjojVBTIDi0f1xPEbudiZkIkXJ/jCyd5a7JKIiOgBiHrk5uLFixg0aBAGDRoEAHjhhRcwaNAgvPbaawCA7OxspKen69pXVlbixRdfRP/+/TFmzBhcuXIFR48exbhx40Spn8xDUE9n+HeTo7xKi83xaWKXQ0RED0gidLBrYNVqNRQKBQoLC3mKinT2Jt7Gsq2J6NJJhjN/fxg2VhZil0RERL/RnO9vkx9zQ9QaJvd3h4fCBneLK/DK7qtI5eBiIiKTxXBDBMDKQoqIR3wAALsuZWLsv2Mxf8N5HL+RA622Qx3cJCIyeaIOKCYyJnOGe8HD0RYbz6Ui9qc7iL1Zs3R3tsO8IC/8cYgnHO042JiIyNhxzA1RPVLvlmBzfBq2X8yAurwaACCzlCJ0YDfMC/aCfzeFyBUSEXUszfn+ZrghMqCsUoO9ibexMS4NP2b/euuOQC8nhAV7YZK/O6wteXaXiKitMdwYwHBDLSEIAhLS8rExLg0HkrJRfX8cTpdOMswepsTTw7vDXWErcpVEROaL4cYAhht6ULlF5dh6PgNff5+GHHUFAMBCKsHEfq6YF+SNoJ7OkEgkIldJRGReGG4MYLih1lKl0eLItRxsjEvF9yl5uvUPuXbCvGBvPDmoG+xlHLNPRNQaGG4MYLihtnBDpcamuDTsvnwbpZUaAICDzBLTAz0xL9gLvbp2ErlCIiLTxnBjAMMNtSV1eRV2JWRiU1wabv1mIsBRvbtgXpAXxvVxhYWUp6yIiJqL4cYAhhtqD1qtgLO/3MVX59Jw7EYOaj9l3RxtMSeoO2YN7Q5n3qCTiKjJGG4MYLih9paRV4qvv0/HtgvpyC+tAgBYW0oxdYAHwoK9EKB0FLdAIiITwHBjAMMNiaW8SoPvrmRhY1wakm4X6tYHKB0RFuSFKQPcecNOIqIGMNwYwHBDYhMEAYkZBdgUl4Z9P2SjUqMFADjbW2PmUCXmDO8OTyc7kaskIjIuDDcGMNyQMblbXIFtFzLwdXwasgrLAQBSCTCujyvCg70xwqcz58whIgLDjUEMN2SMqjVaHLuRi41xqTibfE+3vldXe8wL8sL0QE842FiJWCERkbgYbgxguCFjl5xbhE1xadiZkImS+3Pm2Ftb4InB3RAW7I2HXB1ErpCIqP0x3BjAcEOmoriiGrsvZeKruDQk5xbr1gf37IywYC9M6OsKSwvetJOIOgaGGwMYbsjUCIKAuFv3sPFcGmKu50Bz/6ad7gobPD2sO2YN646uDjKRqyQialsMNwYw3JApyyoow5bv0/HN+XTcK6kEAFhZSDC5vzvCgr0xuLsjByATkVliuDGA4YbMQUW1BgeTVPgqLhWX0wt06/27yREW5I0/DPTgnDlEZFYYbgxguCFzk5RZiI1xqfjflSxUVNfMmeNoZ4UZQ5SYO9wL3TtzzhwiMn0MNwYw3JC5yi+pxPaLGdgUn4bM/DIAgEQCPOzrgrBgL4zu3RVS3rSTiEwUw40BDDdk7jRaAbE3c/FVXBpO/XRHt967sx3mBnnhj4FKKOw4Zw4RmRaGGwMYbqgjuXWnGJvj07EjIQNF5dUAAFsrC4QO8sC8IG/09eBngIhMA8ONAQw31BGVVlZjz+UsbIxLxQ1VkW79MG9nzAv2wmP+brDinDlEZMQYbgxguKGOTBAEXEjNx1dxqTh8VYXq+3PmuDjIMHtYdzw9vDtc5TYiV0lEVBfDjQEMN0Q1ctTl2PJ9OracT8edogoAgKVUgon+bggP9sZQbyfOmUNERoPhxgCGGyJ9ldVaHL6mwsa4VFxIzdet93NzQFiwN0IHecDO2lLEComIGG4MYrghatiPWWpsik/FnstZKKuquWmng40l/hioxLxgL/ToYi9yhUTUUTHcGMBwQ9S4wtIq7EjIwOb4NKTeK9WtH/1QV4QHe2GsrwssOGcOEbUjhhsDGG6Imk6rFXDq5zvYGJeGEzdzUfuvhdLZFnOHe2HGECWc7K3FLZKIOoTmfH+Leu3nqVOnMHXqVHh4eEAikWDPnj2NbhMbG4vBgwdDJpPBx8cH0dHRbV4nUUcllUow1tcF6+cPxcmXHsaS0T2hsLVCRl4Zog7eQFDUMfxtxxVcvV0odqlERDqihpuSkhIEBATg888/b1L7lJQUTJkyBQ8//DASExOxfPlyLFq0CIcPH27jSomoe2c7/GNyH8RHjsN70wegn4ccFdVa7EjIxOOfnsET/z2LPZdvo6JaI3apRNTBGc1pKYlEgt27dyM0NLTBNn//+9+xf/9+XL16Vbdu1qxZKCgowKFDh5r0PjwtRdQ6BEHApfQCbIpLxf6kbFRpav4p6dLJGrOG1syZ4+FoK3KVRGQuTOa0VHPFxcVh/PjxeusmTpyIuLi4BrepqKiAWq3WW4jowUkkEgR6OeGjWYNwbsU4vDjhIbjJbXC3uBKfnUjGqPdO4NlNCTj3y10Yyd9QRNRBmFS4UalUcHV11Vvn6uoKtVqNsrKyereJioqCQqHQLUqlsj1KJepQujrI8Py43jjz94exas5gBPV0hkYr4NA1FZ5e+z0e/fAUNsWloriiWuxSiagDMKlw0xKRkZEoLCzULRkZGWKXRGS2LC2kmNTfHVuXBOPIX0djblB32Flb4OfcYry69xqC/nUMr++9iuTcYrFLJSIzZlLTjrq5uSEnJ0dvXU5ODuRyOWxt6z+3L5PJIJPJ2qM8IvqNh1wd8M/Q/nj5MT98m5CJjfFpuHWnBF/FpeGruDSM8OmMsGBvjPNzgSVv2klErcikwk1wcDAOHDigty4mJgbBwcEiVUREjZHbWGH+iB4ID/HG2eR72BiXiqPXc3A2+R7OJt+Dh8IGc4K8MGuoEp078Q8RInpwol4tVVxcjOTkZADAoEGD8J///AcPP/wwnJ2d0b17d0RGRuL27dvYuHEjgJpLwf39/REREYFnnnkGx48fx1/+8hfs378fEydObNJ78mopIvFl5pfi6+/Tse1CBvJKKgEA1hZSPD7AHWEh3hiodBS3QCIyOiYzQ3FsbCwefvjhOuvDw8MRHR2N+fPnIzU1FbGxsXrb/PWvf8WPP/4IT09PvPrqq5g/f36T35Phhsh4lFdpsP+HbGyMT8OVjALd+gGeCoQFe+PxAe6wsbIQr0AiMhomE27EwHBDZJwSMwqwMS4V+37IRmW1FgDgZGeFmUO7Y87w7lA624lcIRGJieHGAIYbIuOWV1KJbRdqbtp5u6BmigepBHjEzxXhIV4Y0asLpLxpJ1GHw3BjAMMNkWnQaAUcu56DTfFpOP3zXd36nl3sMS/YC9MDPSG3sRKxQiJqTww3BjDcEJme5NxibI5Pw86ETN1EgHbWFnhiUDeEBXvD181B5AqJqK0x3BjAcENkuoorqrH78m1sikvFTzm/TgQ4vIczwkO8MaGvK6w4Zw6RWWK4MYDhhsj0CYKA+Ft52BSfisPXcqDR1vwz5iqXYeYQJYJ6doa/p4KnrYjMCMONAQw3ROYlu7AM33yfji3n03G3uFLvuV5d7RGgdMRApSMCPB3h5+4AmSUvLScyRQw3BjDcEJmnimoNDl1V4ciPOfghswAZeXVvpmttIUUfDzkGeioQoHREgNIRPTrb8+orIhPAcGMAww1Rx3CvuAI/ZBYiMaMAVzILcCWjAPmlVXXaOcgsMUCpQICno+4oj6vcRoSKicgQhhsDGG6IOiZBEJCZX1YTdu4HnqTbhSiv0tZp6ya3QYCy5ujOQE9Hjt8hMgIMNwYw3BBRrWqNFj/lFOuO7CRmFOCnnCJo6/lXkeN3iMTFcGMAww0RGVJaWY1rWWpd2LnC8TtERoHhxgCGGyJqLo7fIRIfw40BDDdE9KAEQUBGXhkSMwvwA8fvELULhhsDGG6IqC1w/A5R22K4MYDhhojaS2llNa7eVuuuzuL4HaKWY7gxgOGGiMTE8TtELcNwYwDDDREZE47fIWoahhsDGG6IyNhx/A5RXQw3BjDcEJEp4vgd6ugYbgxguCEic8HxO9SRMNwYwHBDROaK43fInDHcGMBwQ0QdCcfvkLlguDGA4YaIOjqO3yFTxHBjAMMNEVFdHL9Dxo7hxgCGGyKixnH8DhkbhhsDGG6IiFqG43dITAw3BjDcEBG1nt+O30nMLMAPHL9DbYThxgCGGyKitsXxO9QWGG4MYLghImpfvx2/cyWjZrma1fD4nT7uDlA620HpZFfzX2dbKJ3tOI6ng2O4MYDhhohIfM0Zv1NLYWuF7rVhx8kOns52NY+dbNHNyZZjeswcw40BDDdERMapdvzOL3eKkZFXivS8UmTklyEzrxT3SioNbiuRAK4ONujubAfP++FHWRt+nG3h6mDD8T0mzuTCzeeff473338fKpUKAQEB+PTTTzFs2LB620ZHR2PBggV662QyGcrLy5v0Xgw3RESmp6SiGhn5pcjIK0NGXun9/7//OL8UpZUag9tbW0jRzcn2/ukuW91pr9rwo7C1gkTC8GPMmvP9bdlONTVo27ZteOGFF7B69WoMHz4cH330ESZOnIibN2/CxcWl3m3kcjlu3rype8xfSCIi82Yvs4Sfmxx+bnW/1ARBQF5Jpe5IT0ZeKTLz7x/5yStDVkEZKjVapNwtQcrdknpf30Fmef80l63eWJ/uznbwdLKDjRVPeZkS0Y/cDB8+HEOHDsVnn30GANBqtVAqlXj++eexYsWKOu2jo6OxfPlyFBQUtOj9eOSGiKhjqdZooVKXIz2vFJn3j/T89rTXnaKKRl+jq4MMSifb+0d67O6P+akJQu4KG1haSNuhJx2byRy5qaysREJCAiIjI3XrpFIpxo8fj7i4uAa3Ky4uhpeXF7RaLQYPHox//etf6NevX71tKyoqUFHx6y+uWq1uvQ4QEZHRs7SQwtOp5ggMetV9vrxKo3ekp/a0V3pezXifoopq3CmqwJ2iClxKL6j7+lIJPBxtdQOdlboAVHP6q7O9Nc8wtDNRw83du3eh0Wjg6uqqt97V1RU3btyodxtfX1+sX78eAwYMQGFhIf79738jJCQE165dg6enZ532UVFRePPNN9ukfiIiMn02VhbwcXGAj4tDnecEQUBhWRUy8sruH+m5P9bn/umv2/k1p7zS7x8JAu7VeQ07a4v7occWnrpxPr9e9WUvE32EiNkR9bRUVlYWunXrhnPnziE4OFi3/uWXX8bJkyfx/fffN/oaVVVV6NOnD2bPno233nqrzvP1HblRKpU8LUVERA9MqxWQU1T+a/i5H4Ay7z/OKSpHY9+yne2t4fmbIz3ddXP82MLD0RZWPOUFwIROS3Xp0gUWFhbIycnRW5+TkwM3N7cmvYaVlRUGDRqE5OTkep+XyWSQyWQPXCsREdHvSaUSuCts4a6wxbAeznWer6jW4HZ+GTLyy+6P+SnVXfWVnleKwrIq3CupxL2SSlzJKKj7+hLAXaF/yuu3c/10dZDxlFc9RA031tbWCAwMxLFjxxAaGgqgZkDxsWPHsHTp0ia9hkajQVJSEiZPntyGlRIRETWfzNICPbt2Qs+unep9Xl1e9esl7fWc9qqo1uJ2QRluF5QhHnn1vL60zuXtnNXZCC4Ff+GFFxAeHo4hQ4Zg2LBh+Oijj1BSUqKbyyYsLAzdunVDVFQUAGDlypUICgqCj48PCgoK8P777yMtLQ2LFi0SsxtERETNJrexQj8PBfp5KOo8p9UKuFtcoXekJ+M3R36yC8tQUa1Fcm4xknOL6339jjqrs+jhZubMmbhz5w5ee+01qFQqDBw4EIcOHdINMk5PT4dU+uv5xvz8fCxevBgqlQpOTk4IDAzEuXPn0LdvX7G6QERE1OqkUglc5DZwkdsg0Kvu81UaLbIKynQTGf4afn6d1bmwrApJtwuRdLuwzvbmPKuz6PPctDfOc0NERB3B72d1Tr8/uaGpzupsMgOKiYiIqG00NqvzvZJKvfE9vz3lZeqzOvPIDREREemp1miRXViud1n7bwc7Nzars49LJxx9YUyr1sQjN0RERNRilhZS3UzL9c3qXFZZM6tz3cHONeN9lE627V/0bzDcEBERUbPYWlugt6sDervWP6tzRbVWhKp+xWkPiYiIqNVIJBLR76LOcENERERmheGGiIiIzArDDREREZkVhhsiIiIyKww3REREZFYYboiIiMisMNwQERGRWWG4ISIiIrPCcENERERmheGGiIiIzArDDREREZkVhhsiIiIyKww3REREZFYsxS6gvQmCAABQq9UiV0JERERNVfu9Xfs9bkiHCzdFRUUAAKVSKXIlRERE1FxFRUVQKBQG20iEpkQgM6LVapGVlQUHBwdIJJJWfW21Wg2lUomMjAzI5fJWfW1jYO79A8y/j+yf6TP3PrJ/pq+t+igIAoqKiuDh4QGp1PComg535EYqlcLT07NN30Mul5vtLy1g/v0DzL+P7J/pM/c+sn+mry362NgRm1ocUExERERmheGGiIiIzArDTSuSyWR4/fXXIZPJxC6lTZh7/wDz7yP7Z/rMvY/sn+kzhj52uAHFREREZN545IaIiIjMCsMNERERmRWGGyIiIjIrDDdERERkVhhumunzzz+Ht7c3bGxsMHz4cJw/f95g+x07dsDPzw82Njbo378/Dhw40E6Vtkxz+hcdHQ2JRKK32NjYtGO1zXPq1ClMnToVHh4ekEgk2LNnT6PbxMbGYvDgwZDJZPDx8UF0dHSb19lSze1fbGxsnf0nkUigUqnap+BmioqKwtChQ+Hg4AAXFxeEhobi5s2bjW5nSp/BlvTRlD6Hq1atwoABA3STuwUHB+PgwYMGtzGl/dfc/pnSvqvPO++8A4lEguXLlxtsJ8Y+ZLhphm3btuGFF17A66+/jkuXLiEgIAATJ05Ebm5uve3PnTuH2bNnY+HChbh8+TJCQ0MRGhqKq1evtnPlTdPc/gE1M1BmZ2frlrS0tHasuHlKSkoQEBCAzz//vEntU1JSMGXKFDz88MNITEzE8uXLsWjRIhw+fLiNK22Z5vav1s2bN/X2oYuLSxtV+GBOnjyJiIgIxMfHIyYmBlVVVXj00UdRUlLS4Dam9hlsSR8B0/kcenp64p133kFCQgIuXryIRx55BNOmTcO1a9fqbW9q+6+5/QNMZ9/93oULF7BmzRoMGDDAYDvR9qFATTZs2DAhIiJC91ij0QgeHh5CVFRUve1nzJghTJkyRW/d8OHDhT/96U9tWmdLNbd/GzZsEBQKRTtV17oACLt37zbY5uWXXxb69eunt27mzJnCxIkT27Cy1tGU/p04cUIAIOTn57dLTa0tNzdXACCcPHmywTam9hn8vab00ZQ/h4IgCE5OTsK6devqfc7U958gGO6fqe67oqIioXfv3kJMTIwwZswYYdmyZQ22FWsf8shNE1VWViIhIQHjx4/XrZNKpRg/fjzi4uLq3SYuLk6vPQBMnDixwfZiakn/AKC4uBheXl5QKpWN/oViakxp/z2IgQMHwt3dHRMmTMDZs2fFLqfJCgsLAQDOzs4NtjH1fdiUPgKm+TnUaDTYunUrSkpKEBwcXG8bU95/TekfYJr7LiIiAlOmTKmzb+oj1j5kuGmiu3fvQqPRwNXVVW+9q6trg2MUVCpVs9qLqSX98/X1xfr167F3715s3rwZWq0WISEhyMzMbI+S21xD+0+tVqOsrEykqlqPu7s7Vq9ejV27dmHXrl1QKpUYO3YsLl26JHZpjdJqtVi+fDlGjBgBf3//BtuZ0mfw95raR1P7HCYlJaFTp06QyWR49tlnsXv3bvTt27fetqa4/5rTP1PbdwCwdetWXLp0CVFRUU1qL9Y+7HB3BafWExwcrPcXSUhICPr06YM1a9bgrbfeErEyagpfX1/4+vrqHoeEhOCXX37Bhx9+iE2bNolYWeMiIiJw9epVnDlzRuxS2kxT+2hqn0NfX18kJiaisLAQO3fuRHh4OE6ePNlgADA1zemfqe27jIwMLFu2DDExMUY/8Jnhpom6dOkCCwsL5OTk6K3PycmBm5tbvdu4ubk1q72YWtK/37OyssKgQYOQnJzcFiW2u4b2n1wuh62trUhVta1hw4YZfWBYunQp9u3bh1OnTsHT09NgW1P6DP5Wc/r4e8b+ObS2toaPjw8AIDAwEBcuXMDHH3+MNWvW1GlrivuvOf37PWPfdwkJCcjNzcXgwYN16zQaDU6dOoXPPvsMFRUVsLCw0NtGrH3I01JNZG1tjcDAQBw7dky3TqvV4tixYw2eTw0ODtZrDwAxMTEGz7+KpSX9+z2NRoOkpCS4u7u3VZntypT2X2tJTEw02v0nCAKWLl2K3bt34/jx4+jRo0ej25jaPmxJH3/P1D6HWq0WFRUV9T5navuvPob693vGvu/GjRuHpKQkJCYm6pYhQ4Zgzpw5SExMrBNsABH3YZsOVzYzW7duFWQymRAdHS38+OOPwpIlSwRHR0dBpVIJgiAI8+bNE1asWKFrf/bsWcHS0lL497//LVy/fl14/fXXBSsrKyEpKUmsLhjU3P69+eabwuHDh4VffvlFSEhIEGbNmiXY2NgI165dE6sLBhUVFQmXL18WLl++LAAQ/vOf/wiXL18W0tLSBEEQhBUrVgjz5s3Ttb9165ZgZ2cn/O1vfxOuX78ufP7554KFhYVw6NAhsbpgUHP79+GHHwp79uwRfv75ZyEpKUlYtmyZIJVKhaNHj4rVBYP+/Oc/CwqFQoiNjRWys7N1S2lpqa6NqX8GW9JHU/ocrlixQjh58qSQkpIi/PDDD8KKFSsEiUQiHDlyRBAE099/ze2fKe27hvz+ailj2YcMN8306aefCt27dxesra2FYcOGCfHx8brnxowZI4SHh+u13759u/DQQw8J1tbWQr9+/YT9+/e3c8XN05z+LV++XNfW1dVVmDx5snDp0iURqm6a2kuff7/U9ik8PFwYM2ZMnW0GDhwoWFtbCz179hQ2bNjQ7nU3VXP79+677wq9evUSbGxsBGdnZ2Hs2LHC8ePHxSm+CerrGwC9fWLqn8GW9NGUPofPPPOM4OXlJVhbWwtdu3YVxo0bp/viFwTT33/N7Z8p7buG/D7cGMs+lAiCILTtsSEiIiKi9sMxN0RERGRWGG6IiIjIrDDcEBERkVlhuCEiIiKzwnBDREREZoXhhoiIiMwKww0RERGZFYYbIurwJBIJ9uzZI3YZRNRKGG6ISFTz58+HRCKpszz22GNil0ZEJop3BSci0T322GPYsGGD3jqZTCZSNURk6njkhohEJ5PJ4Obmprc4OTkBqDlltGrVKkyaNAm2trbo2bMndu7cqbd9UlISHnnkEdja2qJz585YsmQJiouL9dqsX78e/fr1g0wmg7u7O5YuXar3/N27d/HEE0/Azs4OvXv3xv/+97+27TQRtRmGGyIyeq+++iqmT5+OK1euYM6cOZg1axauX78OACgpKcHEiRPh5OSECxcuYMeOHTh69KheeFm1ahUiIiKwZMkSJCUl4X//+x98fHz03uPNN9/EjBkz8MMPP2Dy5MmYM2cO8vLy2rWfRNRK2vzWnEREBoSHhwsWFhaCvb293vL2228LglBzp+xnn31Wb5vhw4cLf/7znwVBEIQvvvhCcHJyEoqLi3XP79+/X5BKpYJKpRIEQRA8PDyEV155pcEaAAj/93//p3tcXFwsABAOHjzYav0kovbDMTdEJLqHH34Yq1at0lvn7Oys+//g4GC954KDg5GYmAgAuH79OgICAmBvb697fsSIEdBqtbh58yYkEgmysrIwbtw4gzUMGDBA9//29vaQy+XIzc1taZeISEQMN0QkOnt7+zqniVqLra1tk9pZWVnpPZZIJNBqtW1REhG1MY65ISKjFx8fX+dxnz59AAB9+vTBlStXUFJSonv+7NmzkEql8PX1hYODA7y9vXHs2LF2rZmIxMMjN0QkuoqKCqhUKr11lpaW6NKlCwBgx44dGDJkCEaOHImvv/4a58+fx5dffgkAmDNnDl5//XWEh4fjjTfewJ07d/D8889j3rx5cHV1BQC88cYbePbZZ+Hi4oJJkyahqKgIZ8+exfPPP9++HSWidsFwQ0SiO3ToENzd3fXW+fr64saNGwBqrmTaunUrnnvuObi7u+Obb75B3759AQB2dnY4fPgwli1bhqFDh8LOzg7Tp0/Hf/7zH91rhYeHo7y8HB9++CFeeukldOnSBU899VT7dZCI2pVEEARB7CKIiBoikUiwe/duhIaGil0KEZkIjrkhIiIis8JwQ0RERGaFY26IyKjxzDkRNReP3BAREZFZYbghIiIis8JwQ0RERGaF4YaIiIjMCsMNERERmRWGGyIiIjIrDDdERERkVhhuiIiIyKww3BAREZFZ+X9611boj/snDQAAAABJRU5ErkJggg==\n" + ] }, - "metadata": {} + "metadata": {}, + "output_type": "display_data" } + ], + "source": [ + "import torch\n", + "import os\n", + "import matplotlib.pyplot as plt\n", + "\n", + "# Define a custom callback-like function to handle actions at the end of each epoch\n", + "class CustomCallback:\n", + " def __init__(self, model, lora_name, lora_rank, text_gen):\n", + " self.model = model\n", + " self.lora_name = lora_name\n", + " self.lora_rank = lora_rank\n", + " self.text_gen = text_gen # text_gen function for evaluation\n", + "\n", + " def on_epoch_end(self, epoch):\n", + " # Save LoRA weights at the end of each epoch\n", + " model_name = f\"./{self.lora_name}_{self.lora_rank}_epoch{epoch+1}.lora.pt\"\n", + " self.model.save_pretrained(model_name, token=access_token) # Save model with LoRA weights locally\n", + "\n", + " # Evaluate the model using text generation\n", + " print(f\"Epoch {epoch + 1} finished. Running evaluation:\")\n", + " self.text_gen(\"Write a title\")\n", + " self.text_gen(\"Write a poem\")\n", + "\n", + "# Assuming train is your DataLoader and gemma_lm is your model\n", + "callback = CustomCallback(gemma_lm, lora_name, lora_rank, text_gen)\n", + "\n", + "# Training loop with callback-like behavior\n", + "losses = []\n", + "for epoch in range(train_epoch):\n", + " epoch_loss = 0\n", + " for batch in train: # Assuming `train` is a DataLoader or similar iterable\n", + " optimizer.zero_grad()\n", + "\n", + " inputs = tokenizer(batch, return_tensors=\"pt\", max_length=token_limit, truncation=True, padding=True)\n", + " labels = inputs[\"input_ids\"]\n", + " outputs = gemma_lm(**inputs, labels=labels)\n", + " loss = outputs.loss\n", + "\n", + " loss.backward()\n", + " optimizer.step()\n", + "\n", + " epoch_loss += loss.item()\n", + "\n", + " losses.append(epoch_loss / len(train)) # Store average loss per epoch\n", + "\n", + " # Run custom callback at the end of each epoch\n", + " callback.on_epoch_end(epoch)\n", + "\n", + "# Plot training loss over epochs\n", + "plt.plot(losses)\n", + "plt.xlabel(\"Epoch\")\n", + "plt.ylabel(\"Loss\")\n", + "plt.title(\"Training Loss Over Epochs\")\n", + "plt.show()" ] }, { @@ -1328,6 +1171,11 @@ }, { "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "kVe4vjgCngsd" + }, + "outputs": [], "source": [ "# Example Code for Load LoRA\n", "\n", @@ -1339,12 +1187,7 @@ "\n", "# # Load the LoRA adapter into the model using PeftModel\n", "# gemma_lm = PeftModel.from_pretrained(gemma_lm, lora_weights_path)" - ], - "metadata": { - "id": "kVe4vjgCngsd" - }, - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -1359,6 +1202,24 @@ }, { "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "K2JUE2IilwNi" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Gemma output:\n", + "user\n", + "Write a title\n", + "model\n", + "Capitรฃes da Areia\n" + ] + } + ], "source": [ "import torch\n", "\n", @@ -1389,28 +1250,6 @@ "\n", "# Generate text 5 times using the top_k sampling strategy\n", "text_gen_with_top_k(\"Write a title\", token_limit=100, top_k=50)\n" - ], - "metadata": { - "id": "K2JUE2IilwNi", - "outputId": "1ffdd099-1e0f-4a66-a7c6-81b43cc643e3", - "colab": { - "base_uri": "https://localhost:8080/" - } - }, - "execution_count": null, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "\n", - "Gemma output:\n", - "user\n", - "Write a title\n", - "model\n", - "Capitรฃes da Areia\n" - ] - } ] }, { @@ -1426,16 +1265,12 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "qC-MLxYWM1HU", - "outputId": "f26faf29-ce26-4a5c-e6ff-f2125148249b", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "qC-MLxYWM1HU" }, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "\n", "Gemma output:\n", @@ -1500,15 +1335,10 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "4TcvzBH995FE", - "outputId": "26a4be25-9a51-41f9-ad28-ef1708bf440f", - "colab": { - "base_uri": "https://localhost:8080/" - } + "id": "4TcvzBH995FE" }, "outputs": [ { - "output_type": "execute_result", "data": { "text/plain": [ "('./my_gemma2_lt_pt/tokenizer_config.json',\n", @@ -1518,8 +1348,9 @@ " './my_gemma2_lt_pt/tokenizer.json')" ] }, + "execution_count": 18, "metadata": {}, - "execution_count": 18 + "output_type": "execute_result" } ], "source": [ @@ -1546,4551 +1377,73 @@ }, { "cell_type": "code", - "source": [ - "# Upload the model to Hugging Face Hub\n", - "my_model_name = \"my_gemma2_pt\"\n", - "writeToken = userdata.get(\"HF_WRITE_TOKEN\")\n", - "hf_repo_id = f\"{my_hf_username}/{my_model_name}\" # Correct format\n", - "gemma_lm.push_to_hub(hf_repo_id, token=writeToken)" - ], + "execution_count": null, "metadata": { - "id": "AK31-LuXpwen", - "outputId": "e96865e3-4e8c-4acf-e256-c5ec31b6f931", - "colab": { - "base_uri": "https://localhost:8080/", - "height": 136, - "referenced_widgets": [ - "84cd1a70535f4272a0206041373ff281", - "325cf823a7244a4b8e25462ae4912ea3", - "325f38ff3fef43cbbd248ff3ff714c43", - "de4fae4a6e3e420d86414a572846a9e0", - "4d729ba11c9442528af79d6e377a292c", - "49a42d4bcb8b4684a69656ee8b0449df", - "50b06db300ce432c9235071f48a18432", - "a31850a948e344848d43caae277f3200", - "fe5291b55e7147b682b671dad99d16e6", - "b485e94c1509478da6dd7a04c41e055d", - "98e4874bfa904312a4a9b00ca2ea8577", - "a163fe0a8b594568bf102caf54856484", - "dd53c0e07382441fbad818b0c3852006", - "3c316ae120e445c8b4bbc8260cf14ab4", - "9387a006080845ce9ef9aeb41fbd6c6e", - "c49cd450c3d24511a1efe272820c5f0a", - "99a5ad5c04b4456ba65118367cd2ea5a", - "d74745e493704920b56855e0cb73a39e", - "bd53b02c9808436ead8b213886e76bed", - "3f1bd9e04dc64bca88b99998339df788", - "d082ae773106445aa6ec56167b2a5cb9", - "030123f151674251ac0f0be0f8f713dc" - ] - } + "id": "AK31-LuXpwen" }, - "execution_count": null, "outputs": [ { - "output_type": "display_data", "data": { - "text/plain": [ - "README.md: 0%| | 0.00/5.17k [00:00 Date: Tue, 22 Oct 2024 09:40:45 +0900 Subject: [PATCH 17/43] add copyright --- ...ne_tuning_Gemma_Transformers_Edition.ipynb | 72 +++++++++++++++---- 1 file changed, 57 insertions(+), 15 deletions(-) diff --git a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb index 02caf27..535ee86 100644 --- a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb +++ b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb @@ -1,5 +1,14 @@ { "cells": [ + { + "cell_type": "markdown", + "source": [ + "##### Copyright 2024 Google LLC." + ], + "metadata": { + "id": "mQGu7iBsqnU_" + } + }, { "cell_type": "code", "execution_count": null, @@ -95,7 +104,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "TT7GexJnZZCj" + "id": "TT7GexJnZZCj", + "outputId": "d0e9366a-055e-4839-c395-7794241640ea" }, "outputs": [ { @@ -137,7 +147,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "WNn86PiiXTNf" + "id": "WNn86PiiXTNf", + "outputId": "7dbda7f4-1d25-42b9-ecbf-95d7a25115a3" }, "outputs": [ { @@ -212,7 +223,23 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "ywcDWVhAXb_9" + "id": "ywcDWVhAXb_9", + "outputId": "489e5ed0-fd38-4870-c6c8-d5ddc9894e9a", + "colab": { + "referenced_widgets": [ + "5dcc5fb3ad9e41eeb877fdd04f09ba7d", + "f4188acdf47f4cce86c02cc0d2bf46c7", + "eee1dd11b43e435cb226ee6df594291d", + "9dc263e2dbfa4d9d9a56d62a69a57a8f", + "ecb1c6f7effa46b296e3574cddcb88de", + "64324fda5c814874aaa2e97fa78ab3f0", + "43bccde2f2804b089e4e40081058855f", + "488f3d23e6ee4370b0e3750c4610acc1", + "9ac1990111d64f919644a292b4ea2d36", + "b91c257be15341179ce8a262b3962ef0", + "379696127cc841a08cb6ca5000180032" + ] + } }, "outputs": [ { @@ -496,7 +523,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "xR1riO7NURFF" + "id": "xR1riO7NURFF", + "outputId": "8cba66a8-c6f4-4dab-eb4b-1aa95ca95817" }, "outputs": [ { @@ -589,7 +617,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "ZiS-KU9osh_N" + "id": "ZiS-KU9osh_N", + "outputId": "c03e319a-4526-456c-aa6f-0063c75afad8" }, "outputs": [ { @@ -677,7 +706,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "Vh9s8m_PgoAH" + "id": "Vh9s8m_PgoAH", + "outputId": "063b01cf-e6dc-40a8-dc66-59a143e8e75d" }, "outputs": [ { @@ -828,7 +858,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "YQiQxLFKfyzx" + "id": "YQiQxLFKfyzx", + "outputId": "ac7eded2-26d8-41df-da9a-27da04e168c9" }, "outputs": [ { @@ -989,7 +1020,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "YKpmDIfXh1Kx" + "id": "YKpmDIfXh1Kx", + "outputId": "da765e33-ba8d-46a9-cfbd-a629c5cb796e" }, "outputs": [ { @@ -1204,7 +1236,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "K2JUE2IilwNi" + "id": "K2JUE2IilwNi", + "outputId": "7f47740f-088f-4764-ac06-98663286b652" }, "outputs": [ { @@ -1265,7 +1298,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "qC-MLxYWM1HU" + "id": "qC-MLxYWM1HU", + "outputId": "4d060372-6b67-4982-ed55-d5bd16e1ce60" }, "outputs": [ { @@ -1335,7 +1369,8 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "4TcvzBH995FE" + "id": "4TcvzBH995FE", + "outputId": "e0a42750-2107-47ce-b2b6-244eedde7535" }, "outputs": [ { @@ -1379,7 +1414,14 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "AK31-LuXpwen" + "id": "AK31-LuXpwen", + "outputId": "a562d1fd-2b75-493b-e31e-99eb42c3e128", + "colab": { + "referenced_widgets": [ + "84cd1a70535f4272a0206041373ff281", + "a163fe0a8b594568bf102caf54856484" + ] + } }, "outputs": [ { @@ -1436,8 +1478,8 @@ "metadata": { "accelerator": "GPU", "colab": { - "name": "Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb", - "toc_visible": true + "toc_visible": true, + "provenance": [] }, "kernelspec": { "display_name": "Python 3", @@ -1446,4 +1488,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} +} \ No newline at end of file From 64154659748ac4ba63e25407c2017277ea94a5ac Mon Sep 17 00:00:00 2001 From: Juyeong Ji Date: Tue, 22 Oct 2024 09:42:11 +0900 Subject: [PATCH 18/43] nbfmt --- ...ne_tuning_Gemma_Transformers_Edition.ipynb | 71 +++++-------------- 1 file changed, 19 insertions(+), 52 deletions(-) diff --git a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb index 535ee86..2318a59 100644 --- a/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb +++ b/Workshops/Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb @@ -2,12 +2,12 @@ "cells": [ { "cell_type": "markdown", - "source": [ - "##### Copyright 2024 Google LLC." - ], "metadata": { "id": "mQGu7iBsqnU_" - } + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] }, { "cell_type": "code", @@ -104,8 +104,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "TT7GexJnZZCj", - "outputId": "d0e9366a-055e-4839-c395-7794241640ea" + "id": "TT7GexJnZZCj" }, "outputs": [ { @@ -147,8 +146,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "WNn86PiiXTNf", - "outputId": "7dbda7f4-1d25-42b9-ecbf-95d7a25115a3" + "id": "WNn86PiiXTNf" }, "outputs": [ { @@ -223,23 +221,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "ywcDWVhAXb_9", - "outputId": "489e5ed0-fd38-4870-c6c8-d5ddc9894e9a", - "colab": { - "referenced_widgets": [ - "5dcc5fb3ad9e41eeb877fdd04f09ba7d", - "f4188acdf47f4cce86c02cc0d2bf46c7", - "eee1dd11b43e435cb226ee6df594291d", - "9dc263e2dbfa4d9d9a56d62a69a57a8f", - "ecb1c6f7effa46b296e3574cddcb88de", - "64324fda5c814874aaa2e97fa78ab3f0", - "43bccde2f2804b089e4e40081058855f", - "488f3d23e6ee4370b0e3750c4610acc1", - "9ac1990111d64f919644a292b4ea2d36", - "b91c257be15341179ce8a262b3962ef0", - "379696127cc841a08cb6ca5000180032" - ] - } + "id": "ywcDWVhAXb_9" }, "outputs": [ { @@ -523,8 +505,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "xR1riO7NURFF", - "outputId": "8cba66a8-c6f4-4dab-eb4b-1aa95ca95817" + "id": "xR1riO7NURFF" }, "outputs": [ { @@ -617,8 +598,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "ZiS-KU9osh_N", - "outputId": "c03e319a-4526-456c-aa6f-0063c75afad8" + "id": "ZiS-KU9osh_N" }, "outputs": [ { @@ -706,8 +686,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "Vh9s8m_PgoAH", - "outputId": "063b01cf-e6dc-40a8-dc66-59a143e8e75d" + "id": "Vh9s8m_PgoAH" }, "outputs": [ { @@ -858,8 +837,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "YQiQxLFKfyzx", - "outputId": "ac7eded2-26d8-41df-da9a-27da04e168c9" + "id": "YQiQxLFKfyzx" }, "outputs": [ { @@ -1020,8 +998,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "YKpmDIfXh1Kx", - "outputId": "da765e33-ba8d-46a9-cfbd-a629c5cb796e" + "id": "YKpmDIfXh1Kx" }, "outputs": [ { @@ -1236,8 +1213,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "K2JUE2IilwNi", - "outputId": "7f47740f-088f-4764-ac06-98663286b652" + "id": "K2JUE2IilwNi" }, "outputs": [ { @@ -1298,8 +1274,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "qC-MLxYWM1HU", - "outputId": "4d060372-6b67-4982-ed55-d5bd16e1ce60" + "id": "qC-MLxYWM1HU" }, "outputs": [ { @@ -1369,8 +1344,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "4TcvzBH995FE", - "outputId": "e0a42750-2107-47ce-b2b6-244eedde7535" + "id": "4TcvzBH995FE" }, "outputs": [ { @@ -1414,14 +1388,7 @@ "cell_type": "code", "execution_count": null, "metadata": { - "id": "AK31-LuXpwen", - "outputId": "a562d1fd-2b75-493b-e31e-99eb42c3e128", - "colab": { - "referenced_widgets": [ - "84cd1a70535f4272a0206041373ff281", - "a163fe0a8b594568bf102caf54856484" - ] - } + "id": "AK31-LuXpwen" }, "outputs": [ { @@ -1478,8 +1445,8 @@ "metadata": { "accelerator": "GPU", "colab": { - "toc_visible": true, - "provenance": [] + "name": "Workshop_How_to_Fine_tuning_Gemma_Transformers_Edition.ipynb", + "toc_visible": true }, "kernelspec": { "display_name": "Python 3", @@ -1488,4 +1455,4 @@ }, "nbformat": 4, "nbformat_minor": 0 -} \ No newline at end of file +} From 91d3f21ab6080b36b820e4fd6a73e8d9de0a2331 Mon Sep 17 00:00:00 2001 From: Gusthema <435453+gustheman@users.noreply.github.com> Date: Tue, 22 Oct 2024 17:45:19 +0100 Subject: [PATCH 19/43] fixing fine tuning workshop link in the main README file fixing a broken link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d8cfe18..befe491 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo ## Workshops and technical talks | Notebook | Description | | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -| [Workshop_How_to_Fine_tuning_Gemma.ipynb](Workshops/Workshop_How_to_Fine_tuning_Gemma.ipynb/Keras_Gemma_2_Quickstart.ipynb) | Recommended finetuning notebook for getting started | +| [Workshop_How_to_Fine_tuning_Gemma.ipynb](Workshops/Workshop_How_to_Fine_tuning_Gemma.ipynb) | Recommended finetuning notebook for getting started | | [Self_extend_Gemma.ipynb](Gemma/Self_extend_Gemma.ipynb) | Self-extend context window for Gemma in the I/O 2024 [Keras talk](https://www.youtube.com/watch?v=TV7qCk1dBWA) | | [Gemma_control_vectors.ipynb](Gemma/Gemma_control_vectors.ipynb) | Implement [control vectors](https://arxiv.org/abs/2310.01405) with Gemma in the I/O 2024 [Keras talk](https://www.youtube.com/watch?v=TV7qCk1dBWA) | From c2c28e4f7ed04ea61fa7d292c4e9bacdff4a571a Mon Sep 17 00:00:00 2001 From: "nidhin.devan" Date: Fri, 25 Oct 2024 13:57:12 +0530 Subject: [PATCH 20/43] RouteLLM with Gemma and Gemini notebook added --- ...Using_Gemini_and_Gemma_with_RouteLLM.ipynb | 426 ++++++++++++++++++ README.md | 1 + WISHLIST.md | 1 - 3 files changed, 427 insertions(+), 1 deletion(-) create mode 100644 Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb diff --git a/Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb b/Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb new file mode 100644 index 0000000..fb02380 --- /dev/null +++ b/Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb @@ -0,0 +1,426 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "d4ES2HG8TMYT" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "Sagji9qaTPlK" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QmJNavKnTOei" + }, + "source": [ + "# Getting Started with Gemma 2, Gemini, and RouteLLM\n", + "\n", + "[Gemma](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n", + "Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.\n", + "\n", + "[Gemini](https://blog.google/technology/ai/google-gemini-ai/) is a large language model developed by Google AI. It is a multimodal model, meaning it can process and generate text, code, images, and audio. Gemini is considered one of the most advanced language models available, and it has been shown to outperform other models on various tasks, such as question answering, summarization, and translation.\n", + "\n", + "[RouteLLM](https://github.com/lm-sys/RouteLLM) is a framework that helps you optimize your LLM usage by routing queries to the most appropriate model based on their complexity. It can significantly reduce costs without sacrificing performance. You can easily integrate it into your existing applications and experiment with different routing strategies.\n", + "\n", + "In this notebook, you will learn how to route between the Gemini and the Gemma 2 models using **RouteLLM** in a Google Colab environment. You'll install the necessary packages, set up the models, and run a sample prompt.\n", + "\n", + "\n", + "\n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IAvbefp1fWUk" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you must have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n", + "\n", + "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **T4 GPU**.\n", + "\n", + "### Setup Hugging Face and Gemini\n", + "\n", + "**Before you dive into the tutorial, let's get you set up with Hugging Face and Gemma:**\n", + "\n", + "#### Hugging Face setup\n", + "\n", + "1. **Hugging Face Account:** If you don't already have one, you can create a free one by clicking [here](https://huggingface.co/join).\n", + "\n", + "2. **Hugging Face Token:** Generate a Hugging Face access (preferably `write` permission) token by clicking [here](https://huggingface.co/settings/tokens). You'll need this token later in the tutorial.\n", + "\n", + "#### Gemini setup\n", + "\n", + "1. **Gemini Token:** To use the Gemini API, you need an API key. You can create a key with a few clicks in [Google AI Studio](https://aistudio.google.com/app/apikey).\n", + "\n", + "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QA1CAiSGlOHf" + }, + "source": [ + "### Configure your HF token and Gemini token\n", + "\n", + "Add your Hugging Face token and Gemini token to the Colab Secrets manager to securely store it.\n", + "\n", + "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + "2. Create a new secret with the name `HF_TOKEN`.\n", + "3. Copy/paste your HF token key into the Value input box of `HF_TOKEN`.\n", + "4. Toggle the button on the left to allow notebook access to the secret.\n", + "5. Create a new secret with the name `GOOGLE_API_KEY`.\n", + "6. Copy/paste your Gemini token key into the Value input box of `GOOGLE_API_KEY`.\n", + "7. Toggle the button on the left to allow notebook access to the secret." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "pFNuO5RsmWfn" + }, + "outputs": [], + "source": [ + "import os\n", + "from google.colab import userdata\n", + "\n", + "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n", + "# vars as appropriate for your system.\n", + "os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")\n", + "os.environ[\"GEMINI_API_KEY\"] = userdata.get(\"GOOGLE_API_KEY\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gly2y32DwXH_" + }, + "source": [ + "Currently, RouteLLM checks for `OPENAI_API_KEY` before starting. You can set the `OPENAI_API_KEY` to a dummy value as a temporary workaround. The collaborators for RouteLLM are currently working on to fix this [issue](https://github.com/lm-sys/RouteLLM/issues/19)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "eZCUFnSlzTup" + }, + "outputs": [], + "source": [ + "os.environ[\"OPENAI_API_KEY\"] = \"dummy\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TjqCf1R6pJtY" + }, + "source": [ + "### Install dependencies\n", + "\n", + "You'll need to install a few Python packages and dependencies for RouteLLM and Gemini API.\n", + "\n", + "Run the following cell to install or upgrade it:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BsQey4_-SunS" + }, + "outputs": [], + "source": [ + "# Install RouteLLM package.\n", + "! pip install \"routellm[serve,eval]\"\n", + "\n", + "# Install the Python SDK for the Gemini API, `google-generativeai`.\n", + "! pip install -q google-generativeai" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dp2HxAxzqvSX" + }, + "source": [ + "### Setup Gemma2 with Ollama\n", + "\n", + "To use a local model like Gemma 2 with RouteLLM you need [Ollama](https://ollama.com/).\n", + "\n", + "Ollama is an open-source framework for building and running large language models (LLMs). It's designed to be flexible and customizable, allowing developers to train and deploy their models or fine-tune existing ones. Ollama is a popular choice for those who want to experiment with LLMs or build custom models without relying on proprietary platforms." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "V9xBkvRcyBvj" + }, + "source": [ + "#### Install Ollama" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "BRFue2F5YtS0" + }, + "outputs": [], + "source": [ + "!curl https://ollama.ai/install.sh | sh" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AID-4tDpqOTE" + }, + "source": [ + "#### Start Ollama as a background subprocess" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ZR3TgObHdvLz" + }, + "outputs": [], + "source": [ + "import subprocess\n", + "import time\n", + "\n", + "process = subprocess.Popen(\"ollama serve\", shell=True)\n", + "time.sleep(5)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gzp2EVqrywFG" + }, + "source": [ + "#### Run Gemma 2 in Ollama as a background subprocess" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "KBF9QIady2nN" + }, + "outputs": [], + "source": [ + "process = subprocess.Popen(\"ollama run gemma2\", shell=True)\n", + "time.sleep(5)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2NeDPsUXyKB_" + }, + "source": [ + "#### Check if Ollama is running\n", + "\n", + "Run the following command to see whether Ollama is up and running. Continue to the next cell when the output of the following command says \"**Ollama is running**\"." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "HdkLSYwSO3tp" + }, + "outputs": [], + "source": [ + "!curl localhost:11434" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6WQqh6kQVaG_" + }, + "source": [ + "## How RouteLLM works?\n", + "\n", + "\n", + "RouteLLM is a systematic framework for preference-data-based LLM routing. There are two models in RouteLLM's routing setup: a weaker but less costly model and a stronger but more costly model.\n", + "\n", + "When a prompt is sent to RouteLLM, an underlying routing approach routes the prompt to either the strong or weak model for inference. RouteLLM offers four different routing approaches:\n", + "\n", + "1. **Similarity-weighted (SW) ranking**: A similarity-weighted (SW) ranking router that performs a \"weighted Elo calculation\" based on similarity.\n", + "\n", + "2. **Matrix factorization**: A matrix factorization model that learns a scoring function for how well a model can answer a prompt.\n", + "\n", + "3. **BERT classifier**: A BERT classifier that predicts which model can provide a better response.\n", + "\n", + "4. **Causal LLM classifier**: A causal LLM classifier that predicts which model can provide a better response.\n", + "\n", + "Read more about these routing methods and their performance stats from [lmsys's blog](https://lmsys.org/blog/2024-07-01-routellm/).\n", + "\n", + "You can also refer to the research paper, [RouteLLM: Learning to Route LLMs with Preference\n", + "Data](https://arxiv.org/abs/2406.18665) published by the creators." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tso2qSwazCGE" + }, + "source": [ + "## Routing between Gemini and Gemma 2 using the `routellm` library\n", + "\n", + "\n", + "You can use **RouteLLM** in your Python code using the `routellm` library.\n", + "\n", + "You will initialize the `Controller` from the `routellm` library by specifying the routers, strong_model, and weak_model. Here's what each argument of `Controller` does:\n", + "\n", + "- `routers`: Name of the router. For this notebook, you will choose `bert` since it is easier to run `bert` in a colab environment.\n", + "- `strong_model`: A stronger, more expensive model. For this example you will use **Gemini Pro**.\n", + "- `weak_model`: A weaker but cheaper model. You can put your local **Gemma 2** model here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "tUzFyAAnS16k" + }, + "outputs": [], + "source": [ + "from routellm.controller import Controller\n", + "\n", + "client = Controller(\n", + " routers=[\"bert\"], # Use `bert` router\n", + " strong_model=\"gemini/gemini-pro\",\n", + " weak_model=\"ollama_chat/gemma2\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "E3UAE4yloBAj" + }, + "source": [ + "**Threshold calibration** is essential for balancing cost and quality in LLM routing. The optimal threshold depends on your router and incoming queries. Calibrate your queries using a sample and your desired routing percentage. RouteLLM supports default calibration based on the public [Chatbot Arena dataset](https://huggingface.co/datasets/lmsys/lmsys-arena-human-preference-55k).\n", + "\n", + "For this example, you will calibrate the threshold for `bert` such that 30% of the calls are routed to the stronger model. To calculate this threshold, you can run the `routellm.calibrate_threshold` command and provide the following values.\n", + "\n", + "`--routers`: bert\n", + "\n", + "`--strong-model-pct` : 0.3" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xnj66rs5MJtz" + }, + "outputs": [], + "source": [ + "!python -m routellm.calibrate_threshold --task calibrate --routers bert --strong-model-pct 0.3 --config config.example.yaml" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sWEaTvZ5tEAU" + }, + "source": [ + "The `client.chat.completions.create` function lets you prompt your RouteLLM setup. You can specify the threshold value you obtained in the previous step in the `model` argument of the `client.chat.completions.create` function. If the router is `bert` and the threshold is **0.46514**, pass `router-bert-0.46514` to the `model` argument. You can pass your chat messages to the `messages` argument." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zWUwIOCVlAFG" + }, + "outputs": [], + "source": [ + "response = client.chat.completions.create(\n", + " # This tells RouteLLM to use the bert router with a cost threshold of 0.46514\n", + " model=\"router-bert-0.46514\",\n", + " messages=[\n", + " {\"role\": \"user\", \"content\": \"Hello!\"}\n", + " ]\n", + ")\n", + "\n", + "print(\"Selected model: {model}\\n\".format(model=response.model))\n", + "print(\"Response: {model_response}\".format(\n", + " model_response=response.choices[0].message.content))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zhSHtO_sZyl_" + }, + "source": [ + "To learn more about the Python usage of **RouteLLM**, visit [RouteLLM's GitHub page](https://github.com/lm-sys/RouteLLM)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xxKuGtofZ08F" + }, + "source": [ + "## Conclusion\n", + "\n", + "Congratulations! You've successfully routed between the Gemini and the Gemma 2 models using **RouteLLM** in a Google Colab environment." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Using_Gemini_and_Gemma_with_RouteLLM.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/README.md b/README.md index a963e8e..e91abc9 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Using_Gemma_with_Llamafile.ipynb](Gemma/Using_Gemma_with_Llamafile.ipynb) | Run Gemma models using [Llamafile](https://github.com/Mozilla-Ocho/llamafile/). | | [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | | [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | +| [Using_Gemini_and_Gemma_with_RouteLLM.ipynb](Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb) | Route Gemma and Gemini models using [RouteLLM](https://github.com/lm-sys/RouteLLM/). | | [Integrate_with_Mesop.ipynb](Gemma/Integrate_with_Mesop.ipynb) | Integrate Gemma with [Google Mesop](https://google.github.io/mesop/). | | [Integrate_with_OneTwo.ipynb](Gemma/Integrate_with_OneTwo.ipynb) | Integrate Gemma with [Google OneTwo](https://github.com/google-deepmind/onetwo). | | [Deploy_with_vLLM.ipynb](Gemma/Deploy_with_vLLM.ipynb) | Deploy a Gemma model using [vLLM](https://github.com/vllm-project/vllm). | diff --git a/WISHLIST.md b/WISHLIST.md index ea337d3..6c991ec 100644 --- a/WISHLIST.md +++ b/WISHLIST.md @@ -2,7 +2,6 @@ A wish list of cookbooks showcasing: * Inference * Integration with [Google GenKit](https://firebase.google.com/products/genkit) - * Gemma+Gemini with [routerLLM](https://github.com/lm-sys/RouteLLM) * [SGLang](https://github.com/sgl-project/sglang) integration * Fine-tuning From 153cef55dfcd66bdde723edf354c3db87cc041ae Mon Sep 17 00:00:00 2001 From: Wei Wei <5577741+windmaple@users.noreply.github.com> Date: Mon, 28 Oct 2024 14:55:26 +0800 Subject: [PATCH 21/43] Remove versions in requirements.txt for easier maintenance --- .../business-email-assistant/requirements.txt | 128 +++++----- .../requirements.txt.bak | 64 +++++ .../gemma-web-service/requirements.txt | 146 +++++------ .../gemma-web-service/requirements.txt.bak | 73 ++++++ Gemma/spoken-language-tasks/requirements.txt | 226 +++++++++--------- .../requirements.txt.bak | 113 +++++++++ 6 files changed, 500 insertions(+), 250 deletions(-) create mode 100644 Gemma/business-email-assistant/requirements.txt.bak create mode 100644 Gemma/personal-code-assistant/gemma-web-service/requirements.txt.bak create mode 100644 Gemma/spoken-language-tasks/requirements.txt.bak diff --git a/Gemma/business-email-assistant/requirements.txt b/Gemma/business-email-assistant/requirements.txt index 374b139..1212ee6 100644 --- a/Gemma/business-email-assistant/requirements.txt +++ b/Gemma/business-email-assistant/requirements.txt @@ -1,64 +1,64 @@ -absl-py==2.1.0 -astunparse==1.6.3 -blinker==1.8.2 -certifi==2024.8.30 -charset-normalizer==3.3.2 -click==8.1.7 -flask==3.0.3 -flatbuffers==24.3.25 -gast==0.6.0 -google-pasta==0.2.0 -grpcio==1.66.1 -h5py==3.11.0 -idna==3.10 -importlib-metadata==8.5.0 -itsdangerous==2.2.0 -jax==0.4.30 -jax-cuda12-pjrt==0.4.30 -jax-cuda12-plugin==0.4.30 -jaxlib==0.4.30 -jinja2==3.1.4 -kagglehub==0.3.0 -keras==3.5.0 -keras-nlp==0.14.4 -libclang==18.1.1 -Markdown==3.7 -markdown-it-py==3.0.0 -MarkupSafe==2.1.5 -mdurl==0.1.2 -ml-dtypes==0.4.1 -namex==0.0.8 -numpy==1.26.4 -nvidia-cublas-cu12==12.6.1.4 -nvidia-cuda-cupti-cu12==12.6.68 -nvidia-cuda-nvcc-cu12==12.6.68 -nvidia-cuda-runtime-cu12==12.6.68 -nvidia-cudnn-cu12==9.3.0.75 -nvidia-cufft-cu12==11.2.6.59 -nvidia-cusolver-cu12==11.6.4.69 -nvidia-cusparse-cu12==12.5.3.3 -nvidia-nccl-cu12==2.22.3 -nvidia-nvjitlink-cu12==12.6.68 -opt-einsum==3.3.0 -optree==0.12.1 -packaging==24.1 -protobuf==4.25.4 -pygments==2.18.0 -python-dotenv==1.0.1 -regex==2024.9.11 -requests==2.32.3 -rich==13.8.1 -scipy==1.13.1 -six==1.16.0 -tensorboard==2.17.1 -tensorboard-data-server==0.7.2 -tensorflow==2.17.0 -tensorflow-io-gcs-filesystem==0.37.1 -tensorflow-text==2.17.0 -termcolor==2.4.0 -tqdm==4.66.5 -typing-extensions==4.12.2 -urllib3==2.2.3 -werkzeug==3.0.4 -wrapt==1.16.0 -zipp==3.20.2 +absl-pyg +astunparseg +blinkerg +certifig +charset-normalizerg +clickg +flaskg +flatbuffersg +gastg +google-pastag +grpciog +h5pyg +idnag +importlib-metadatag +itsdangerousg +jaxg +jax-cuda12-pjrtg +jax-cuda12-pluging +jaxlibg +jinja2g +kagglehubg +kerasg +keras-nlpg +libclangg +Markdowng +markdown-it-pyg +MarkupSafeg +mdurlg +ml-dtypesg +namexg +numpyg +nvidia-cublas-cu12g +nvidia-cuda-cupti-cu12g +nvidia-cuda-nvcc-cu12g +nvidia-cuda-runtime-cu12g +nvidia-cudnn-cu12g +nvidia-cufft-cu12g +nvidia-cusolver-cu12g +nvidia-cusparse-cu12g +nvidia-nccl-cu12g +nvidia-nvjitlink-cu12g +opt-einsumg +optreeg +packagingg +protobufg +pygmentsg +python-dotenvg +regexg +requestsg +richg +scipyg +sixg +tensorboardg +tensorboard-data-serverg +tensorflowg +tensorflow-io-gcs-filesystemg +tensorflow-textg +termcolorg +tqdmg +typing-extensionsg +urllib3g +werkzeugg +wraptg +zippg diff --git a/Gemma/business-email-assistant/requirements.txt.bak b/Gemma/business-email-assistant/requirements.txt.bak new file mode 100644 index 0000000..374b139 --- /dev/null +++ b/Gemma/business-email-assistant/requirements.txt.bak @@ -0,0 +1,64 @@ +absl-py==2.1.0 +astunparse==1.6.3 +blinker==1.8.2 +certifi==2024.8.30 +charset-normalizer==3.3.2 +click==8.1.7 +flask==3.0.3 +flatbuffers==24.3.25 +gast==0.6.0 +google-pasta==0.2.0 +grpcio==1.66.1 +h5py==3.11.0 +idna==3.10 +importlib-metadata==8.5.0 +itsdangerous==2.2.0 +jax==0.4.30 +jax-cuda12-pjrt==0.4.30 +jax-cuda12-plugin==0.4.30 +jaxlib==0.4.30 +jinja2==3.1.4 +kagglehub==0.3.0 +keras==3.5.0 +keras-nlp==0.14.4 +libclang==18.1.1 +Markdown==3.7 +markdown-it-py==3.0.0 +MarkupSafe==2.1.5 +mdurl==0.1.2 +ml-dtypes==0.4.1 +namex==0.0.8 +numpy==1.26.4 +nvidia-cublas-cu12==12.6.1.4 +nvidia-cuda-cupti-cu12==12.6.68 +nvidia-cuda-nvcc-cu12==12.6.68 +nvidia-cuda-runtime-cu12==12.6.68 +nvidia-cudnn-cu12==9.3.0.75 +nvidia-cufft-cu12==11.2.6.59 +nvidia-cusolver-cu12==11.6.4.69 +nvidia-cusparse-cu12==12.5.3.3 +nvidia-nccl-cu12==2.22.3 +nvidia-nvjitlink-cu12==12.6.68 +opt-einsum==3.3.0 +optree==0.12.1 +packaging==24.1 +protobuf==4.25.4 +pygments==2.18.0 +python-dotenv==1.0.1 +regex==2024.9.11 +requests==2.32.3 +rich==13.8.1 +scipy==1.13.1 +six==1.16.0 +tensorboard==2.17.1 +tensorboard-data-server==0.7.2 +tensorflow==2.17.0 +tensorflow-io-gcs-filesystem==0.37.1 +tensorflow-text==2.17.0 +termcolor==2.4.0 +tqdm==4.66.5 +typing-extensions==4.12.2 +urllib3==2.2.3 +werkzeug==3.0.4 +wrapt==1.16.0 +zipp==3.20.2 diff --git a/Gemma/personal-code-assistant/gemma-web-service/requirements.txt b/Gemma/personal-code-assistant/gemma-web-service/requirements.txt index 21b4d68..0d09074 100644 --- a/Gemma/personal-code-assistant/gemma-web-service/requirements.txt +++ b/Gemma/personal-code-assistant/gemma-web-service/requirements.txt @@ -1,73 +1,73 @@ -absl-py==2.1.0 -annotated-types==0.7.0 -anyio==4.4.0 -astunparse==1.6.3 -certifi==2024.8.30 -charset-normalizer==3.3.2 -click==8.1.7 -exceptiongroup==1.2.2 -fastapi==0.109.0 -flatbuffers==24.3.25 -gast==0.6.0 -google-pasta==0.2.0 -grpcio==1.66.1 -h11==0.14.0 -h5py==3.11.0 -httptools==0.6.1 -idna==3.8 -jax==0.4.31 -jax-cuda12-pjrt==0.4.31 -jax-cuda12-plugin==0.4.31 -jaxlib==0.4.31 -kagglehub==0.3.0 -keras==3.5.0 -keras-nlp==0.14.4 -libclang==18.1.1 -Markdown==3.7 -markdown-it-py==3.0.0 -MarkupSafe==2.1.5 -mdurl==0.1.2 -ml-dtypes==0.4.0 -namex==0.0.8 -numpy==1.26.4 -nvidia-cublas-cu12==12.6.1.4 -nvidia-cuda-cupti-cu12==12.6.68 -nvidia-cuda-nvcc-cu12==12.6.68 -nvidia-cuda-runtime-cu12==12.6.68 -nvidia-cudnn-cu12==9.4.0.58 -nvidia-cufft-cu12==11.2.6.59 -nvidia-cusolver-cu12==11.6.4.69 -nvidia-cusparse-cu12==12.5.3.3 -nvidia-nccl-cu12==2.23.4 -nvidia-nvjitlink-cu12==12.6.68 -opt-einsum==3.3.0 -optree==0.12.1 -packaging==24.1 -protobuf==4.25.4 -pydantic==2.9.1 -pydantic_core==2.23.3 -Pygments==2.18.0 -python-dotenv==1.0.1 -PyYAML==6.0.2 -regex==2024.9.11 -requests==2.32.3 -rich==13.8.1 -scipy==1.14.1 -six==1.16.0 -sniffio==1.3.1 -starlette==0.35.1 -tensorboard==2.17.1 -tensorboard-data-server==0.7.2 -tensorflow==2.17.0 -tensorflow-io-gcs-filesystem==0.37.1 -tensorflow-text==2.17.0 -termcolor==2.4.0 -tqdm==4.66.5 -typing_extensions==4.12.2 -urllib3==2.2.3 -uvicorn==0.30.6 -uvloop==0.20.0 -watchfiles==0.24.0 -websockets==13.0.1 -Werkzeug==3.0.4 -wrapt==1.16.0 +absl-pyg +annotated-typesg +anyiog +astunparseg +certifig +charset-normalizerg +clickg +exceptiongroupg +fastapig +flatbuffersg +gastg +google-pastag +grpciog +h11g +h5pyg +httptoolsg +idnag +jaxg +jax-cuda12-pjrtg +jax-cuda12-pluging +jaxlibg +kagglehubg +kerasg +keras-nlpg +libclangg +Markdowng +markdown-it-pyg +MarkupSafeg +mdurlg +ml-dtypesg +namexg +numpyg +nvidia-cublas-cu12g +nvidia-cuda-cupti-cu12g +nvidia-cuda-nvcc-cu12g +nvidia-cuda-runtime-cu12g +nvidia-cudnn-cu12g +nvidia-cufft-cu12g +nvidia-cusolver-cu12g +nvidia-cusparse-cu12g +nvidia-nccl-cu12g +nvidia-nvjitlink-cu12g +opt-einsumg +optreeg +packagingg +protobufg +pydanticg +pydantic_coreg +Pygmentsg +python-dotenvg +PyYAMLg +regexg +requestsg +richg +scipyg +sixg +sniffiog +starletteg +tensorboardg +tensorboard-data-serverg +tensorflowg +tensorflow-io-gcs-filesystemg +tensorflow-textg +termcolorg +tqdmg +typing_extensionsg +urllib3g +uvicorng +uvloopg +watchfilesg +websocketsg +Werkzeugg +wraptg diff --git a/Gemma/personal-code-assistant/gemma-web-service/requirements.txt.bak b/Gemma/personal-code-assistant/gemma-web-service/requirements.txt.bak new file mode 100644 index 0000000..21b4d68 --- /dev/null +++ b/Gemma/personal-code-assistant/gemma-web-service/requirements.txt.bak @@ -0,0 +1,73 @@ +absl-py==2.1.0 +annotated-types==0.7.0 +anyio==4.4.0 +astunparse==1.6.3 +certifi==2024.8.30 +charset-normalizer==3.3.2 +click==8.1.7 +exceptiongroup==1.2.2 +fastapi==0.109.0 +flatbuffers==24.3.25 +gast==0.6.0 +google-pasta==0.2.0 +grpcio==1.66.1 +h11==0.14.0 +h5py==3.11.0 +httptools==0.6.1 +idna==3.8 +jax==0.4.31 +jax-cuda12-pjrt==0.4.31 +jax-cuda12-plugin==0.4.31 +jaxlib==0.4.31 +kagglehub==0.3.0 +keras==3.5.0 +keras-nlp==0.14.4 +libclang==18.1.1 +Markdown==3.7 +markdown-it-py==3.0.0 +MarkupSafe==2.1.5 +mdurl==0.1.2 +ml-dtypes==0.4.0 +namex==0.0.8 +numpy==1.26.4 +nvidia-cublas-cu12==12.6.1.4 +nvidia-cuda-cupti-cu12==12.6.68 +nvidia-cuda-nvcc-cu12==12.6.68 +nvidia-cuda-runtime-cu12==12.6.68 +nvidia-cudnn-cu12==9.4.0.58 +nvidia-cufft-cu12==11.2.6.59 +nvidia-cusolver-cu12==11.6.4.69 +nvidia-cusparse-cu12==12.5.3.3 +nvidia-nccl-cu12==2.23.4 +nvidia-nvjitlink-cu12==12.6.68 +opt-einsum==3.3.0 +optree==0.12.1 +packaging==24.1 +protobuf==4.25.4 +pydantic==2.9.1 +pydantic_core==2.23.3 +Pygments==2.18.0 +python-dotenv==1.0.1 +PyYAML==6.0.2 +regex==2024.9.11 +requests==2.32.3 +rich==13.8.1 +scipy==1.14.1 +six==1.16.0 +sniffio==1.3.1 +starlette==0.35.1 +tensorboard==2.17.1 +tensorboard-data-server==0.7.2 +tensorflow==2.17.0 +tensorflow-io-gcs-filesystem==0.37.1 +tensorflow-text==2.17.0 +termcolor==2.4.0 +tqdm==4.66.5 +typing_extensions==4.12.2 +urllib3==2.2.3 +uvicorn==0.30.6 +uvloop==0.20.0 +watchfiles==0.24.0 +websockets==13.0.1 +Werkzeug==3.0.4 +wrapt==1.16.0 diff --git a/Gemma/spoken-language-tasks/requirements.txt b/Gemma/spoken-language-tasks/requirements.txt index 3644e9a..a793c9a 100644 --- a/Gemma/spoken-language-tasks/requirements.txt +++ b/Gemma/spoken-language-tasks/requirements.txt @@ -1,113 +1,113 @@ -absl-py==2.1.0 -aiohappyeyeballs==2.4.0 -aiohttp==3.10.5 -aiosignal==1.3.1 -annotated-types==0.7.0 -astunparse==1.6.3 -async-timeout==4.0.3 -attrs==24.2.0 -bleach==6.1.0 -blinker==1.8.2 -cachetools==5.5.0 -certifi==2024.8.30 -charset-normalizer==3.3.2 -click==8.1.7 -contourpy==1.3.0 -cycler==0.12.1 -datasets==2.21.0 -dill==0.3.8 -filelock==3.15.4 -flask==3.0.3 -flatbuffers==24.3.25 -fonttools==4.53.1 -frozenlist==1.4.1 -fsspec==2024.6.1 -gast==0.6.0 -google-ai-generativelanguage==0.6.6 -google-api-core==2.19.2 -google-api-python-client==2.143.0 -google-auth==2.34.0 -google-auth-httplib2==0.2.0 -google-generativeai==0.7.2 -google-pasta==0.2.0 -googleapis-common-protos==1.65.0 -grpcio==1.66.1 -grpcio-status==1.62.3 -h5py==3.11.0 -httplib2==0.22.0 -huggingface-hub==0.24.6 -idna==3.8 -importlib-metadata==8.4.0 -importlib-resources==6.4.4 -itsdangerous==2.2.0 -jax==0.4.30 -jax-cuda12-pjrt==0.4.30 -jax-cuda12-plugin==0.4.30 -jaxlib==0.4.30 -jinja2==3.1.4 -kagglehub==0.2.9 -keras==3.5.0 -keras-nlp==0.14.4 -kiwisolver==1.4.7 -libclang==18.1.1 -Markdown==3.7 -markdown-it-py==3.0.0 -MarkupSafe==2.1.5 -matplotlib==3.9.2 -mdurl==0.1.2 -ml-dtypes==0.4.0 -multidict==6.0.5 -multiprocess==0.70.16 -namex==0.0.8 -numpy==1.26.4 -nvidia-cublas-cu12==12.6.1.4 -nvidia-cuda-cupti-cu12==12.6.68 -nvidia-cuda-nvcc-cu12==12.6.68 -nvidia-cuda-runtime-cu12==12.6.68 -nvidia-cudnn-cu12==9.3.0.75 -nvidia-cufft-cu12==11.2.6.59 -nvidia-cusolver-cu12==11.6.4.69 -nvidia-cusparse-cu12==12.5.3.3 -nvidia-nccl-cu12==2.22.3 -nvidia-nvjitlink-cu12==12.6.68 -opt-einsum==3.3.0 -optree==0.12.1 -packaging==24.1 -pandas==2.2.2 -pillow==10.4.0 -proto-plus==1.24.0 -protobuf==4.25.4 -pyarrow==17.0.0 -pyasn1==0.6.0 -pyasn1-modules==0.4.0 -pydantic==2.8.2 -pydantic-core==2.20.1 -pygments==2.18.0 -pyparsing==3.1.4 -python-dateutil==2.9.0.post0 -python-dotenv==1.0.1 -pytz==2024.1 -PyYAML==6.0.2 -regex==2024.7.24 -requests==2.32.3 -rich==13.8.0 -rsa==4.9 -scipy==1.13.1 -six==1.16.0 -tensorboard==2.17.1 -tensorboard-data-server==0.7.2 -tensorflow==2.17.0 -tensorflow-io-gcs-filesystem==0.37.1 -tensorflow-text==2.17.0 -termcolor==2.4.0 -tqdm==4.66.5 -typing-extensions==4.12.2 -tzdata==2024.1 -uritemplate==4.1.1 -urllib3==2.2.2 -webencodings==0.5.1 -werkzeug==3.0.4 -wrapt==1.16.0 -xxhash==3.5.0 -yarl==1.9.7 -zipp==3.20.1 +absl-pyg +aiohappyeyeballsg +aiohttpg +aiosignalg +annotated-typesg +astunparseg +async-timeoutg +attrsg +bleachg +blinkerg +cachetoolsg +certifig +charset-normalizerg +clickg +contourpyg +cyclerg +datasetsg +dillg +filelockg +flaskg +flatbuffersg +fonttoolsg +frozenlistg +fsspecg +gastg +google-ai-generativelanguageg +google-api-coreg +google-api-python-clientg +google-authg +google-auth-httplib2g +google-generativeaig +google-pastag +googleapis-common-protosg +grpciog +grpcio-statusg +h5pyg +httplib2g +huggingface-hubg +idnag +importlib-metadatag +importlib-resourcesg +itsdangerousg +jaxg +jax-cuda12-pjrtg +jax-cuda12-pluging +jaxlibg +jinja2g +kagglehubg +kerasg +keras-nlpg +kiwisolverg +libclangg +Markdowng +markdown-it-pyg +MarkupSafeg +matplotlibg +mdurlg +ml-dtypesg +multidictg +multiprocessg +namexg +numpyg +nvidia-cublas-cu12g +nvidia-cuda-cupti-cu12g +nvidia-cuda-nvcc-cu12g +nvidia-cuda-runtime-cu12g +nvidia-cudnn-cu12g +nvidia-cufft-cu12g +nvidia-cusolver-cu12g +nvidia-cusparse-cu12g +nvidia-nccl-cu12g +nvidia-nvjitlink-cu12g +opt-einsumg +optreeg +packagingg +pandasg +pillowg +proto-plusg +protobufg +pyarrowg +pyasn1g +pyasn1-modulesg +pydanticg +pydantic-coreg +pygmentsg +pyparsingg +python-dateutilg +python-dotenvg +pytzg +PyYAMLg +regexg +requestsg +richg +rsag +scipyg +sixg +tensorboardg +tensorboard-data-serverg +tensorflowg +tensorflow-io-gcs-filesystemg +tensorflow-textg +termcolorg +tqdmg +typing-extensionsg +tzdatag +uritemplateg +urllib3g +webencodingsg +werkzeugg +wraptg +xxhashg +yarlg +zippg diff --git a/Gemma/spoken-language-tasks/requirements.txt.bak b/Gemma/spoken-language-tasks/requirements.txt.bak new file mode 100644 index 0000000..3644e9a --- /dev/null +++ b/Gemma/spoken-language-tasks/requirements.txt.bak @@ -0,0 +1,113 @@ +absl-py==2.1.0 +aiohappyeyeballs==2.4.0 +aiohttp==3.10.5 +aiosignal==1.3.1 +annotated-types==0.7.0 +astunparse==1.6.3 +async-timeout==4.0.3 +attrs==24.2.0 +bleach==6.1.0 +blinker==1.8.2 +cachetools==5.5.0 +certifi==2024.8.30 +charset-normalizer==3.3.2 +click==8.1.7 +contourpy==1.3.0 +cycler==0.12.1 +datasets==2.21.0 +dill==0.3.8 +filelock==3.15.4 +flask==3.0.3 +flatbuffers==24.3.25 +fonttools==4.53.1 +frozenlist==1.4.1 +fsspec==2024.6.1 +gast==0.6.0 +google-ai-generativelanguage==0.6.6 +google-api-core==2.19.2 +google-api-python-client==2.143.0 +google-auth==2.34.0 +google-auth-httplib2==0.2.0 +google-generativeai==0.7.2 +google-pasta==0.2.0 +googleapis-common-protos==1.65.0 +grpcio==1.66.1 +grpcio-status==1.62.3 +h5py==3.11.0 +httplib2==0.22.0 +huggingface-hub==0.24.6 +idna==3.8 +importlib-metadata==8.4.0 +importlib-resources==6.4.4 +itsdangerous==2.2.0 +jax==0.4.30 +jax-cuda12-pjrt==0.4.30 +jax-cuda12-plugin==0.4.30 +jaxlib==0.4.30 +jinja2==3.1.4 +kagglehub==0.2.9 +keras==3.5.0 +keras-nlp==0.14.4 +kiwisolver==1.4.7 +libclang==18.1.1 +Markdown==3.7 +markdown-it-py==3.0.0 +MarkupSafe==2.1.5 +matplotlib==3.9.2 +mdurl==0.1.2 +ml-dtypes==0.4.0 +multidict==6.0.5 +multiprocess==0.70.16 +namex==0.0.8 +numpy==1.26.4 +nvidia-cublas-cu12==12.6.1.4 +nvidia-cuda-cupti-cu12==12.6.68 +nvidia-cuda-nvcc-cu12==12.6.68 +nvidia-cuda-runtime-cu12==12.6.68 +nvidia-cudnn-cu12==9.3.0.75 +nvidia-cufft-cu12==11.2.6.59 +nvidia-cusolver-cu12==11.6.4.69 +nvidia-cusparse-cu12==12.5.3.3 +nvidia-nccl-cu12==2.22.3 +nvidia-nvjitlink-cu12==12.6.68 +opt-einsum==3.3.0 +optree==0.12.1 +packaging==24.1 +pandas==2.2.2 +pillow==10.4.0 +proto-plus==1.24.0 +protobuf==4.25.4 +pyarrow==17.0.0 +pyasn1==0.6.0 +pyasn1-modules==0.4.0 +pydantic==2.8.2 +pydantic-core==2.20.1 +pygments==2.18.0 +pyparsing==3.1.4 +python-dateutil==2.9.0.post0 +python-dotenv==1.0.1 +pytz==2024.1 +PyYAML==6.0.2 +regex==2024.7.24 +requests==2.32.3 +rich==13.8.0 +rsa==4.9 +scipy==1.13.1 +six==1.16.0 +tensorboard==2.17.1 +tensorboard-data-server==0.7.2 +tensorflow==2.17.0 +tensorflow-io-gcs-filesystem==0.37.1 +tensorflow-text==2.17.0 +termcolor==2.4.0 +tqdm==4.66.5 +typing-extensions==4.12.2 +tzdata==2024.1 +uritemplate==4.1.1 +urllib3==2.2.2 +webencodings==0.5.1 +werkzeug==3.0.4 +wrapt==1.16.0 +xxhash==3.5.0 +yarl==1.9.7 +zipp==3.20.1 From 27b596766a12a281b5da95c06b7c458ae316fc27 Mon Sep 17 00:00:00 2001 From: Kinar R Date: Mon, 28 Oct 2024 17:35:35 +0530 Subject: [PATCH 22/43] Added Gemma Genkit RAG notebook --- ...emma_with_Firebase_Genkit_and_Ollama.ipynb | 1544 +++++++++++++++++ 1 file changed, 1544 insertions(+) create mode 100644 Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb diff --git a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb new file mode 100644 index 0000000..bfebd69 --- /dev/null +++ b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb @@ -0,0 +1,1544 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "c4b7e9a1" + }, + "source": [ + "# Building a RAG Application with Genkit, Firebase, Ollama, Gemma, Dotprompt, and Gradio\n", + "\n", + "\n", + "\n", + "In this comprehensive tutorial, you will learn how to build a **Retrieval-Augmented Generation (RAG)** application using cutting-edge technologies:\n", + "\n", + "[**Genkit**](#) is a framework designed to help you build AI-powered applications and features. It provides open-source libraries for Node.js and Go, plus developer tools for testing and debugging.\n", + "\n", + "[**Gemma**](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs) available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n", + "\n", + "[**Ollama**](https://ollama.ai/) is a tool that simplifies running language models locally. It allows you to manage and serve multiple models efficiently, making it easier to deploy and test AI models on your machine. With Ollama, you can switch between different models and versions seamlessly, providing flexibility in development and experimentation.\n", + "\n", + "[**Firebase**](https://firebase.google.com/) is a comprehensive app development platform by Google that provides services like real-time databases, authentication, cloud storage, hosting, and machine learning. In this tutorial, we will utilize the **Cloud Firestore**, a scalable, flexible NoSQL cloud database to store and sync data for client- and server-side development.\n", + "\n", + "[**Gradio**](https://gradio.app/) is an open-source Python library for creating user-friendly web interfaces to interact with machine learning models. It allows you to quickly create customizable UI components to interact with your models and also generate shareable web apps that anyone can use.\n", + "\n", + "By integrating these technologies, you will build a powerful RAG application capable of providing accurate and contextually relevant responses based on your custom data.\n", + "\n", + "## What you'll learn\n", + "\n", + "- **Setting Up the Development Environment**: Install and configure Node.js, Genkit, Firebase, Ollama, and Gradio within a Colab notebook.\n", + "- **Managing Prompts with Dotprompt**: Modularize your prompts into separate `.prompt` files using **Dotprompt** for better organization and maintainability.\n", + "- **Indexing Documents with Genkit Flows**: Use Genkit's flows to embed and index your data, making it retrievable for your RAG application.\n", + "- **Building a Chatbot Interface**: Create a user-friendly chatbot interface with Gradio to interact with your app.\n", + "\n", + "\n", + "Let's get started on building your RAG application!\n", + "\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "78f2d1e5" + }, + "source": [ + "## **Setup**\n", + "\n", + "Before you begin, make sure you have:\n", + "\n", + "- A basic Google Cloud account.\n", + "- Basic knowledge of Node.js and TypeScript.\n", + "- Familiarity with Colab notebooks.\n", + "- The latest version of **Google Cloud SDK** installed." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "setup-colab-runtime" + }, + "source": [ + "## Select the Colab Runtime\n", + "\n", + "In this section, you'll configure Google Colab and set up the tools needed for this project. You'll be using Google Colab as the environment to run your code, so make sure to follow these steps carefully.\n", + "\n", + "1. **Open Google Colab** and create a new notebook.\n", + "2. In the upper-right corner of the Colab window, click on the **โ–พ (Additional connection options)** button.\n", + "3. Select **Change runtime type**.\n", + "4. Under **Hardware accelerator**, choose **GPU**.\n", + "5. Ensure that the **GPU type** is set to **T4**.\n", + "\n", + "This setup will give you enough computing power to run the Gemma model smoothly.\n", + "\n", + "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "configure-credentials" + }, + "source": [ + "### Configure Your Credentials\n", + "\n", + "First, get your Google API key from: https://aistudio.google.com/app/apikey\n", + "\n", + "You need to set up credentials for Google AI Studio in Google Colab. This allows you to authenticate and securely interact with different services, such as Google AI Studio.\n", + "\n", + "\n", + "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + "2. **Add Google API Key**:\n", + " - Create a new secret named `GOOGLE_API_KEY`.\n", + " - Paste your Google API Key into the Value input box.\n", + " - Toggle the button to allow notebook access to the secret.\n", + "\n", + "Now, set the environment variables in your notebook:" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0c9e3f25" + }, + "source": [ + "## **Install dependencies**\n", + "\n", + "To build the RAG application, you need to install various tools and libraries. Let's get started with installing the dependencies." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "d7328eb3" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m50.4/50.4 kB\u001b[0m \u001b[31m533.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m42.3/42.3 MB\u001b[0m \u001b[31m17.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m319.8/319.8 kB\u001b[0m \u001b[31m14.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m94.7/94.7 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m76.4/76.4 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m78.0/78.0 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m436.6/436.6 kB\u001b[0m \u001b[31m27.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m141.9/141.9 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m10.9/10.9 MB\u001b[0m \u001b[31m57.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m63.7/63.7 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m73.3/73.3 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m130.2/130.2 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h>>> Installing ollama to /usr/local\n", + ">>> Downloading Linux amd64 bundle\n", + "############################################################################################# 100.0%\n", + ">>> Creating ollama user...\n", + ">>> Adding ollama user to video group...\n", + ">>> Adding current user to ollama group...\n", + ">>> Creating ollama systemd service...\n", + "WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.\n", + ">>> The Ollama API is now available at 127.0.0.1:11434.\n", + ">>> Install complete. Run \"ollama\" from the command line.\n", + "\u001b[38;5;79m2024-10-17 12:00:38 - Installing pre-requisites\u001b[0m\n", + "Get:1 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease [3,626 B]\n", + "Get:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease [1,581 B]\n", + "Hit:3 http://archive.ubuntu.com/ubuntu jammy InRelease\n", + "Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]\n", + "Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]\n", + "Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 Packages [1,031 kB]\n", + "Ign:7 https://r2u.stat.illinois.edu/ubuntu jammy InRelease\n", + "Get:8 https://r2u.stat.illinois.edu/ubuntu jammy Release [5,713 B]\n", + "Get:9 https://r2u.stat.illinois.edu/ubuntu jammy Release.gpg [793 B]\n", + "Get:10 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]\n", + "Get:11 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,162 kB]\n", + "Get:12 https://r2u.stat.illinois.edu/ubuntu jammy/main all Packages [8,396 kB]\n", + "Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,451 kB]\n", + "Get:14 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2,372 kB]\n", + "Get:15 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease [18.1 kB]\n", + "Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [3,200 kB]\n", + "Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [3,278 kB]\n", + "Hit:18 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n", + "Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,648 kB]\n", + "Hit:20 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n", + "Get:21 https://r2u.stat.illinois.edu/ubuntu jammy/main amd64 Packages [2,598 kB]\n", + "Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [33.7 kB]\n", + "Get:23 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy/main amd64 Packages [33.9 kB]\n", + "Fetched 26.6 MB in 3s (9,194 kB/s)\n", + "Reading package lists... Done\n", + "W: Skipping acquire of configured file 'main/source/Sources' as repository 'https://r2u.stat.illinois.edu/ubuntu jammy InRelease' does not seem to provide it (sources.list entry misspelt?)\n", + "Reading package lists... Done\n", + "Building dependency tree... Done\n", + "Reading state information... Done\n", + "ca-certificates is already the newest version (20240203~22.04.1).\n", + "curl is already the newest version (7.81.0-1ubuntu1.18).\n", + "gnupg is already the newest version (2.2.27-3ubuntu2.1).\n", + "gnupg set to manually installed.\n", + "The following NEW packages will be installed:\n", + " apt-transport-https\n", + "0 upgraded, 1 newly installed, 0 to remove and 52 not upgraded.\n", + "Need to get 1,510 B of archives.\n", + "After this operation, 170 kB of additional disk space will be used.\n", + "Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 apt-transport-https all 2.4.13 [1,510 B]\n", + "Fetched 1,510 B in 0s (11.0 kB/s)\n", + "Selecting previously unselected package apt-transport-https.\n", + "(Reading database ... 123629 files and directories currently installed.)\n", + "Preparing to unpack .../apt-transport-https_2.4.13_all.deb ...\n", + "Unpacking apt-transport-https (2.4.13) ...\n", + "Setting up apt-transport-https (2.4.13) ...\n", + "Hit:1 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease\n", + "Hit:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 InRelease\n", + "Get:3 https://deb.nodesource.com/node_20.x nodistro InRelease [12.1 kB]\n", + "Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease\n", + "Hit:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease\n", + "Ign:6 https://r2u.stat.illinois.edu/ubuntu jammy InRelease\n", + "Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease\n", + "Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease\n", + "Get:9 https://deb.nodesource.com/node_20.x nodistro/main amd64 Packages [9,254 B]\n", + "Hit:10 https://r2u.stat.illinois.edu/ubuntu jammy Release\n", + "Hit:11 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease\n", + "Hit:13 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease\n", + "Hit:14 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease\n", + "Fetched 21.4 kB in 1s (20.7 kB/s)\n", + "Reading package lists... Done\n", + "W: Skipping acquire of configured file 'main/source/Sources' as repository 'https://r2u.stat.illinois.edu/ubuntu jammy InRelease' does not seem to provide it (sources.list entry misspelt?)\n", + "\u001b[1;34m2024-10-17 12:00:51 - Repository configured successfully.\u001b[0m\n", + "\u001b[38;5;79m2024-10-17 12:00:51 - To install Node.js, run: apt-get install nodejs -y\u001b[0m\n", + "\u001b[38;5;79m2024-10-17 12:00:51 - You can use N|solid Runtime as a node.js alternative\u001b[0m\n", + "\u001b[1;32m2024-10-17 12:00:51 - To install N|solid Runtime, run: apt-get install nsolid -y \n", + "\u001b[0m\n", + "Reading package lists... Done\n", + "Building dependency tree... Done\n", + "Reading state information... Done\n", + "The following NEW packages will be installed:\n", + " nodejs\n", + "0 upgraded, 1 newly installed, 0 to remove and 52 not upgraded.\n", + "Need to get 31.8 MB of archives.\n", + "After this operation, 197 MB of additional disk space will be used.\n", + "Get:1 https://deb.nodesource.com/node_20.x nodistro/main amd64 nodejs amd64 20.18.0-1nodesource1 [31.8 MB]\n", + "Fetched 31.8 MB in 1s (49.2 MB/s)\n", + "debconf: unable to initialize frontend: Dialog\n", + "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 78, <> line 1.)\n", + "debconf: falling back to frontend: Readline\n", + "debconf: unable to initialize frontend: Readline\n", + "debconf: (This frontend requires a controlling tty.)\n", + "debconf: falling back to frontend: Teletype\n", + "dpkg-preconfigure: unable to re-open stdin: \n", + "Selecting previously unselected package nodejs.\n", + "(Reading database ... 123633 files and directories currently installed.)\n", + "Preparing to unpack .../nodejs_20.18.0-1nodesource1_amd64.deb ...\n", + "Unpacking nodejs (20.18.0-1nodesource1) ...\n", + "Setting up nodejs (20.18.0-1nodesource1) ...\n", + "Processing triggers for man-db (2.10.2-1) ...\n", + "\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "added 257 packages in 19s\n", + "\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0K\n", + "\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0K53 packages are looking for funding\n", + "\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0K run `npm fund` for details\n", + "\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0K\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\n", + "added 201 packages in 15s\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K24 packages are looking for funding\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K run `npm fund` for details\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\n", + "added 253 packages, and audited 455 packages in 25s\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K37 packages are looking for funding\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K run `npm fund` for details\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\n", + "found \u001b[32m\u001b[1m0\u001b[22m\u001b[39m vulnerabilities\n", + "\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0K\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "added 2 packages, and audited 457 packages in 2s\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K37 packages are looking for funding\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K run `npm fund` for details\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "found \u001b[32m\u001b[1m0\u001b[22m\u001b[39m vulnerabilities\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "added 12 packages, and audited 469 packages in 3s\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K38 packages are looking for funding\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K run `npm fund` for details\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\n", + "found \u001b[32m\u001b[1m0\u001b[22m\u001b[39m vulnerabilities\n", + "\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0K\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0Kโ น\u001b[1G\u001b[0Kโ ธ\u001b[1G\u001b[0Kโ ผ\u001b[1G\u001b[0Kโ ด\u001b[1G\u001b[0Kโ ฆ\u001b[1G\u001b[0Kโ ง\u001b[1G\u001b[0Kโ ‡\u001b[1G\u001b[0Kโ \u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0K\n", + "added 1 package, and audited 470 packages in 1s\n", + "\u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0K\n", + "\u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0K38 packages are looking for funding\n", + "\u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0K run `npm fund` for details\n", + "\u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0K\n", + "found \u001b[32m\u001b[1m0\u001b[22m\u001b[39m vulnerabilities\n", + "\u001b[1G\u001b[0Kโ ‹\u001b[1G\u001b[0K" + ] + } + ], + "source": [ + "# Install Gradio\n", + "!pip install -q gradio\n", + "\n", + "# Install Ollama\n", + "!curl -fsSL https://ollama.ai/install.sh | sh\n", + "\n", + "# Install Node.js\n", + "!curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -\n", + "!sudo apt-get install -y nodejs\n", + "\n", + "# Install Genkit CLI and plugins\n", + "!npm i -g genkit\n", + "!npm i --save genkitx-ollama\n", + "!npm i --save @genkit-ai/firebase\n", + "!npm i --save @genkit-ai/googleai\n", + "!npm i --save @genkit-ai/dotprompt\n", + "!npm i llm-chunk" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5e0f4df2" + }, + "source": [ + "## Run Gemma using Ollama\n", + "\n", + "You will use `Ollama` to run the Gemma language model locally. This tool will allow you to interact with the AI and use it in your RAG chatbot application." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hA8ua09eXKzf" + }, + "source": [ + "First, start the Ollama server. This will run in the background and allow you to call different AI models." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5E5J_0wZXKQ0" + }, + "outputs": [], + "source": [ + "import subprocess\n", + "import time\n", + "\n", + "ollama_serve_process = subprocess.Popen(\"OLLAMA_KEEP_ALIVE=-1 ollama serve\", shell=True)\n", + "time.sleep(5)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "01bfdd4c" + }, + "source": [ + "Ollama provides a library of pre-configured models, including Gemma 2 models. You can browse the available Gemma 2 models at the [Ollama Gemma 2 Model Catalog](https://ollama.com/library/gemma2). This allows you to switch between different Gemma 2 models easily. In this notebook, you'll use the [gemma2:2b](https://ollama.com/library/gemma2:2b) model.\n", + "\n", + "To test if the Gemma model is running correctly, use the following command to ask the model a simple question:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "bfad771e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The capital of China is **Beijing**. \n", + "\n", + "\n", + "\n" + ] + } + ], + "source": [ + "ollama_run_process = subprocess.Popen(\n", + " \"ollama run gemma2:2b 'What is the capital of China?'\",\n", + " shell=True, stdout=subprocess.PIPE, text=True\n", + ")\n", + "\n", + "print(ollama_run_process.communicate()[0])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e0a71a35" + }, + "source": [ + "You should see the model's response in the output." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "85c1bb7b" + }, + "source": [ + "## Set up the Firebase project\n", + "\n", + "Firebase will be used to store and manage your data. In this case, we'll use Cloud Firestore, a NoSQL database that makes it easy to store and retrieve the information that will be used by the chatbot.\n", + "\n", + "Before you can continue, you need to set up a Firebase project:\n", + "\n", + "1. If you haven't already, create a Firebase project: In the [Firebase console](https://console.firebase.google.com/), click Add project, then follow the on-screen instructions to create a Firebase project or to add Firebase services to an existing GCP project.\n", + "\"Welcome\n", + "\n", + "2. Then, open your project and go to the **Project settings** page, create a service account and download the service account key file using **Generate new private key**. Keep this file safe, since it grants administrator access to your project. \n", + "\"Project \n", + "\"Service\n", + "\n", + "\n", + "3. Upload the JSON service account key file and set its location in `GOOGLE_APPLICATION_CREDENTIALS` environmental variable.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "yuPcUoQRZtf3" + }, + "outputs": [], + "source": [ + "import os\n", + "from google.colab import files\n", + "\n", + "uploaded = files.upload()\n", + "\n", + "for fn in uploaded.keys():\n", + " print('User uploaded file \"{name}\" with length {length} bytes'.format(\n", + " name=fn, length=len(uploaded[fn])))\n", + "\n", + " with open('/content/' + fn, 'wb') as f:\n", + " f.write(uploaded[fn])\n", + "\n", + " os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = '/content/' + fn" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NOGTSWz_a1kZ" + }, + "source": [ + "This will allow you to authenticate to Firebase and use its services." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "etqnpcRgaexr" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "my-genkit-gemma-firebase-demo\n" + ] + } + ], + "source": [ + "# Get the project ID\n", + "import json\n", + "\n", + "with open(os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"], 'r') as f:\n", + " data = json.load(f)\n", + " PROJECT_ID = data['project_id']\n", + " print(PROJECT_ID)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QCfndtotoec-" + }, + "source": [ + "## Create a Cloud Firestore database\n", + "\n", + "Now that you've set up Firebase and your development environment, let's move on to creating the Firestore database.\n", + "\n", + "* Navigate to the **Cloud Firestore** section of the [Firebase console](https://console.firebase.google.com/project/_/firestore). You'll be prompted to select an existing Firebase project. Follow the database creation workflow.\n", + "\"Service\n", + "\n", + "* To simplify our demo, let's select a starting mode for your **Cloud Firestore Security Rules**. We'll pick **Test mode** to get quickly started.\n", + "\n", + "* Pick a location and this setting will be your project's default Google Cloud Platform (GCP) resource location.\n", + "\n", + "**Note**: Test mode allows open access to your database, which is insecure for production environments. Remember to update your security rules before deploying your application." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5bOgqUuAxTxQ" + }, + "source": [ + "## Create a vector index for Firestore\n", + "\n", + "Before you can perform a nearest neighbor search with your vector embeddings, you must create a corresponding index." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TsBVtzpMx5Lt" + }, + "source": [ + "To do this, let's first authenticate the Google Cloud SDK to simplify its creation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "jeIxgHH6eQO4" + }, + "outputs": [], + "source": [ + "!gcloud auth login --no-launch-browser" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JIH4LCxzpEAu" + }, + "source": [ + "Follow the authentication flow in your browser, and copy the authentication code back into the Colab notebook when prompted." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KSP8HmW9rB-L" + }, + "source": [ + "Firestore depends on indexes to provide fast and efficient querying on collections.\n", + "\n", + "**Note**: \"index\" here refers to database indexes, and not Genkit's indexer and retriever abstractions.\n", + "\n", + "The tutorial requires the embedding field to be indexed to work.\n", + "\n", + "Run the following `gcloud` command as described in the [Firestore docs](https://firebase.google.com/docs/firestore/vector-search?authuser=0#create_and_manage_vector_indexes) to create a single-field vector index.\n", + "\n", + "* `collection-group` is the ID of the collection group.\n", + "* `vector-field` is the name of the field that contains the vector embedding.\n", + "* `field-config` includes the vector configuration (vector dimension and index type). The dimension is an integer up to 2048. The index type must be flat. You also specify the `field-path` here which is `embedding`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "zJRRcwttoS2k" + }, + "outputs": [], + "source": [ + "%%bash -s \"$PROJECT_ID\"\n", + "\n", + "# Set the current project ID\n", + "gcloud config set project $1\n", + "\n", + "# Create a vector index\n", + "gcloud alpha firestore indexes composite create \\\n", + " --project=$1 \\\n", + " --collection-group=merch \\\n", + " --query-scope=COLLECTION \\\n", + " --field-config=vector-config='{\"dimension\":\"768\",\"flat\": \"{}\"}',field-path=embedding" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-SIj1nF6sqj6" + }, + "source": [ + "## Retrieval-Augmented Generation (RAG)\n", + "\n", + "**Firebase Genkit** provides abstractions that help you build retrieval-augmented generation (RAG) flows, as well as plugins that provide integrations with related tools.\n", + "\n", + "What is RAG?\n", + "Retrieval-augmented generation is a technique used to incorporate external sources of information into an LLMโ€™s responses. It's important to be able to do so because, while LLMs are typically trained on a broad body of material, practical use of LLMs often requires specific domain knowledge (for example, you might want to use an LLM to answer customers' questions about your companyโ€™s products).\n", + "\n", + "The core Genkit framework offers the abstractions to help you do RAG:\n", + "\n", + "* **Indexers**: add documents to an `\"index\"`.\n", + "* **Embedders**: transforms documents into a vector representation\n", + "* **Retrievers**: retrieve documents from an `\"index\"`, given a query.\n", + "These definitions are broad on purpose because Genkit is un-opinionated about what an `\"index\"` is or how exactly documents are retrieved from it. Genkit only provides a `Document` format and everything else is defined by the retriever or indexer implementation provider.\n", + "\n", + "\n", + "You'll soon learn how it's possible to ingest a collection of product descriptions into a vector database and retrieve them for use in a flow that determines what items are available. You can even ask general questions about your custom data and Gemma should be able to make sense out of the relevant context that's retrieved from a user query." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7c59aeab" + }, + "source": [ + "### Genkit Project Setup\n", + "\n", + "Create a new project directory and initialize an NPM project." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "4a3c2f56" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "/content/genkit-gemma-sample\n", + "\u001b[1G\u001b[0KWrote to /content/genkit-gemma-sample/package.json:\n", + "\n", + "{\n", + " \"name\": \"genkit-gemma-sample\",\n", + " \"version\": \"1.0.0\",\n", + " \"main\": \"index.js\",\n", + " \"scripts\": {\n", + " \"test\": \"echo \\\"Error: no test specified\\\" && exit 1\"\n", + " },\n", + " \"keywords\": [],\n", + " \"author\": \"\",\n", + " \"license\": \"ISC\",\n", + " \"description\": \"\"\n", + "}\n", + "\n", + "\n", + "\n", + "\u001b[1G\u001b[0Kโ ™\u001b[1G\u001b[0K" + ] + } + ], + "source": [ + "# Create project directory\n", + "!mkdir genkit-gemma-sample\n", + "%cd genkit-gemma-sample\n", + "\n", + "# Initialize NPM project\n", + "!npm init -y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e6cc9098" + }, + "source": [ + "Initialize a Genkit project and create a sample Ollama project that uses the older Gemma model. You'll update this to use the latest model later. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "d8e38328" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[?25l\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully installed NPM packages\n", + "\u001b[?25l\u001b[36mโ ‹\u001b[39m Updating tsconfig.json\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully updated tsconfig.json\n", + "\u001b[?25l\u001b[36mโ ‹\u001b[39m Updating package.json\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully updated package.json\n", + "\u001b[?25l\u001b[36mโ ‹\u001b[39m Generating sample file\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully generated sample file (src/index.ts)\n", + "If you don't have Ollama already installed and configured, refer to https://developers.google.com/genkit/plugins/ollama\n", + "\n", + "Genkit successfully initialized.\n", + "\u001b[?25h" + ] + } + ], + "source": [ + "!genkit init --model ollama --non-interactive" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0a4a0b1a" + }, + "source": [ + "### Prepare Data for RAG\n", + "\n", + "Create a file named `products.txt` that contains descriptions of the products you want the chatbot to know about." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "6aeff45c" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing products.txt\n" + ] + } + ], + "source": [ + "%%writefile products.txt\n", + "\n", + "**Google Pixel 8 Pro - Obsidian - Google Store**\n", + "\n", + "The Google Pixel 8 Pro is Google's latest flagship smartphone, featuring a 6.7-inch LTPO OLED display with a 120Hz refresh rate for smooth and vibrant visuals.\n", + "Powered by the Google Tensor G3 chip, it offers exceptional performance, advanced AI capabilities, and enhanced security with the Titan M2 security chip.\n", + "The Pixel 8 Pro boasts a versatile triple-camera system, including a 50 MP main sensor, a 48 MP telephoto lens, and a 48 MP ultra-wide lens, enabling you to capture high-quality photos and videos in various lighting conditions.\n", + "Innovative camera features like Magic Eraser, Night Sight, and Super Res Zoom enhance your photography experience. The device supports 5G connectivity, has an all-day battery life with fast charging and wireless charging capabilities, and runs on the latest Android OS with guaranteed software updates.\n", + "\n", + "* **Price:** Starting at $999\n", + "* **Reviews:** 4.8 out of 5 stars based on customer reviews on the Google Store\n", + "\n", + "---\n", + "\n", + "**Samsung Galaxy Watch5 Pro - 45mm Bluetooth Smartwatch - Black Titanium - Samsung**\n", + "\n", + "The Samsung Galaxy Watch5 Pro is a premium smartwatch designed for outdoor enthusiasts and fitness aficionados.\n", + "Featuring a durable Titanium case and Sapphire Crystal Glass, it's built to withstand tough conditions.\n", + "The watch includes advanced health monitoring features like ECG, blood pressure measurement, and body composition analysis. It offers GPS route tracking, turn-by-turn navigation, and has a battery life of up to 80 hours.\n", + "The Galaxy Watch5 Pro runs on Wear OS Powered by Samsung, providing access to a wide range of apps.\n", + "\n", + "* **Price:** $449.99\n", + "* **Reviews:** 4.5 out of 5 stars based on customer reviews on the Samsung website\n", + "\n", + "---\n", + "\n", + "**Dell XPS 13 Laptop - 13.4-inch FHD+ Display - Platinum Silver - Dell**\n", + "\n", + "The Dell XPS 13 is a compact and powerful laptop featuring a 13.4-inch InfinityEdge FHD+ display.\n", + "Powered by up to the 11th Gen Intel Core processors, it delivers excellent performance for multitasking and creative work.\n", + "The laptop boasts a sleek design with a machined aluminum chassis and a carbon fiber palm rest.\n", + "It includes up to 16 GB of RAM and up to 1 TB of SSD storage. With a long battery life and Wi-Fi 6 connectivity, the XPS 13 is ideal for professionals on the go.\n", + "\n", + "* **Price:** Starting at $999.99\n", + "* **Reviews:** 4.6 out of 5 stars based on customer reviews on Dell's website\n", + "\n", + "---\n", + "\n", + "**Bose QuietComfort 45 Wireless Noise-Cancelling Headphones - Black - Bose**\n", + "\n", + "The Bose QuietComfort 45 headphones offer world-class noise cancellation with two modes: Quiet and Aware.\n", + "They deliver high-fidelity audio with a balanced performance at any volume. The headphones are lightweight and feature synthetic leather ear cushions for all-day comfort.\n", + "With up to 24 hours of battery life on a single charge, they are perfect for long flights or extended listening sessions. The headphones support Bluetooth 5.1 for a strong and reliable wireless connection.\n", + "\n", + "* **Price:** $329\n", + "* **Reviews:** 4.8 out of 5 stars based on customer reviews on Bose's website\n", + "\n", + "---\n", + "\n", + "**Canon EOS R6 Mirrorless Camera Body - Canon Online Store**\n", + "\n", + "The Canon EOS R6 is a full-frame mirrorless camera designed for both enthusiasts and professionals.\n", + "It features a 20.1 MP CMOS sensor and the DIGIC X image processor, providing excellent image quality and high-speed performance.\n", + "The camera offers up to 12 fps mechanical shutter and 20 fps electronic (silent) shutter, making it ideal for action photography.\n", + "It includes 4K UHD video recording, in-body image stabilization, and Dual Pixel CMOS AF II for fast and accurate autofocus.\n", + "The EOS R6 has built-in Wi-Fi and Bluetooth for easy sharing and remote control.\n", + "\n", + "* **Price:** $2,499\n", + "* **Reviews:** 4.7 out of 5 stars based on customer reviews on the Canon Online Store\n", + "\n", + "---\n", + "\n", + "**Apple AirPods Pro (2nd Generation) - White - Apple Store**\n", + "\n", + "The Apple AirPods Pro (2nd Generation) offer superior sound quality with Active Noise Cancellation and Adaptive Transparency.\n", + "Equipped with the H2 chip, they deliver high-fidelity audio with personalized Spatial Audio features.\n", + "The earbuds come with four sizes of silicone ear tips for a customizable fit and include touch controls for media playback and volume adjustment.\n", + "With improved battery life, you get up to 6 hours of listening time on a single charge and up to 30 hours with the MagSafe Charging Case.\n", + "\n", + "* **Price:** $249\n", + "* **Reviews:** 4.7 out of 5 stars based on customer reviews on the Apple Store\n", + "\n", + "---\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "775e4609" + }, + "source": [ + "### Create the Prompt File with **Dotprompt**\n", + "\n", + "Firebase Genkit provides the Dotprompt plugin and text format to help you write and organize your generative AI prompts.\n", + "\n", + "Dotprompt helps you organize and manage the prompts used by the language model. Create a .prompt file to define how the model should interact with the data and users. This makes it easier to maintain and version your prompts, similar to how you manage code.\n", + "\n", + "\n", + "Create `assistant.prompt` in the `src/prompts` directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "98Ztvuuc_fLh" + }, + "outputs": [], + "source": [ + "# Create a `prompts` directory to store your Dotprompts\n", + "!mkdir -p prompts" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "2ee08e36" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing prompts/assistant.prompt\n" + ] + } + ], + "source": [ + "%%writefile prompts/assistant.prompt\n", + "---\n", + "model: ollama/gemma2:2b\n", + "config:\n", + " temperature: 0.8\n", + "input:\n", + " schema:\n", + " data(array): string\n", + " question: string\n", + "output:\n", + " format: text\n", + "---\n", + "You are acting as a helpful AI assistant that can answer questions using the data that's available.\n", + "\n", + "Use only the context provided to answer the question.\n", + "If you don't know, do not make up an answer.\n", + "\n", + "Context:\n", + "{{#each data~}}\n", + "- {{this}}\n", + "{{~/each}}\n", + "\n", + "Question:\n", + "{{question}}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "429e0b4d" + }, + "source": [ + "### Chunking, Embedding and Indexing\n", + "\n", + "You will then use Genkit to embed and index these product descriptions into Firestore so that the chatbot can retrieve them when answering questions by running the following steps:\n", + "\n", + "* **Chunking**: Next, use `llm-chunk` to break these product descriptions into smaller, manageable chunks. Chunking the data helps ensure that the content is in a suitable size for embedding, making it more effective when working with vector representations. The `llm-chunk` library provides a simple way to split the text into segments that can be vectorized.\n", + "\n", + "* **Embedding**: An embedder is a function that takes content (text, images, audio, etc.) and creates a numeric vector that encodes the semantic meaning of the original content. To populate your Firestore collection, use the `Gecko embeddings` from Google AI along with the `Firebase Admin SDK`.\n", + "\n", + "* **Indexing**: Once the embeddings are created, index them into Firestore so that they can be used later for similarity searches. Store both the text and its embedding in Firestore.\n", + "\n", + "Create `embedFlow.ts` in the `src/flows` directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "b223519a" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing src/embedFlow.ts\n" + ] + } + ], + "source": [ + "%%writefile src/embedFlow.ts\n", + "\n", + "import { configureGenkit } from \"@genkit-ai/core\";\n", + "import { embed } from \"@genkit-ai/ai/embedder\";\n", + "import { defineFlow, run } from \"@genkit-ai/flow\";\n", + "import { textEmbeddingGecko001, googleAI } from \"@genkit-ai/googleai\";\n", + "import { FieldValue, getFirestore } from \"firebase-admin/firestore\";\n", + "import { chunk } from \"llm-chunk\";\n", + "import * as z from \"zod\";\n", + "import { readFile } from \"fs/promises\";\n", + "import path from \"path\";\n", + "\n", + "// Configuration for indexing process\n", + "const indexConfig = {\n", + " collection: \"merch\", // Firestore collection to store the data\n", + " contentField: \"text\", // Field name for the text content\n", + " vectorField: \"embedding\", // Field name for the embedding vector\n", + " embedder: textEmbeddingGecko001, // Embedder model to use\n", + "};\n", + "\n", + "// Configure Genkit with Google AI plugin\n", + "// Firebase Genkit has a configuration and plugin system.\n", + "// Every Genkit app starts with configuration where you specify the plugins\n", + "// you want to use and configure various subsystems.\n", + "configureGenkit({\n", + " plugins: [googleAI({ apiVersion: ['v1', 'v1beta'] })],\n", + " enableTracingAndMetrics: false,\n", + "});\n", + "\n", + "// Initialize Firestore instance\n", + "const firestore = getFirestore();\n", + "\n", + "// Create chunking config\n", + "// This example uses the llm-chunk library which provides a simple text\n", + "// splitter to break up documents into segments that can be vectorized.\n", + "const chunkingConfig = {\n", + " minLength: 1000,\n", + " maxLength: 2000,\n", + " splitter: 'sentence',\n", + " overlap: 100,\n", + " // Split text into chunks using '---' as delimiter\n", + " delimiters: '---',\n", + "} as any;\n", + "\n", + "// Define embed flow\n", + "export const embedFlow = defineFlow(\n", + " {\n", + " name: \"embedFlow\", // Name of the flow\n", + " inputSchema: z.void(), // No input is expected\n", + " outputSchema: z.void(), // No output is returned\n", + " },\n", + " async () => {\n", + " // Read text data from file\n", + " const filePath = path.resolve('products.txt');\n", + " const textData = await run(\"extract-text\", () => extractText(filePath));\n", + "\n", + " // Divide the text into segments.\n", + " const chunks = await run('chunk-it', async () =>\n", + " chunk(textData, chunkingConfig)\n", + " );\n", + "\n", + " // Index chunks into Firestore.\n", + " await run(\"index-chunks\", async () => indexToFirestore(chunks));\n", + " }\n", + ");\n", + "\n", + "// Function to index chunks into Firestore\n", + "async function indexToFirestore(data: string[]) {\n", + " for (const text of data) {\n", + " // Generate embedding for the text chunk\n", + " const embedding = await embed({\n", + " embedder: indexConfig.embedder,\n", + " content: text,\n", + " });\n", + "\n", + " // Add the text and embedding to Firestore\n", + " await firestore.collection(indexConfig.collection).add({\n", + " [indexConfig.vectorField]: FieldValue.vector(embedding),\n", + " [indexConfig.contentField]: text,\n", + " });\n", + " }\n", + "}\n", + "\n", + "// Function to read text content from a file\n", + "async function extractText(filePath: string) {\n", + " const f = path.resolve(filePath);\n", + " return await readFile(f, 'utf-8');\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rwO8RRmDdJmw" + }, + "source": [ + "By storing both the text and its embedding, we can later perform similarity searches to find relevant product descriptions based on user queries. This makes it possible for the chatbot to retrieve and provide accurate, contextually relevant answers." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "80ab1d4e" + }, + "source": [ + "### Configuration and plugins\n", + "\n", + "Firebase Genkit has a configuration and plugin system. Every Genkit app starts with configuration where you specify the plugins you want to use and configure various subsystems.\n", + "\n", + "Create `config.ts` in the `src` directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "f83f1c3b" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing src/config.ts\n" + ] + } + ], + "source": [ + "%%writefile src/config.ts\n", + "\n", + "import { configureGenkit } from '@genkit-ai/core';\n", + "import { firebase } from '@genkit-ai/firebase';\n", + "import { googleAI } from '@genkit-ai/googleai';\n", + "import { ollama } from 'genkitx-ollama';\n", + "import { dotprompt } from '@genkit-ai/dotprompt';\n", + "import { initializeApp, applicationDefault } from 'firebase-admin/app';\n", + "import { getFirestore } from 'firebase-admin/firestore';\n", + "\n", + "// Initialize Firebase Admin SDK\n", + "const app = initializeApp({\n", + " credential: applicationDefault(),\n", + "});\n", + "\n", + "export const firestore = getFirestore(app);\n", + "\n", + "// Configure Genkit\n", + "configureGenkit({\n", + " plugins: [\n", + " firebase(),\n", + " googleAI({ apiVersion: ['v1', 'v1beta'] }),\n", + " ollama({\n", + " // Ollama provides an interface to many generative models. Here,\n", + " // we specify Google's Gemma 2 model. The models you specify must already\n", + " // be downloaded and available to the Ollama server.\n", + " models: [{ name: 'gemma2:2b' }],\n", + " // The address of your Ollama API server. This is often a different host\n", + " // from your app backend (which runs Genkit), in order to run Ollama on\n", + " // a GPU-accelerated machine.\n", + " serverAddress: 'http://127.0.0.1:11434',\n", + " }),\n", + " dotprompt(),\n", + " ],\n", + " // Log debug output to tbe console.\n", + " logLevel: 'debug',\n", + " // Perform OpenTelemetry instrumentation and enable trace collection.\n", + " enableTracingAndMetrics: true,\n", + "});" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "82f727e4" + }, + "source": [ + "### Defining a RAG Flow\n", + "\n", + "Next, create a flow named `chatbotFlow` that will allow the chatbot to interact with the data you indexed earlier. This flow will combine a retriever (to get relevant information from Firebase) with a prompt that helps format responses. A retriever is a concept that encapsulates logic related to any kind of document retrieval. The most popular retrieval cases typically include retrieval from vector stores; however, in Genkit, a retriever can be any function that returns data. In this case, the retriever is responsible for finding the most relevant product descriptions from Firestore based on the user's question. It uses the **embeddings** and **cosine similarity** to find the closest matches, ensuring that the retrieved information is highly relevant to the query.\n", + "\n", + "Create `memory.ts` and `chatbotFlow.ts` in the `src` directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RPbtRN6lOTk6" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing src/memory.ts\n" + ] + } + ], + "source": [ + "%%writefile src/memory.ts\n", + "\n", + "import { MessageData } from '@genkit-ai/ai/model';\n", + "\n", + "const chatHistory: Record = {};\n", + "\n", + "export interface HistoryStore {\n", + " load(id: string): Promise;\n", + " save(id: string, history: MessageData[]): Promise;\n", + "}\n", + "\n", + "// You'll also use an in-memory store to store the chat history.\n", + "export function inMemoryStore(): HistoryStore {\n", + " return {\n", + " async load(id: string): Promise {\n", + " return chatHistory[id];\n", + " },\n", + " async save(id: string, history: MessageData[]) {\n", + " chatHistory[id] = history;\n", + " },\n", + " };\n", + "}" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cf6b3579" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Writing src/chatbotFlow.ts\n" + ] + } + ], + "source": [ + "%%writefile src/chatbotFlow.ts\n", + "\n", + "import { defineFlow, run } from '@genkit-ai/flow';\n", + "import { defineFirestoreRetriever } from '@genkit-ai/firebase';\n", + "import { retrieve } from '@genkit-ai/ai/retriever';\n", + "import { textEmbeddingGecko001 } from '@genkit-ai/googleai';\n", + "import { z } from 'zod';\n", + "\n", + "import { firestore } from './config';\n", + "import { inMemoryStore } from './memory.js';\n", + "\n", + "import { promptRef } from '@genkit-ai/dotprompt';\n", + "\n", + "// Define Firestore retriever\n", + "const retrieverRef = defineFirestoreRetriever({\n", + " name: \"merchRetriever\",\n", + " firestore,\n", + " collection: \"merch\", // Collection containing merchandise data\n", + " contentField: \"text\", // Field for product descriptions\n", + " vectorField: \"embedding\", // Field for embeddings\n", + " embedder: textEmbeddingGecko001, // Embedding model\n", + " distanceMeasure: \"COSINE\", // Similarity metric\n", + "});\n", + "\n", + "// Define the prompt reference\n", + "const assistantPrompt = promptRef('assistant');\n", + "\n", + "// To store the chat history\n", + "const historyStore = inMemoryStore();\n", + "\n", + "// Define chatbot flow\n", + "export const chatbotFlow = defineFlow(\n", + " {\n", + " name: \"chatbotFlow\",\n", + " inputSchema: z.string(),\n", + " outputSchema: z.string(),\n", + " },\n", + " async (question) => {\n", + " const conversationId = '0';\n", + "\n", + " // Retrieve conversation history.\n", + " const history = await run(\n", + " 'retrieve-history',\n", + " conversationId,\n", + " async () => {\n", + " return (await historyStore?.load(conversationId)) || [];\n", + " }\n", + " );\n", + "\n", + " // Retrieve relevant documents\n", + " const docs = await retrieve({\n", + " retriever: retrieverRef,\n", + " query: question,\n", + " options: { limit: 5 },\n", + " });\n", + "\n", + " // Run the prompt\n", + " const mainResp = await assistantPrompt.generate({\n", + " history: history,\n", + " input: {\n", + " data: docs.map((doc) => doc.content[0].text || \"\"),\n", + " question: question,\n", + " },\n", + " });\n", + "\n", + " // Save history.\n", + " await run(\n", + " 'save-history',\n", + " {\n", + " conversationId: conversationId,\n", + " history: mainResp.toHistory(),\n", + " },\n", + " async () => {\n", + " await historyStore?.save(conversationId, mainResp.toHistory());\n", + " }\n", + " );\n", + "\n", + " // Handle the response from the model API. In this sample, we just convert\n", + " // it to a string, but more complicated flows might coerce the response into\n", + " // structured output or chain the response into another LLM call, etc.\n", + " return mainResp.text();\n", + " }\n", + ");" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gpdXL-fWeLK5" + }, + "source": [ + "The retriever works with the LLM (Gemma 2 via Ollama) to create a Retrieval-Augmented Generation (RAG) flow. The retriever fetches relevant documents, which are then used by the language model to generate accurate responses, combining general knowledge with specific, relevant information from your custom data.\n", + "\n", + "The chatbotFlow consists of several key steps:\n", + "\n", + "* **Firestore Retriever**: The `retrieverRef` specifies how to fetch data from Firestore, using fields like `contentField` (product descriptions) and `vectorField` (embeddings) to locate relevant information.\n", + "\n", + "* **Prompt Reference**: The `assistantPrompt` references the prompt you created earlier using Dotprompt, determining how the assistant should format responses.\n", + "\n", + "* **Retrieve and Generate Response**: The chatbot flow retrieves relevant documents and uses them as context to generate a response. It utilizes historical context to provide a coherent and contextually relevant answer." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a7e30620" + }, + "source": [ + "Finally, wrap up the Genkit app by defining the chatbotFlow and embedFlow in `src/index.ts`. This script starts a flow server that exposes your flows as HTTP endpoints, allowing you to interact with the flows you have defined:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "00e69014" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Overwriting src/index.ts\n" + ] + } + ], + "source": [ + "%%writefile src/index.ts\n", + "\n", + "import { startFlowsServer } from '@genkit-ai/flow';\n", + "import { chatbotFlow } from './chatbotFlow';\n", + "import { embedFlow } from './embedFlow';\n", + "\n", + "// Start a flow server, which exposes your flows as HTTP endpoints. This call\n", + "// must come last, after all of your plug-in configuration and flow definitions.\n", + "// You can optionally specify a subset of flows to serve, and configure some\n", + "// HTTP server options, but by default, the flow server serves all defined flows.\n", + "startFlowsServer({\n", + " flows: [chatbotFlow, embedFlow],\n", + "});" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZsXPjWZR21KK" + }, + "source": [ + "### Start the Genkit server\n", + "\n", + "Automatically press `Enter` or `\\n` to accept the following terms.\n", + "\n", + "\n", + "> The Genkit CLI and Developer UI use cookies and similar technologies from Google to deliver and enhance the quality of its services and to analyze usage. Learn more at https://policies.google.com/technologies/cookies" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "qdT8HV46UPAD" + }, + "outputs": [], + "source": [ + "import os\n", + "from google.colab import userdata\n", + "\n", + "os.environ['GOOGLE_API_KEY'] = userdata.get('GOOGLE_API_KEY')\n", + "\n", + "command = [\n", + " \"genkit\", \"start\", \"-o\", \"--port\", \"8081\"\n", + "]\n", + "\n", + "# Create a file to write logs\n", + "with open(\"genkit.log\", \"w\") as logfile:\n", + " # Use subprocess.Popen to run the command with nohup-like behavior\n", + " genkit_process = subprocess.Popen(\n", + " command,\n", + " stdout=logfile,\n", + " stderr=subprocess.STDOUT,\n", + " stdin=subprocess.PIPE,\n", + " start_new_session=True # This is similar to nohup behavior, detaches from terminal\n", + " )\n", + " # Send an Enter key (\\n) to the process to accept the terms\n", + " genkit_process.stdin.write(b'\\n')\n", + " genkit_process.stdin.flush()\n", + "\n", + "# Sleep for 60 seconds\n", + "time.sleep(60)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fPmbRUhZ7IAZ" + }, + "source": [ + "## Expose the Genkit Tools Web API\n", + "\n", + "Use Colab's proxy to expose the server's Tools API endpoint. You can access the web interface this way in case you need to debug any issues." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "xhteLvOh7MyO" + }, + "outputs": [], + "source": [ + "# Uncomment the following code to access the web interface\n", + "\n", + "# from google.colab.output import eval_js\n", + "# proxy_url = eval_js(\"google.colab.kernel.proxyPort(8081)\")\n", + "\n", + "# print(f\"The Genkit Tools UI is accessible at: {proxy_url}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a68ccaa9" + }, + "source": [ + "## Use `embedFlow` to Index Documents\n", + "\n", + "Now, run `embedFlow` using an HTTP POST curl request to index the documents inside the `.txt` file" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ee1c9bf1" + }, + "outputs": [], + "source": [ + "!curl -X POST \"http://127.0.0.1:3400/embedFlow\" \\\n", + " -H \"Content-Type: application/json\" \\\n", + " -d '{}'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "a74e7493" + }, + "source": [ + "You should see a message indicating that the documents have been indexed successfully." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0S2F5WQg7ZX4" + }, + "source": [ + "## Use `chatbotFlow` to try the RAG out\n", + "\n", + "Finally, you can query the RAG chatbot to ask some simple questions about your data." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "AU3nUgtq7Zra" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\"result\":\"The price of the Pixel 8 Pro starts at $999. \\n\"}" + ] + } + ], + "source": [ + "!curl -X POST \"http://127.0.0.1:3400/chatbotFlow\" \\\n", + " -H \"Content-Type: application/json\" \\\n", + " -d '{\"data\": \"What is the price of the Pixel 8 Pro?\"}'" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ac1827e8" + }, + "source": [ + "## (Optional) Chat using the Gradio Chatbot Interface\n", + "\n", + "Create a simple web interface using **Gradio**.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "7e137fc4" + }, + "outputs": [], + "source": [ + "import gradio as gr\n", + "import requests\n", + "\n", + "\n", + "def chat(question, history):\n", + " try:\n", + " response = requests.post(\n", + " \"http://127.0.0.1:3400/chatbotFlow\",\n", + " headers={\"Content-Type\": \"application/json\"},\n", + " json={\"data\": question}\n", + " )\n", + "\n", + " # Check for HTTP request errors\n", + " response.raise_for_status()\n", + "\n", + " json_response = response.json()\n", + "\n", + " if 'result' in json_response:\n", + " return json_response['result']\n", + " except Exception as e:\n", + " print(f\"An unexpected error occurred: {e}\")\n", + " return \"Sorry, an unexpected error occurred.\"\n", + "\n", + "\n", + "gr.ChatInterface(\n", + " chat,\n", + " chatbot=gr.Chatbot(\n", + " show_copy_button=True,\n", + " elem_id=\"chatbot\",\n", + " render=False,\n", + " render_markdown=True,\n", + " height=300\n", + " ),\n", + " textbox=gr.Textbox(placeholder=\"Ask me a question\", container=False, scale=7),\n", + " title=\"Firebase Genkit RAG Chatbot\",\n", + " description=\"Ask any question about products.\",\n", + " theme=\"soft\",\n", + " examples=[\n", + " {\"text\": \"What is the price of the Pixel 8 Pro?\"},\n", + " {\"text\": \"Tell me about the battery life of the Samsung Galaxy Watch5 Pro.\"},\n", + " {\"text\": \"Does the Google Pixel 8 Pro support 5G connectivity?\"}\n", + " ],\n", + " show_progress=True\n", + ").launch(debug=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "98f76e0e" + }, + "source": [ + "This will generate a public URL you can use to access the chatbot interface." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "201c6035" + }, + "source": [ + "Now that your documents are indexed and the Gradio interface is running, you can start interacting with your RAG application.\n", + "\n", + "Open the Gradio interface using the URL provided and ask questions about the data, such as:\n", + "\n", + "- \"What is the price of the Pixel 8 Pro?\"\n", + "- \"Tell me about the battery life of the Samsung Galaxy Watch5 Pro.\"\n", + "- \"Does the Google Pixel 8 Pro support 5G connectivity?\"\n", + "\n", + "You should receive answers based on the data you provided in the `.txt` file." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ga6bfaC4VjHN" + }, + "source": [ + "## Cleanup\n", + "\n", + "Let's clean up everything as we've approached the end of the tutorial." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "wWnfVKqoVkSC" + }, + "outputs": [], + "source": [ + "# Terminate all processes\n", + "ollama_serve_process.terminate()\n", + "ollama_run_process.terminate()\n", + "genkit_process.terminate()\n", + "\n", + "# Delete Firebase project (press Y to confirm)\n", + "!gcloud projects delete \"$PROJECT_ID\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fba92753" + }, + "source": [ + "Congratulations! You have successfully built a RAG application using **Genkit**, **Firebase**, **Ollama**, **Gemma**, **Dotprompt**, and **Gradio**, all within a Colab notebook." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Gemma_with_Firebase_Genkit_and_Ollama.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} From a398f7097c3afa290935ab7c99e9a752a7ec6bff Mon Sep 17 00:00:00 2001 From: Kinar R Date: Mon, 28 Oct 2024 17:52:19 +0530 Subject: [PATCH 23/43] Update README and WISHLIST for the Genkit RAG notebook --- README.md | 5 +++-- WISHLIST.md | 1 - 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e91abc9..57e7a9f 100644 --- a/README.md +++ b/README.md @@ -78,13 +78,14 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Minimal_RAG.ipynb](Gemma/Minimal_RAG.ipynb) | Minimal example of building a RAG system with Gemma using [Google UniSim](https://github.com/google/unisim) and [Hugging Face](https://huggingface.co/). | | [RAG_PDF_Search_in_multiple_documents_on_Colab.ipynb](Gemma/RAG_PDF_Search_in_multiple_documents_on_Colab.ipynb) | RAG PDF Search in multiple documents using Gemma 2 2B on Google Colab. | | [Using_Gemma_with_LangChain.ipynb](Gemma/Using_Gemma_with_LangChain.ipynb) | Examples to demonstrate using Gemma with [LangChain](https://www.langchain.com/). | -| [Using_Gemma_with_Elasticsearch_and_LangChain.ipynb](Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb) | Example to demonstrate a RAG using Gemma with [Elasticsearch](https://www.elastic.co/elasticsearch/), [Ollama](https://www.ollama.com/) and [LangChain](https://www.langchain.com/). | +| [Using_Gemma_with_Elasticsearch_and_LangChain.ipynb](Gemma/Using_Gemma_with_Elasticsearch_and_LangChain.ipynb) | Example to demonstrate using Gemma with [Elasticsearch](https://www.elastic.co/elasticsearch/), [Ollama](https://www.ollama.com/) and [LangChain](https://www.langchain.com/). | +| [Gemma_with_Firebase_Genkit_and_Ollama.ipynb](Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb) | Example to demonstrate using Gemma with [Firebase Genkit](https://firebase.google.com/docs/genkit/) and [Ollama](https://www.ollama.com/) | | [Gemma_RAG_LlamaIndex.ipynb](Gemma/Gemma_RAG_LlamaIndex.ipynb) | RAG example with [LlamaIndex](https://www.llamaindex.ai/) using Gemma. | | **Finetuning** | | | [Finetune_with_Axolotl.ipynb](Gemma/Finetune_with_Axolotl.ipynb) | Finetune Gemma using [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl). | | [Finetune_with_XTuner.ipynb](Gemma/Finetune_with_XTuner.ipynb) | Finetune Gemma using [XTuner](https://github.com/InternLM/xtuner). | | [Finetune_with_LLaMA_Factory.ipynb](Gemma/Finetune_with_LLaMA_Factory.ipynb) | Finetune Gemma using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). | -| [Custom_Vocabulary.ipynb](Gemma/Custom_Vocabulary.ipynb) | Demonstrate how to use a custom vocabulary "<unused[0-98]>" tokens in Gemma. | +| [Custom_Vocabulary.ipynb](Gemma/Custom_Vocabulary.ipynb) | Demonstrate how to use a custom vocabulary "<unused[0-98]>" tokens in Gemma. | | **Alignment** | | | [Aligning_DPO_Gemma_2b_it.ipynb](Gemma/Aligning_DPO_Gemma_2b_it.ipynb) | Demonstrate how to align a Gemma model using DPO (Direct Preference Optimization) with [Hugging Face TRL](https://huggingface.co/docs/trl/en/index). | | **Evaluation** | | diff --git a/WISHLIST.md b/WISHLIST.md index 6c991ec..cef0cc9 100644 --- a/WISHLIST.md +++ b/WISHLIST.md @@ -1,7 +1,6 @@ A wish list of cookbooks showcasing: * Inference - * Integration with [Google GenKit](https://firebase.google.com/products/genkit) * [SGLang](https://github.com/sgl-project/sglang) integration * Fine-tuning From 8c6f295a09730ee61b34b01cc92a2b767dd0f664 Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 29 Oct 2024 11:46:25 +0530 Subject: [PATCH 24/43] Fix nblint warnings Update we -> you and added copyright header --- ...emma_with_Firebase_Genkit_and_Ollama.ipynb | 60 ++++++++++++------- 1 file changed, 38 insertions(+), 22 deletions(-) diff --git a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb index bfebd69..6be8a8e 100644 --- a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb +++ b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb @@ -1,5 +1,36 @@ { "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "Tce3stUlHN0L" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "tuOe1ymfHZPu" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, { "cell_type": "markdown", "metadata": { @@ -18,7 +49,7 @@ "\n", "[**Ollama**](https://ollama.ai/) is a tool that simplifies running language models locally. It allows you to manage and serve multiple models efficiently, making it easier to deploy and test AI models on your machine. With Ollama, you can switch between different models and versions seamlessly, providing flexibility in development and experimentation.\n", "\n", - "[**Firebase**](https://firebase.google.com/) is a comprehensive app development platform by Google that provides services like real-time databases, authentication, cloud storage, hosting, and machine learning. In this tutorial, we will utilize the **Cloud Firestore**, a scalable, flexible NoSQL cloud database to store and sync data for client- and server-side development.\n", + "[**Firebase**](https://firebase.google.com/) is a comprehensive app development platform by Google that provides services like real-time databases, authentication, cloud storage, hosting, and machine learning. In this tutorial, you will utilize the **Cloud Firestore**, a scalable, flexible NoSQL cloud database to store and sync data for client- and server-side development.\n", "\n", "[**Gradio**](https://gradio.app/) is an open-source Python library for creating user-friendly web interfaces to interact with machine learning models. It allows you to quickly create customizable UI components to interact with your models and also generate shareable web apps that anyone can use.\n", "\n", @@ -387,7 +418,7 @@ "source": [ "## Set up the Firebase project\n", "\n", - "Firebase will be used to store and manage your data. In this case, we'll use Cloud Firestore, a NoSQL database that makes it easy to store and retrieve the information that will be used by the chatbot.\n", + "Firebase will be used to store and manage your data. In this case, you'll use Cloud Firestore, a NoSQL database that makes it easy to store and retrieve the information that will be used by the chatbot.\n", "\n", "Before you can continue, you need to set up a Firebase project:\n", "\n", @@ -472,7 +503,7 @@ "* Navigate to the **Cloud Firestore** section of the [Firebase console](https://console.firebase.google.com/project/_/firestore). You'll be prompted to select an existing Firebase project. Follow the database creation workflow.\n", "\"Service\n", "\n", - "* To simplify our demo, let's select a starting mode for your **Cloud Firestore Security Rules**. We'll pick **Test mode** to get quickly started.\n", + "* To simplify our demo, let's select a starting mode for your **Cloud Firestore Security Rules**. You'll pick **Test mode** to get quickly started.\n", "\n", "* Pick a location and this setting will be your project's default Google Cloud Platform (GCP) resource location.\n", "\n", @@ -651,22 +682,7 @@ "metadata": { "id": "d8e38328" }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[?25l\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‡\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ \u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ‹\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ™\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ น\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ธ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ผ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ด\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ฆ\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[36mโ ง\u001b[39m Installing NPM packages\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully installed NPM packages\n", - "\u001b[?25l\u001b[36mโ ‹\u001b[39m Updating tsconfig.json\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully updated tsconfig.json\n", - "\u001b[?25l\u001b[36mโ ‹\u001b[39m Updating package.json\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully updated package.json\n", - "\u001b[?25l\u001b[36mโ ‹\u001b[39m Generating sample file\u001b[2K\u001b[1G\u001b[?25h\u001b[32mโœ”\u001b[39m Successfully generated sample file (src/index.ts)\n", - "If you don't have Ollama already installed and configured, refer to https://developers.google.com/genkit/plugins/ollama\n", - "\n", - "Genkit successfully initialized.\n", - "\u001b[?25h" - ] - } - ], + "outputs": [], "source": [ "!genkit init --model ollama --non-interactive" ] @@ -974,7 +990,7 @@ "id": "rwO8RRmDdJmw" }, "source": [ - "By storing both the text and its embedding, we can later perform similarity searches to find relevant product descriptions based on user queries. This makes it possible for the chatbot to retrieve and provide accurate, contextually relevant answers." + "By storing both the text and its embedding, you can later perform similarity searches to find relevant product descriptions based on user queries. This makes it possible for the chatbot to retrieve and provide accurate, contextually relevant answers." ] }, { @@ -1030,7 +1046,7 @@ " googleAI({ apiVersion: ['v1', 'v1beta'] }),\n", " ollama({\n", " // Ollama provides an interface to many generative models. Here,\n", - " // we specify Google's Gemma 2 model. The models you specify must already\n", + " // you specify Google's Gemma 2 model. The models you specify must already\n", " // be downloaded and available to the Ollama server.\n", " models: [{ name: 'gemma2:2b' }],\n", " // The address of your Ollama API server. This is often a different host\n", @@ -1498,7 +1514,7 @@ "source": [ "## Cleanup\n", "\n", - "Let's clean up everything as we've approached the end of the tutorial." + "Let's clean up everything as you've approached the end of the tutorial." ] }, { From 1511542f6a571c0bbcd2864a538f9acb166ab5bc Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 29 Oct 2024 11:56:13 +0530 Subject: [PATCH 25/43] Fix spacing to push the screenshot to the next line --- Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb index 6be8a8e..202c535 100644 --- a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb +++ b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb @@ -501,7 +501,7 @@ "Now that you've set up Firebase and your development environment, let's move on to creating the Firestore database.\n", "\n", "* Navigate to the **Cloud Firestore** section of the [Firebase console](https://console.firebase.google.com/project/_/firestore). You'll be prompted to select an existing Firebase project. Follow the database creation workflow.\n", - "\"Service\n", + " \"Service\n", "\n", "* To simplify our demo, let's select a starting mode for your **Cloud Firestore Security Rules**. You'll pick **Test mode** to get quickly started.\n", "\n", From 2b80fbbf12363598a23c86fd4fc60c366c863562 Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 29 Oct 2024 12:01:58 +0530 Subject: [PATCH 26/43] Fixed image from appearing inline --- Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb index 202c535..84057a9 100644 --- a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb +++ b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb @@ -500,8 +500,8 @@ "\n", "Now that you've set up Firebase and your development environment, let's move on to creating the Firestore database.\n", "\n", - "* Navigate to the **Cloud Firestore** section of the [Firebase console](https://console.firebase.google.com/project/_/firestore). You'll be prompted to select an existing Firebase project. Follow the database creation workflow.\n", - " \"Service\n", + "* Navigate to the **Cloud Firestore** section of the [Firebase console](https://console.firebase.google.com/project/_/firestore). You'll be prompted to select an existing Firebase project. Follow the database creation workflow. \n", + "\"Service\n", "\n", "* To simplify our demo, let's select a starting mode for your **Cloud Firestore Security Rules**. You'll pick **Test mode** to get quickly started.\n", "\n", From 2557d9615791c6ea83f6ae081fca84e2be8d8940 Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 29 Oct 2024 12:26:29 +0530 Subject: [PATCH 27/43] Remove irrelevant comment in text cell --- Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb index 84057a9..d0a08f7 100644 --- a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb +++ b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb @@ -126,9 +126,7 @@ "2. **Add Google API Key**:\n", " - Create a new secret named `GOOGLE_API_KEY`.\n", " - Paste your Google API Key into the Value input box.\n", - " - Toggle the button to allow notebook access to the secret.\n", - "\n", - "Now, set the environment variables in your notebook:" + " - Toggle the button to allow notebook access to the secret." ] }, { From 43ebaf1f587ebdfb145ecf336da387dc24e46ae8 Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 29 Oct 2024 12:32:02 +0530 Subject: [PATCH 28/43] Shortened notebook main title and added link to the Genkit's docs in the intro --- Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb index d0a08f7..19d4c1c 100644 --- a/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb +++ b/Gemma/Gemma_with_Firebase_Genkit_and_Ollama.ipynb @@ -37,13 +37,13 @@ "id": "c4b7e9a1" }, "source": [ - "# Building a RAG Application with Genkit, Firebase, Ollama, Gemma, Dotprompt, and Gradio\n", + "# Building a RAG Application with Firebase Genkit, Ollama and Gemma\n", "\n", "\n", "\n", "In this comprehensive tutorial, you will learn how to build a **Retrieval-Augmented Generation (RAG)** application using cutting-edge technologies:\n", "\n", - "[**Genkit**](#) is a framework designed to help you build AI-powered applications and features. It provides open-source libraries for Node.js and Go, plus developer tools for testing and debugging.\n", + "[**Genkit**](https://firebase.google.com/docs/genkit) is a framework designed to help you build AI-powered applications and features. It provides open-source libraries for Node.js and Go, plus developer tools for testing and debugging.\n", "\n", "[**Gemma**](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs) available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n", "\n", From 50806d998654cc68375adb3a5cf0935eb87838cd Mon Sep 17 00:00:00 2001 From: Lionel Date: Wed, 30 Oct 2024 13:44:21 +0100 Subject: [PATCH 29/43] Fix 'g' suffix in requirements.txt package names --- .../business-email-assistant/requirements.txt | 128 +++++++++--------- 1 file changed, 64 insertions(+), 64 deletions(-) diff --git a/Gemma/business-email-assistant/requirements.txt b/Gemma/business-email-assistant/requirements.txt index 1212ee6..ce5edc3 100644 --- a/Gemma/business-email-assistant/requirements.txt +++ b/Gemma/business-email-assistant/requirements.txt @@ -1,64 +1,64 @@ -absl-pyg -astunparseg -blinkerg -certifig -charset-normalizerg -clickg -flaskg -flatbuffersg -gastg -google-pastag -grpciog -h5pyg -idnag -importlib-metadatag -itsdangerousg -jaxg -jax-cuda12-pjrtg -jax-cuda12-pluging -jaxlibg -jinja2g -kagglehubg -kerasg -keras-nlpg -libclangg -Markdowng -markdown-it-pyg -MarkupSafeg -mdurlg -ml-dtypesg -namexg -numpyg -nvidia-cublas-cu12g -nvidia-cuda-cupti-cu12g -nvidia-cuda-nvcc-cu12g -nvidia-cuda-runtime-cu12g -nvidia-cudnn-cu12g -nvidia-cufft-cu12g -nvidia-cusolver-cu12g -nvidia-cusparse-cu12g -nvidia-nccl-cu12g -nvidia-nvjitlink-cu12g -opt-einsumg -optreeg -packagingg -protobufg -pygmentsg -python-dotenvg -regexg -requestsg -richg -scipyg -sixg -tensorboardg -tensorboard-data-serverg -tensorflowg -tensorflow-io-gcs-filesystemg -tensorflow-textg -termcolorg -tqdmg -typing-extensionsg -urllib3g -werkzeugg -wraptg -zippg +absl-py +astunparse +blinker +certifi +charset-normalizer +click +flask +flatbuffers +gast +google-pasta +grpcio +h5py +idna +importlib-metadata +itsdangerous +jax +jax-cuda12-pjrt +jax-cuda12-plugin +jaxlib +jinja2 +kagglehub +keras +keras-nlp +libclang +Markdown +markdown-it-py +MarkupSafe +mdurl +ml-dtypes +namex +numpy +nvidia-cublas-cu12 +nvidia-cuda-cupti-cu12 +nvidia-cuda-nvcc-cu12 +nvidia-cuda-runtime-cu12 +nvidia-cudnn-cu12 +nvidia-cufft-cu12 +nvidia-cusolver-cu12 +nvidia-cusparse-cu12 +nvidia-nccl-cu12 +nvidia-nvjitlink-cu12 +opt-einsum +optree +packaging +protobuf +pygments +python-dotenv +regex +requests +rich +scipy +six +tensorboard +tensorboard-data-server +tensorflow +tensorflow-io-gcs-filesystem +tensorflow-text +termcolor +tqdm +typing-extensions +urllib3 +werkzeug +wrapt +zipp From 12c15d26a9b54d57bc2d2341199fb2713759b85a Mon Sep 17 00:00:00 2001 From: Lionel La Rocca <45712054+LionelLR@users.noreply.github.com> Date: Wed, 30 Oct 2024 13:47:53 +0100 Subject: [PATCH 30/43] Fix 'g' suffix in requirements.txt package names --- .../gemma-web-service/requirements.txt | 146 +++++++++--------- 1 file changed, 73 insertions(+), 73 deletions(-) diff --git a/Gemma/personal-code-assistant/gemma-web-service/requirements.txt b/Gemma/personal-code-assistant/gemma-web-service/requirements.txt index 0d09074..7ed6f6d 100644 --- a/Gemma/personal-code-assistant/gemma-web-service/requirements.txt +++ b/Gemma/personal-code-assistant/gemma-web-service/requirements.txt @@ -1,73 +1,73 @@ -absl-pyg -annotated-typesg -anyiog -astunparseg -certifig -charset-normalizerg -clickg -exceptiongroupg -fastapig -flatbuffersg -gastg -google-pastag -grpciog -h11g -h5pyg -httptoolsg -idnag -jaxg -jax-cuda12-pjrtg -jax-cuda12-pluging -jaxlibg -kagglehubg -kerasg -keras-nlpg -libclangg -Markdowng -markdown-it-pyg -MarkupSafeg -mdurlg -ml-dtypesg -namexg -numpyg -nvidia-cublas-cu12g -nvidia-cuda-cupti-cu12g -nvidia-cuda-nvcc-cu12g -nvidia-cuda-runtime-cu12g -nvidia-cudnn-cu12g -nvidia-cufft-cu12g -nvidia-cusolver-cu12g -nvidia-cusparse-cu12g -nvidia-nccl-cu12g -nvidia-nvjitlink-cu12g -opt-einsumg -optreeg -packagingg -protobufg -pydanticg -pydantic_coreg -Pygmentsg -python-dotenvg -PyYAMLg -regexg -requestsg -richg -scipyg -sixg -sniffiog -starletteg -tensorboardg -tensorboard-data-serverg -tensorflowg -tensorflow-io-gcs-filesystemg -tensorflow-textg -termcolorg -tqdmg -typing_extensionsg -urllib3g -uvicorng -uvloopg -watchfilesg -websocketsg -Werkzeugg -wraptg +absl-py +annotated-types +anyio +astunparse +certifi +charset-normalizer +click +exceptiongroup +fastapi +flatbuffers +gast +google-pasta +grpcio +h11 +h5py +httptools +idna +jax +jax-cuda12-pjrt +jax-cuda12-plugin +jaxlib +kagglehub +keras +keras-nlp +libclang +Markdown +markdown-it-py +MarkupSafe +mdurl +ml-dtypes +namex +numpy +nvidia-cublas-cu12 +nvidia-cuda-cupti-cu12 +nvidia-cuda-nvcc-cu12 +nvidia-cuda-runtime-cu12 +nvidia-cudnn-cu12 +nvidia-cufft-cu12 +nvidia-cusolver-cu12 +nvidia-cusparse-cu12 +nvidia-nccl-cu12 +nvidia-nvjitlink-cu12 +opt-einsum +optree +packaging +protobuf +pydantic +pydantic_core +Pygments +python-dotenv +PyYAML +regex +requests +rich +scipy +six +sniffio +starlette +tensorboard +tensorboard-data-server +tensorflow +tensorflow-io-gcs-filesystem +tensorflow-text +termcolor +tqdm +typing_extensions +urllib3 +uvicorn +uvloop +watchfiles +websockets +Werkzeug +wrapt From ffb02853b2aa208e9dc6349621ef4f30d8a6fa64 Mon Sep 17 00:00:00 2001 From: Lionel La Rocca <45712054+LionelLR@users.noreply.github.com> Date: Wed, 30 Oct 2024 13:49:48 +0100 Subject: [PATCH 31/43] Fix 'g' suffix in requirements.txt package names --- Gemma/spoken-language-tasks/requirements.txt | 226 +++++++++---------- 1 file changed, 113 insertions(+), 113 deletions(-) diff --git a/Gemma/spoken-language-tasks/requirements.txt b/Gemma/spoken-language-tasks/requirements.txt index a793c9a..f639059 100644 --- a/Gemma/spoken-language-tasks/requirements.txt +++ b/Gemma/spoken-language-tasks/requirements.txt @@ -1,113 +1,113 @@ -absl-pyg -aiohappyeyeballsg -aiohttpg -aiosignalg -annotated-typesg -astunparseg -async-timeoutg -attrsg -bleachg -blinkerg -cachetoolsg -certifig -charset-normalizerg -clickg -contourpyg -cyclerg -datasetsg -dillg -filelockg -flaskg -flatbuffersg -fonttoolsg -frozenlistg -fsspecg -gastg -google-ai-generativelanguageg -google-api-coreg -google-api-python-clientg -google-authg -google-auth-httplib2g -google-generativeaig -google-pastag -googleapis-common-protosg -grpciog -grpcio-statusg -h5pyg -httplib2g -huggingface-hubg -idnag -importlib-metadatag -importlib-resourcesg -itsdangerousg -jaxg -jax-cuda12-pjrtg -jax-cuda12-pluging -jaxlibg -jinja2g -kagglehubg -kerasg -keras-nlpg -kiwisolverg -libclangg -Markdowng -markdown-it-pyg -MarkupSafeg -matplotlibg -mdurlg -ml-dtypesg -multidictg -multiprocessg -namexg -numpyg -nvidia-cublas-cu12g -nvidia-cuda-cupti-cu12g -nvidia-cuda-nvcc-cu12g -nvidia-cuda-runtime-cu12g -nvidia-cudnn-cu12g -nvidia-cufft-cu12g -nvidia-cusolver-cu12g -nvidia-cusparse-cu12g -nvidia-nccl-cu12g -nvidia-nvjitlink-cu12g -opt-einsumg -optreeg -packagingg -pandasg -pillowg -proto-plusg -protobufg -pyarrowg -pyasn1g -pyasn1-modulesg -pydanticg -pydantic-coreg -pygmentsg -pyparsingg -python-dateutilg -python-dotenvg -pytzg -PyYAMLg -regexg -requestsg -richg -rsag -scipyg -sixg -tensorboardg -tensorboard-data-serverg -tensorflowg -tensorflow-io-gcs-filesystemg -tensorflow-textg -termcolorg -tqdmg -typing-extensionsg -tzdatag -uritemplateg -urllib3g -webencodingsg -werkzeugg -wraptg -xxhashg -yarlg -zippg +absl-py +aiohappyeyeballs +aiohttp +aiosignal +annotated-types +astunparse +async-timeout +attrs +bleach +blinker +cachetools +certifi +charset-normalizer +click +contourpy +cycler +datasets +dill +filelock +flask +flatbuffers +fonttools +frozenlist +fsspec +gast +google-ai-generativelanguage +google-api-core +google-api-python-client +google-auth +google-auth-httplib2 +google-generativeai +google-pasta +googleapis-common-protos +grpcio +grpcio-status +h5py +httplib2 +huggingface-hub +idna +importlib-metadata +importlib-resources +itsdangerous +jax +jax-cuda12-pjrt +jax-cuda12-plugin +jaxlib +jinja2 +kagglehub +keras +keras-nlp +kiwisolver +libclang +Markdown +markdown-it-py +MarkupSafe +matplotlib +mdurl +ml-dtypes +multidict +multiprocess +namex +numpy +nvidia-cublas-cu12 +nvidia-cuda-cupti-cu12 +nvidia-cuda-nvcc-cu12 +nvidia-cuda-runtime-cu12 +nvidia-cudnn-cu12 +nvidia-cufft-cu12 +nvidia-cusolver-cu12 +nvidia-cusparse-cu12 +nvidia-nccl-cu12 +nvidia-nvjitlink-cu12 +opt-einsum +optree +packaging +pandas +pillow +proto-plus +protobuf +pyarrow +pyasn1 +pyasn1-modules +pydantic +pydantic-core +pygments +pyparsing +python-dateutil +python-dotenv +pytz +PyYAML +regex +requests +rich +rsa +scipy +six +tensorboard +tensorboard-data-server +tensorflow +tensorflow-io-gcs-filesystem +tensorflow-text +termcolor +tqdm +typing-extensions +tzdata +uritemplate +urllib3 +webencodings +werkzeug +wrapt +xxhash +yarl +zipp From 7514a3764d4f51abe869801e6e53db95a26f4106 Mon Sep 17 00:00:00 2001 From: Wei Wei <5577741+windmaple@users.noreply.github.com> Date: Tue, 5 Nov 2024 14:24:33 +0800 Subject: [PATCH 32/43] Update setup info on evaluation notebook --- Gemma/Gemma_evaluation.ipynb | 886 +++++++++++++++++------------------ 1 file changed, 441 insertions(+), 445 deletions(-) diff --git a/Gemma/Gemma_evaluation.ipynb b/Gemma/Gemma_evaluation.ipynb index bd6ce8f..8448089 100644 --- a/Gemma/Gemma_evaluation.ipynb +++ b/Gemma/Gemma_evaluation.ipynb @@ -1,455 +1,451 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "tf845nO8_-hV" - }, - "source": [ - "##### Copyright 2024 Google LLC." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "cellView": "form", - "id": "EEczi3q7AOXD" - }, - "outputs": [], - "source": [ - "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", - "# you may not use this file except in compliance with the License.\n", - "# You may obtain a copy of the License at\n", - "#\n", - "# https://www.apache.org/licenses/LICENSE-2.0\n", - "#\n", - "# Unless required by applicable law or agreed to in writing, software\n", - "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", - "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", - "# See the License for the specific language governing permissions and\n", - "# limitations under the License." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7InkT1mdAJhT" - }, - "source": [ - "# Gemma - Model evaluation\n", - "\n", - "This notebook demonstrates using EleautherAI's Language Model Evaluation Harness to perform model performance benchmark on Gemma2 2B, specifically using a subset of MMLU.\n", - "\n", - " \n", - "
\n", - " Run in Google Colab\n", - "
" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "SKQ4bH7qMGrA" - }, - "source": [ - "## Setup\n", - "\n", - "### Select the Colab runtime\n", - "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n", - "\n", - "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", - "2. Select **Change runtime type**.\n", - "3. Under **Hardware accelerator**, select **T4 GPU**.\n", - "\n", - "### Gemma setup\n", - "\n", - "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n", - "\n", - "* Get access to Gemma on kaggle.com.\n", - "* Select a Colab runtime with sufficient resources to run\n", - " the Gemma 2B model.\n", - "* Generate and configure a Kaggle username and an API key as Colab secrets.\n", - "\n", - "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KJintwsi1o87" - }, - "source": [ - "### Configure your credentials\n", - "\n", - "Add your your Kaggle credentials to the Colab Secrets manager to securely store it.\n", - "\n", - "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", - "2. Create new secrets: `KAGGLE_USERNAME` and `KAGGLE_KEY`\n", - "3. Copy/paste your username into `KAGGLE_USERNAME`\n", - "3. Copy/paste your key into `KAGGLE_KEY`\n", - "4. Toggle the buttons on the left to allow notebook access to the secrets." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ATbyLmuImHTA" - }, - "outputs": [], - "source": [ - "import os\n", - "import kagglehub\n", - "from google.colab import userdata\n", - "\n", - "os.environ[\"KAGGLE_USERNAME\"] = userdata.get('KAGGLE_USERNAME')\n", - "os.environ[\"KAGGLE_KEY\"] = userdata.get('KAGGLE_KEY')\n", - "\n", - "# Pre-allocate 90% of accelerator memory\n", - "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"0.9\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KxDcs0t1A7zX" - }, - "source": [ - "Install the evaluation harness." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Bzim2HVVbuy-" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Collecting git+https://github.com/EleutherAI/lm-evaluation-harness.git\n", - " Cloning https://github.com/EleutherAI/lm-evaluation-harness.git to /tmp/pip-req-build-1s_z1sm5\n", - " Running command git clone --filter=blob:none --quiet https://github.com/EleutherAI/lm-evaluation-harness.git /tmp/pip-req-build-1s_z1sm5\n", - " Resolved https://github.com/EleutherAI/lm-evaluation-harness.git to commit 543617fef9ba885e87f8db8930fbbff1d4e2ca49\n", - " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", - " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", - " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", - "Requirement already satisfied: accelerate>=0.26.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.33.0)\n", - "Requirement already satisfied: evaluate in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.4.2)\n", - "Requirement already satisfied: datasets>=2.16.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.21.0)\n", - "Requirement already satisfied: jsonlines in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (4.0.0)\n", - "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.10.1)\n", - "Requirement already satisfied: peft>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.12.0)\n", - "Requirement already satisfied: pybind11>=2.6.2 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.13.5)\n", - "Requirement already satisfied: pytablewriter in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (1.2.0)\n", - "Requirement already satisfied: rouge-score>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.1.2)\n", - "Requirement already satisfied: sacrebleu>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.4.3)\n", - "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (1.3.2)\n", - "Requirement already satisfied: sqlitedict in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.1.0)\n", - "Requirement already satisfied: torch>=1.8 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.4.0+cu121)\n", - "Requirement already satisfied: tqdm-multiprocess in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.0.11)\n", - "Requirement already satisfied: transformers>=4.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (4.44.2)\n", - "Requirement already satisfied: zstandard in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.23.0)\n", - "Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.3.8)\n", - "Requirement already satisfied: word2number in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (1.1)\n", - "Requirement already satisfied: more-itertools in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (10.3.0)\n", - "Requirement already satisfied: numpy<2.0.0,>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (1.26.4)\n", - "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (24.1)\n", - "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (5.9.5)\n", - "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (6.0.2)\n", - "Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (0.24.6)\n", - "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (0.4.4)\n", - "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (3.15.4)\n", - "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (17.0.0)\n", - "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (2.1.4)\n", - "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (2.32.3)\n", - "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (4.66.5)\n", - "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (3.5.0)\n", - "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (0.70.16)\n", - "Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets>=2.16.0->lm_eval==0.4.4) (2024.6.1)\n", - "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (3.10.5)\n", - "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.4.4) (2.1.0)\n", - "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.4.4) (3.8.1)\n", - "Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.4.4) (1.16.0)\n", - "Requirement already satisfied: portalocker in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (2.10.1)\n", - "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (2024.5.15)\n", - "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (0.9.0)\n", - "Requirement already satisfied: colorama in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (0.4.6)\n", - "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (4.9.4)\n", - "Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.4.4) (1.13.1)\n", - "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.4.4) (1.4.2)\n", - "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.4.4) (3.5.0)\n", - "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (4.12.2)\n", - "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (1.13.2)\n", - "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (3.3)\n", - "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (3.1.4)\n", - "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.1->lm_eval==0.4.4) (0.19.1)\n", - "Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines->lm_eval==0.4.4) (24.2.0)\n", - "Requirement already satisfied: setuptools>=38.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (71.0.4)\n", - "Requirement already satisfied: DataProperty<2,>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (1.0.1)\n", - "Requirement already satisfied: mbstrdecoder<2,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (1.1.3)\n", - "Requirement already satisfied: pathvalidate<4,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (3.2.1)\n", - "Requirement already satisfied: tabledata<2,>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (1.3.3)\n", - "Requirement already satisfied: tcolorpy<1,>=0.0.5 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (0.1.6)\n", - "Requirement already satisfied: typepy<2,>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.4.4) (1.3.2)\n", - "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (2.4.0)\n", - "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (1.3.1)\n", - "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (1.4.1)\n", - "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (6.0.5)\n", - "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (1.9.4)\n", - "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (4.0.3)\n", - "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm_eval==0.4.4) (5.2.0)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (3.3.2)\n", - "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (3.8)\n", - "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (2.0.7)\n", - "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (2024.8.30)\n", - "Requirement already satisfied: python-dateutil<3.0.0,>=2.8.0 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.4.4) (2.8.2)\n", - "Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.4.4) (2024.1)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.8->lm_eval==0.4.4) (2.1.5)\n", - "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score>=0.0.4->lm_eval==0.4.4) (8.1.7)\n", - "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets>=2.16.0->lm_eval==0.4.4) (2024.1)\n", - "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.8->lm_eval==0.4.4) (1.3.0)\n" - ] - } - ], - "source": [ - "!pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hzvwo9Is7mvX" - }, - "source": [ - "Install the Gemma JAX library." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "5EUxBOYImMc1" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", - " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", - " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n" - ] - } - ], - "source": [ - "!pip install -q git+https://github.com/google-deepmind/gemma.git\n", - "from gemma import params as params_lib\n", - "import sentencepiece as spm\n", - "from gemma import transformer as transformer_lib\n", - "from gemma import sampler as sampler_lib" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KNpy8QsB8aTD" - }, - "source": [ - "Download the Gemma model and tokenizer." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ohqi-FOqmRA8" - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Warning: Looks like you're using an outdated `kagglehub` version, please consider updating (latest version: 0.3.0)\n" - ] - } - ], - "source": [ - "GEMMA_VARIANT = 'gemma2-2b-it'\n", - "GEMMA_PATH = kagglehub.model_download(f'google/gemma-2/flax/{GEMMA_VARIANT}')\n", - "CKPT_PATH = os.path.join(GEMMA_PATH, GEMMA_VARIANT)\n", - "TOKENIZER_PATH = os.path.join(GEMMA_PATH, 'tokenizer.model')" - ] - }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "tf845nO8_-hV" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "cellView": "form", + "id": "EEczi3q7AOXD" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7InkT1mdAJhT" + }, + "source": [ + "# Gemma - Model evaluation\n", + "\n", + "This notebook demonstrates using EleautherAI's Language Model Evaluation Harness to perform model performance benchmark on Gemma2 2B, specifically using a subset of MMLU.\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SKQ4bH7qMGrA" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use the T4 GPU but you need a high RAM instance (due to [this](https://github.com/google-deepmind/gemma/issues/57)).\n", + "\n", + "### Gemma setup\n", + "\n", + "To complete this tutorial, you'll first need to complete the setup instructions at [Gemma setup](https://ai.google.dev/gemma/docs/setup). The Gemma setup instructions show you how to do the following:\n", + "\n", + "* Get access to Gemma on kaggle.com.\n", + "* Select a Colab runtime with sufficient resources to run\n", + " the Gemma 2B model.\n", + "* Generate and configure a Kaggle username and an API key as Colab secrets.\n", + "\n", + "After you've completed the Gemma setup, move on to the next section, where you'll set environment variables for your Colab environment." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KJintwsi1o87" + }, + "source": [ + "### Configure your credentials\n", + "\n", + "Add your your Kaggle credentials to the Colab Secrets manager to securely store it.\n", + "\n", + "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + "2. Create new secrets: `KAGGLE_USERNAME` and `KAGGLE_KEY`\n", + "3. Copy/paste your username into `KAGGLE_USERNAME`\n", + "3. Copy/paste your key into `KAGGLE_KEY`\n", + "4. Toggle the buttons on the left to allow notebook access to the secrets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ATbyLmuImHTA" + }, + "outputs": [], + "source": [ + "import os\n", + "import kagglehub\n", + "from google.colab import userdata\n", + "\n", + "os.environ[\"KAGGLE_USERNAME\"] = userdata.get('KAGGLE_USERNAME')\n", + "os.environ[\"KAGGLE_KEY\"] = userdata.get('KAGGLE_KEY')\n", + "\n", + "# Pre-allocate 90% of accelerator memory\n", + "os.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"] = \"0.9\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KxDcs0t1A7zX" + }, + "source": [ + "Install the evaluation harness." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "Bzim2HVVbuy-" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8NM7ALv5E9sh" - }, - "outputs": [], - "source": [ - "params = params_lib.load_and_format_params(CKPT_PATH)\n", - "vocab = spm.SentencePieceProcessor()\n", - "vocab.Load(TOKENIZER_PATH)\n", - "\n", - "transformer_config = transformer_lib.TransformerConfig.from_params(\n", - " params=params,\n", - " cache_size=1024\n", - ")\n", - "\n", - "transformer = transformer_lib.Transformer(transformer_config)\n", - "\n", - "sampler = sampler_lib.Sampler(\n", - " transformer=transformer,\n", - " vocab=vocab,\n", - " params=params['transformer'],\n", - ")" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting git+https://github.com/EleutherAI/lm-evaluation-harness.git\n", + " Cloning https://github.com/EleutherAI/lm-evaluation-harness.git to /tmp/pip-req-build-1s_z1sm5\n", + " Running command git clone --filter=blob:none --quiet https://github.com/EleutherAI/lm-evaluation-harness.git /tmp/pip-req-build-1s_z1sm5\n", + " Resolved https://github.com/EleutherAI/lm-evaluation-harness.git to commit 543617fef9ba885e87f8db8930fbbff1d4e2ca49\n", + " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", + "Requirement already satisfied: accelerate>=0.26.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.33.0)\n", + "Requirement already satisfied: evaluate in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.4.2)\n", + "Requirement already satisfied: datasets>=2.16.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.21.0)\n", + "Requirement already satisfied: jsonlines in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (4.0.0)\n", + "Requirement already satisfied: numexpr in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.10.1)\n", + "Requirement already satisfied: peft>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.12.0)\n", + "Requirement already satisfied: pybind11>=2.6.2 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.13.5)\n", + "Requirement already satisfied: pytablewriter in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (1.2.0)\n", + "Requirement already satisfied: rouge-score>=0.0.4 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.1.2)\n", + "Requirement already satisfied: sacrebleu>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.4.3)\n", + "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (1.3.2)\n", + "Requirement already satisfied: sqlitedict in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.1.0)\n", + "Requirement already satisfied: torch>=1.8 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (2.4.0+cu121)\n", + "Requirement already satisfied: tqdm-multiprocess in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.0.11)\n", + "Requirement already satisfied: transformers>=4.1 in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (4.44.2)\n", + "Requirement already satisfied: zstandard in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.23.0)\n", + "Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (0.3.8)\n", + "Requirement already satisfied: word2number in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (1.1)\n", + "Requirement already satisfied: more-itertools in /usr/local/lib/python3.10/dist-packages (from lm_eval==0.4.4) (10.3.0)\n", + "Requirement already satisfied: numpy<2.0.0,>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (1.26.4)\n", + "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (24.1)\n", + "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (5.9.5)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (6.0.2)\n", + "Requirement already satisfied: huggingface-hub>=0.21.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (0.24.6)\n", + "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.26.0->lm_eval==0.4.4) (0.4.4)\n", + "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (3.15.4)\n", + "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (17.0.0)\n", + "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (2.1.4)\n", + "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (2.32.3)\n", + "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (4.66.5)\n", + "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (3.5.0)\n", + "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (0.70.16)\n", + "Requirement already satisfied: fsspec<=2024.6.1,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.6.1,>=2023.1.0->datasets>=2.16.0->lm_eval==0.4.4) (2024.6.1)\n", + "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=2.16.0->lm_eval==0.4.4) (3.10.5)\n", + "Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.4.4) (2.1.0)\n", + "Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.4.4) (3.8.1)\n", + "Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from rouge-score>=0.0.4->lm_eval==0.4.4) (1.16.0)\n", + "Requirement already satisfied: portalocker in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (2.10.1)\n", + "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (2024.5.15)\n", + "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (0.9.0)\n", + "Requirement already satisfied: colorama in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (0.4.6)\n", + "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from sacrebleu>=1.5.0->lm_eval==0.4.4) (4.9.4)\n", + "Requirement already satisfied: scipy>=1.5.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.4.4) (1.13.1)\n", + "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.4.4) (1.4.2)\n", + "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.24.1->lm_eval==0.4.4) (3.5.0)\n", + "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (4.12.2)\n", + "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (1.13.2)\n", + "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (3.3)\n", + "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.8->lm_eval==0.4.4) (3.1.4)\n", + "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.1->lm_eval==0.4.4) (0.19.1)\n", + "Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines->lm_eval==0.4.4) (24.2.0)\n", + "Requirement already satisfied: setuptools>=38.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (71.0.4)\n", + "Requirement already satisfied: DataProperty<2,>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (1.0.1)\n", + "Requirement already satisfied: mbstrdecoder<2,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (1.1.3)\n", + "Requirement already satisfied: pathvalidate<4,>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (3.2.1)\n", + "Requirement already satisfied: tabledata<2,>=1.3.1 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (1.3.3)\n", + "Requirement already satisfied: tcolorpy<1,>=0.0.5 in /usr/local/lib/python3.10/dist-packages (from pytablewriter->lm_eval==0.4.4) (0.1.6)\n", + "Requirement already satisfied: typepy<2,>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.4.4) (1.3.2)\n", + "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (2.4.0)\n", + "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (1.3.1)\n", + "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (1.4.1)\n", + "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (6.0.5)\n", + "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (1.9.4)\n", + "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.16.0->lm_eval==0.4.4) (4.0.3)\n", + "Requirement already satisfied: chardet<6,>=3.0.4 in /usr/local/lib/python3.10/dist-packages (from mbstrdecoder<2,>=1.0.0->pytablewriter->lm_eval==0.4.4) (5.2.0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (3.3.2)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (3.8)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (2.0.7)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets>=2.16.0->lm_eval==0.4.4) (2024.8.30)\n", + "Requirement already satisfied: python-dateutil<3.0.0,>=2.8.0 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.4.4) (2.8.2)\n", + "Requirement already satisfied: pytz>=2018.9 in /usr/local/lib/python3.10/dist-packages (from typepy[datetime]<2,>=1.3.2->pytablewriter->lm_eval==0.4.4) (2024.1)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.8->lm_eval==0.4.4) (2.1.5)\n", + "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk->rouge-score>=0.0.4->lm_eval==0.4.4) (8.1.7)\n", + "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets>=2.16.0->lm_eval==0.4.4) (2024.1)\n", + "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.8->lm_eval==0.4.4) (1.3.0)\n" + ] + } + ], + "source": [ + "!pip install git+https://github.com/EleutherAI/lm-evaluation-harness.git" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hzvwo9Is7mvX" + }, + "source": [ + "Install the Gemma JAX library." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "5EUxBOYImMc1" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "O4icXT5G9UEZ" - }, - "source": [ - "Create a new Gemma LM calss, following the [New Model Guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md). For MMLU, we only need to implement the loglikelihood() function." - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + " Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ], + "source": [ + "!pip install -q git+https://github.com/google-deepmind/gemma.git\n", + "from gemma import params as params_lib\n", + "import sentencepiece as spm\n", + "from gemma import transformer as transformer_lib\n", + "from gemma import sampler as sampler_lib" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KNpy8QsB8aTD" + }, + "source": [ + "Download the Gemma model and tokenizer." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ohqi-FOqmRA8" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "RV86s4JVtVCi" - }, - "outputs": [], - "source": [ - "import os\n", - "from typing import Optional, Union, List, Dict\n", - "import kagglehub\n", - "import jax\n", - "import jax.numpy as jnp\n", - "from jinja2 import Template\n", - "from tqdm import tqdm\n", - "import lm_eval\n", - "from lm_eval.api.instance import Instance\n", - "from lm_eval.api.model import LM\n", - "from lm_eval.api.registry import register_model\n", - "from lm_eval.models import utils\n", - "\n", - "\n", - "@register_model(\"Gemma2\")\n", - "class Gemma2LM(LM):\n", - "\n", - " def __init__(self, batch_size: Optional[int] = 1):\n", - " self._batch_size = batch_size\n", - " self._rank = 0\n", - " self._world_size = 1\n", - "\n", - " @property\n", - " def batch_size(self):\n", - " return self._batch_size\n", - "\n", - " def loglikelihood(self, requests: list[Instance]) -> list[tuple[float, bool]]:\n", - " results = []\n", - "\n", - " for chunked_request in utils.chunks(tqdm(requests, disable=False), self._batch_size):\n", - " contexts = [req.args[0] for req in chunked_request]\n", - " next_tokens = [req.args[1] for req in chunked_request]\n", - "\n", - " outputs = sampler(input_strings=contexts, total_generation_steps=1)\n", - "\n", - " for i, logits in enumerate(outputs.logits):\n", - " next_token = next_tokens[i]\n", - "\n", - " next_token_id = vocab.EncodeAsIds(next_token)[0]\n", - " logits = jnp.array(logits[0]) # Assuming generating one token\n", - "\n", - " log_probs = jax.nn.log_softmax(logits, axis=-1)\n", - "\n", - " next_token_logprob = log_probs[next_token_id]\n", - "\n", - " is_greedy = next_token_id == log_probs.argmax()\n", - "\n", - " results.append((float(next_token_logprob), is_greedy))\n", - "\n", - " return results\n", - "\n", - " def loglikelihood_rolling(self, requests: list[Instance]) -> list[tuple[float, bool]]:\n", - " # Used to evaluate perplexity; not important for this tutorial\n", - " raise NotImplementedError(\"loglikelihood_rolling is not implemented\")\n", - "\n", - "\n", - " def generate_until(self, requests: list[Instance]) -> list[str]:\n", - " # Not used for MMLU\n", - " raise NotImplementedError(\"generat_until is not implemented\")\n" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Warning: Looks like you're using an outdated `kagglehub` version, please consider updating (latest version: 0.3.0)\n" + ] + } + ], + "source": [ + "GEMMA_VARIANT = 'gemma2-2b-it'\n", + "GEMMA_PATH = kagglehub.model_download(f'google/gemma-2/flax/{GEMMA_VARIANT}')\n", + "CKPT_PATH = os.path.join(GEMMA_PATH, GEMMA_VARIANT)\n", + "TOKENIZER_PATH = os.path.join(GEMMA_PATH, 'tokenizer.model')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8NM7ALv5E9sh" + }, + "outputs": [], + "source": [ + "params = params_lib.load_and_format_params(CKPT_PATH)\n", + "vocab = spm.SentencePieceProcessor()\n", + "vocab.Load(TOKENIZER_PATH)\n", + "\n", + "transformer_config = transformer_lib.TransformerConfig.from_params(\n", + " params=params,\n", + " cache_size=1024\n", + ")\n", + "\n", + "transformer = transformer_lib.Transformer(transformer_config)\n", + "\n", + "sampler = sampler_lib.Sampler(\n", + " transformer=transformer,\n", + " vocab=vocab,\n", + " params=params['transformer'],\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O4icXT5G9UEZ" + }, + "source": [ + "Create a new Gemma LM calss, following the [New Model Guide](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md). For MMLU, we only need to implement the loglikelihood() function." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "RV86s4JVtVCi" + }, + "outputs": [], + "source": [ + "import os\n", + "from typing import Optional, Union, List, Dict\n", + "import kagglehub\n", + "import jax\n", + "import jax.numpy as jnp\n", + "from jinja2 import Template\n", + "from tqdm import tqdm\n", + "import lm_eval\n", + "from lm_eval.api.instance import Instance\n", + "from lm_eval.api.model import LM\n", + "from lm_eval.api.registry import register_model\n", + "from lm_eval.models import utils\n", + "\n", + "\n", + "@register_model(\"Gemma2\")\n", + "class Gemma2LM(LM):\n", + "\n", + " def __init__(self, batch_size: Optional[int] = 1):\n", + " self._batch_size = batch_size\n", + " self._rank = 0\n", + " self._world_size = 1\n", + "\n", + " @property\n", + " def batch_size(self):\n", + " return self._batch_size\n", + "\n", + " def loglikelihood(self, requests: list[Instance]) -> list[tuple[float, bool]]:\n", + " results = []\n", + "\n", + " for chunked_request in utils.chunks(tqdm(requests, disable=False), self._batch_size):\n", + " contexts = [req.args[0] for req in chunked_request]\n", + " next_tokens = [req.args[1] for req in chunked_request]\n", + "\n", + " outputs = sampler(input_strings=contexts, total_generation_steps=1)\n", + "\n", + " for i, logits in enumerate(outputs.logits):\n", + " next_token = next_tokens[i]\n", + "\n", + " next_token_id = vocab.EncodeAsIds(next_token)[0]\n", + " logits = jnp.array(logits[0]) # Assuming generating one token\n", + "\n", + " log_probs = jax.nn.log_softmax(logits, axis=-1)\n", + "\n", + " next_token_logprob = log_probs[next_token_id]\n", + "\n", + " is_greedy = next_token_id == log_probs.argmax()\n", + "\n", + " results.append((float(next_token_logprob), is_greedy))\n", + "\n", + " return results\n", + "\n", + " def loglikelihood_rolling(self, requests: list[Instance]) -> list[tuple[float, bool]]:\n", + " # Used to evaluate perplexity; not important for this tutorial\n", + " raise NotImplementedError(\"loglikelihood_rolling is not implemented\")\n", + "\n", + "\n", + " def generate_until(self, requests: list[Instance]) -> list[str]:\n", + " # Not used for MMLU\n", + " raise NotImplementedError(\"generat_until is not implemented\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SfTlmr7t9JMA" + }, + "source": [ + "Now start the evaluation process. It takes a long time to run through the entire MMLU benchmark, so we are just going to run a subset of it." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "8KZBGmcAuFKk" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "SfTlmr7t9JMA" - }, - "source": [ - "Now start the evaluation process. It takes a long time to run through the entire MMLU benchmark, so we are just going to run a subset of it." - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:lm-eval:Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234\n", + "INFO:lm-eval:Using pre-initialized model\n", + "INFO:lm-eval:`group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. `group`s which aggregate across subtasks must be only defined in a separate group config file, which will be the official way to create groups that support cross-task aggregation as in `mmlu`. Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs for more information.\n", + "WARNING:lm-eval:Overwriting default num_fewshot of mmlu_management from None to 3\n", + "INFO:lm-eval:Setting fewshot random generator seed to 1234\n", + "INFO:lm-eval:Building contexts for mmlu_management on rank 0...\n", + "100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 103/103 [00:00<00:00, 118.20it/s]\n", + "INFO:lm-eval:Running loglikelihood requests\n", + "100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 412/412 [55:42<00:00, 8.11s/it]\n" + ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8KZBGmcAuFKk" - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "INFO:lm-eval:Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234\n", - "INFO:lm-eval:Using pre-initialized model\n", - "INFO:lm-eval:`group` and `group_alias` keys in TaskConfigs are deprecated and will be removed in v0.4.5 of lm_eval. The new `tag` field will be used to allow for a shortcut to a group of tasks one does not wish to aggregate metrics across. `group`s which aggregate across subtasks must be only defined in a separate group config file, which will be the official way to create groups that support cross-task aggregation as in `mmlu`. Please see the v0.4.4 patch notes and our documentation: https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/new_task_guide.md#advanced-group-configs for more information.\n", - "WARNING:lm-eval:Overwriting default num_fewshot of mmlu_management from None to 3\n", - "INFO:lm-eval:Setting fewshot random generator seed to 1234\n", - "INFO:lm-eval:Building contexts for mmlu_management on rank 0...\n", - "100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 103/103 [00:00<00:00, 118.20it/s]\n", - "INFO:lm-eval:Running loglikelihood requests\n", - "100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 412/412 [55:42<00:00, 8.11s/it]\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'mmlu_management': {'alias': 'management', 'acc,none': 0.6601941747572816, 'acc_stderr,none': 0.046897659372781335}}\n" - ] - } - ], - "source": [ - "import lm_eval\n", - "results = lm_eval.simple_evaluate(\n", - " model=Gemma2LM(batch_size=4),\n", - " tasks=[\"mmlu_management\"],\n", - " num_fewshot=3,\n", - ")\n", - "\n", - "print(results['results'])" - ] - } - ], - "metadata": { - "accelerator": "GPU", - "colab": { - "name": "Gemma_evaluation.ipynb", - "toc_visible": true - }, - "kernelspec": { - "display_name": "Python 3", - "name": "python3" + "name": "stdout", + "output_type": "stream", + "text": [ + "{'mmlu_management': {'alias': 'management', 'acc,none': 0.6601941747572816, 'acc_stderr,none': 0.046897659372781335}}\n" + ] } + ], + "source": [ + "import lm_eval\n", + "results = lm_eval.simple_evaluate(\n", + " model=Gemma2LM(batch_size=4),\n", + " tasks=[\"mmlu_management\"],\n", + " num_fewshot=3,\n", + ")\n", + "\n", + "print(results['results'])" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Gemma_evaluation.ipynb", + "toc_visible": true }, - "nbformat": 4, - "nbformat_minor": 0 + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 } From 54dfb6677fc1c1acabbe6967b5d7e049f545588b Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 5 Nov 2024 14:13:49 +0530 Subject: [PATCH 33/43] Added PyTorch/XLA Gemma Fine-tuning notebook - Supports Colab/Kaggle - Needs a TPU --- Gemma/Finetune_with_Torch_XLA.ipynb | 1634 +++++++++++++++++++++++++++ README.md | 1 + 2 files changed, 1635 insertions(+) create mode 100644 Gemma/Finetune_with_Torch_XLA.ipynb diff --git a/Gemma/Finetune_with_Torch_XLA.ipynb b/Gemma/Finetune_with_Torch_XLA.ipynb new file mode 100644 index 0000000..5036956 --- /dev/null +++ b/Gemma/Finetune_with_Torch_XLA.ipynb @@ -0,0 +1,1634 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "Tce3stUlHN0L", + "metadata": { + "id": "Tce3stUlHN0L" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "tuOe1ymfHZPu", + "metadata": { + "cellView": "form", + "id": "tuOe1ymfHZPu" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "id": "introduction", + "metadata": { + "id": "introduction" + }, + "source": [ + "# Fine-tuning Gemma with Torch XLA and Hugging Face TRL\n", + "\n", + "Welcome to this step-by-step guide on fine-tuning the [Gemma](https://huggingface.co/google/gemma-2b) using [Torch XLA](https://github.com/pytorch/xla).\n", + "\n", + "\n", + "[**Gemma**](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights, pre-trained variants, and instruction-tuned variants. Gemma models are well-suited for a variety of text generation tasks, including question answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop or your own cloud infrastructure, democratizing access to state of the art AI models and helping foster innovation for everyone.\n", + "\n", + "[**Torch XLA**](https://pytorch.org/xla/) enables you to leverage the computational power of TPUs (Tensor Processing Units) for efficient training of deep learning models. By interfacing PyTorch with the [XLA (Accelerated Linear Algebra)](https://openxla.org/xla) compiler, Torch XLA translates PyTorch operations into XLA operations that can be executed on TPUs. This means you can write your models in PyTorch as usual, and Torch XLA handles the underlying computations to run them efficiently on TPUs.\n", + "\n", + "[**Transformer Reinforcement Learning (TRL)**](https://github.com/huggingface/trl) is a framework developed by Hugging Face to fine-tune and align both transformer language and diffusion models using methods such as Supervised Fine-Tuning (SFT), Reward Modeling (RM), Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and others.\n", + "\n", + "Integrating PyTorch with XLA allows developers to run PyTorch code on TPUs with minimal changes to their existing codebase. This seamless integration provides the performance benefits of TPUs while maintaining the flexibility and ease of use of the PyTorch framework.\n", + "\n", + "By the end of this notebook, you will learn:\n", + "\n", + "- About Torch XLA\n", + "- How to peform **Parameter-Efficient Fine-Tuning (PEFT)** with the **Low-Rank Adaptation (LoRA)** on [Gemma 2 2B](https://huggingface.co/google/gemma-2-2b) using Hugging Face's **TRL** framework, **Torch XLA** and TPUs.\n", + "\n", + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/google-gemini/gemma-cookbook/blob/main/Gemma/Finetune_with_Torch_XLA.ipynb)\n", + "\n", + "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://www.kaggle.com/notebooks/welcome?src=https://github.com/google-gemini/gemma-cookbook/blob/main/Gemma/Finetune_with_Torch_XLA.ipynb)" + ] + }, + { + "cell_type": "markdown", + "id": "L0Srgc7OGVJj", + "metadata": { + "id": "L0Srgc7OGVJj" + }, + "source": [ + "## Setup\n", + "\n", + "### Selecting the Runtime Environment\n", + "\n", + "To start, you can choose either **Google Colab** or **Kaggle** as your platform. Select one, and proceed from there.\n", + "\n", + "- #### **Google Colab** \"Google\n", + "\n", + " 1. Click **Open in Colab**.\n", + " 2. In the menu, go to **Runtime** > **Change runtime type**.\n", + " 3. Under **Hardware accelerator**, select **TPU**.\n", + " 4. Ensure that the **TPU type** is set to **TPU v2-8**.\n", + "\n", + "- #### **Kaggle** \"Kaggle\"\n", + "\n", + " 1. Click **Open in Kaggle**.\n", + " 2. Click on **Settings** in the right sidebar.\n", + " 3. Under **Accelerator**, select **TPUs**.\n", + " - Note: Kaggle currently provides **TPU v3-8**.\n", + " 4. Save the settings, and the notebook will restart with TPU support.\n", + "\n", + "\n", + "### Gemma using Hugging Face\n", + "\n", + "Before diving into the tutorial, let's set up Gemma:\n", + "\n", + "1. **Create a Hugging Face Account**: If you don't have one, you can sign up for a free account [here](https://huggingface.com/join).\n", + "2. **Access the Gemma Model**: Visit the [Gemma model page](https://huggingface.com/collections/google/gemma-2-release-667d6600fd5220e7b967f315) and accept the usage conditions.\n", + "3. **Generate a Hugging Face Token**: Go to your Hugging Face [settings page](https://huggingface.com/settings/tokens) and generate a new access token (preferably with `write` permissions). You'll need this token later in the tutorial.\n", + "\n", + "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**" + ] + }, + { + "cell_type": "markdown", + "id": "configure-credentials", + "metadata": { + "id": "configure-credentials" + }, + "source": [ + "### Configure Your Credentials\n", + "\n", + "To access private models and datasets, you need to log in to the Hugging Face (HF) ecosystem.\n", + "\n", + "- #### **Google Colab** \"Google\n", + " If you're using Colab, you can securely store your Hugging Face token (`HF_TOKEN`) using the Colab Secrets manager:\n", + " 1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + " 2. **Add Hugging Face Token**:\n", + " - Create a new secret with the **name** `HF_TOKEN`.\n", + " - Copy/paste your token key into the **Value** input box of `HF_TOKEN`.\n", + " - **Toggle** the button on the left to allow notebook access to the secret\n", + "\n", + "- #### **Kaggle** \"Kaggle\"\n", + " To securely use your Hugging Face token (`HF_TOKEN`) in this notebook, you'll need to add it as a secret in your Kaggle environment: \n", + " 1. Open your Kaggle notebook and locate the **Addons** menu at the top in your notebook interface.\n", + " 2. Click on **Secrets** to manage your environment secrets. \n", + " \"The\n", + " 3. **Add Hugging Face Token**:\n", + " - Click on the **Add secret** button.\n", + " - In the **Label** field, enter `HF_TOKEN`. \n", + " - In the **Value** field, paste your Hugging Face token.\n", + " - Click **Save** to add the secret." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "mBPIzOqnGmt-", + "metadata": { + "id": "mBPIzOqnGmt-" + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "if 'google.colab' in sys.modules:\n", + " # Running on Colab\n", + " from google.colab import userdata\n", + " os.environ['HF_TOKEN'] = userdata.get('HF_TOKEN')\n", + "elif os.path.exists('/kaggle/working'):\n", + " # Running on Kaggle\n", + " from kaggle_secrets import UserSecretsClient\n", + " user_secrets = UserSecretsClient()\n", + " os.environ['HF_TOKEN'] = user_secrets.get_secret(\"HF_TOKEN\")\n", + "else:\n", + " # Not running on Colab or Kaggle\n", + " raise EnvironmentError('This notebook is designed to run on Google Colab or Kaggle.')" + ] + }, + { + "cell_type": "markdown", + "id": "ys5-2RMfUroM", + "metadata": { + "id": "ys5-2RMfUroM" + }, + "source": [ + "This code retrieves your secrets and sets them as environment variables, which you will use later in the tutorial." + ] + }, + { + "cell_type": "markdown", + "id": "setting-up-environment", + "metadata": { + "id": "setting-up-environment" + }, + "source": [ + "### Setting Up the Environment\n", + "\n", + "Next, you'll set up the environment by installing all the necessary Python packages for fine-tuning the Gemma model on a TPU VM using Torch XLA.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "setup-code", + "metadata": { + "id": "setup-code" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found existing installation: tensorflow 2.15.0\n", + "Uninstalling tensorflow-2.15.0:\n", + " Successfully uninstalled tensorflow-2.15.0\n", + "Found existing installation: tf_keras 2.15.1\n", + "Uninstalling tf_keras-2.15.1:\n", + " Successfully uninstalled tf_keras-2.15.1\n", + "Collecting tensorflow==2.18.0\n", + " Downloading tensorflow-2.18.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)\n", + "Collecting tf-keras==2.18.0\n", + " Downloading tf_keras-2.18.0-py3-none-any.whl.metadata (1.6 kB)\n", + "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (1.4.0)\n", + "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (1.6.3)\n", + "Requirement already satisfied: flatbuffers>=24.3.25 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (24.3.25)\n", + "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (0.6.0)\n", + "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (0.2.0)\n", + "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (18.1.1)\n", + "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (3.4.0)\n", + "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (24.1)\n", + "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (4.25.5)\n", + "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (2.32.3)\n", + "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (75.1.0)\n", + "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (1.16.0)\n", + "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (2.5.0)\n", + "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (4.12.2)\n", + "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (1.14.1)\n", + "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (1.67.1)\n", + "Collecting tensorboard<2.19,>=2.18 (from tensorflow==2.18.0)\n", + " Downloading tensorboard-2.18.0-py3-none-any.whl.metadata (1.6 kB)\n", + "Collecting keras>=3.5.0 (from tensorflow==2.18.0)\n", + " Downloading keras-3.6.0-py3-none-any.whl.metadata (5.8 kB)\n", + "Requirement already satisfied: numpy<2.1.0,>=1.26.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (1.26.4)\n", + "Requirement already satisfied: h5py>=3.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (3.12.1)\n", + "Collecting ml-dtypes<0.5.0,>=0.4.0 (from tensorflow==2.18.0)\n", + " Downloading ml_dtypes-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB)\n", + "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.18.0) (0.37.1)\n", + "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow==2.18.0) (0.44.0)\n", + "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from keras>=3.5.0->tensorflow==2.18.0) (13.9.3)\n", + "Collecting namex (from keras>=3.5.0->tensorflow==2.18.0)\n", + " Downloading namex-0.0.8-py3-none-any.whl.metadata (246 bytes)\n", + "Collecting optree (from keras>=3.5.0->tensorflow==2.18.0)\n", + " Downloading optree-0.13.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (47 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m47.8/47.8 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow==2.18.0) (3.4.0)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow==2.18.0) (3.10)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow==2.18.0) (2.2.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorflow==2.18.0) (2024.8.30)\n", + "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.19,>=2.18->tensorflow==2.18.0) (3.7)\n", + "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.19,>=2.18->tensorflow==2.18.0) (0.7.2)\n", + "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.19,>=2.18->tensorflow==2.18.0) (3.0.6)\n", + "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.19,>=2.18->tensorflow==2.18.0) (3.0.2)\n", + "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->keras>=3.5.0->tensorflow==2.18.0) (3.0.0)\n", + "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->keras>=3.5.0->tensorflow==2.18.0) (2.18.0)\n", + "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->keras>=3.5.0->tensorflow==2.18.0) (0.1.2)\n", + "Downloading tensorflow-2.18.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (615.3 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m615.3/615.3 MB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading tf_keras-2.18.0-py3-none-any.whl (1.7 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m59.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading keras-3.6.0-py3-none-any.whl (1.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m46.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading ml_dtypes-0.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m67.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading tensorboard-2.18.0-py3-none-any.whl (5.5 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m5.5/5.5 MB\u001b[0m \u001b[31m94.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading namex-0.0.8-py3-none-any.whl (5.8 kB)\n", + "Downloading optree-0.13.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (358 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m358.9/358.9 kB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hInstalling collected packages: namex, optree, ml-dtypes, tensorboard, keras, tensorflow, tf-keras\n", + " Attempting uninstall: ml-dtypes\n", + " Found existing installation: ml-dtypes 0.2.0\n", + " Uninstalling ml-dtypes-0.2.0:\n", + " Successfully uninstalled ml-dtypes-0.2.0\n", + " Attempting uninstall: tensorboard\n", + " Found existing installation: tensorboard 2.15.2\n", + " Uninstalling tensorboard-2.15.2:\n", + " Successfully uninstalled tensorboard-2.15.2\n", + " Attempting uninstall: keras\n", + " Found existing installation: keras 2.15.0\n", + " Uninstalling keras-2.15.0:\n", + " Successfully uninstalled keras-2.15.0\n", + "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", + "tensorflow-text 2.15.0 requires tensorflow<2.16,>=2.15.0; platform_machine != \"arm64\" or platform_system != \"Darwin\", but you have tensorflow 2.18.0 which is incompatible.\u001b[0m\u001b[31m\n", + "\u001b[0mSuccessfully installed keras-3.6.0 ml-dtypes-0.4.1 namex-0.0.8 optree-0.13.0 tensorboard-2.18.0 tensorflow-2.18.0 tf-keras-2.18.0\n", + "Found existing installation: tensorflow 2.18.0\n", + "Uninstalling tensorflow-2.18.0:\n", + " Successfully uninstalled tensorflow-2.18.0\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m230.0/230.0 MB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m44.1/44.1 kB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m10.0/10.0 MB\u001b[0m \u001b[31m73.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m81.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m64.8/64.8 kB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m480.6/480.6 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m179.3/179.3 kB\u001b[0m \u001b[31m12.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m38.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m241.9/241.9 kB\u001b[0m \u001b[31m16.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m124.6/124.6 kB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m318.7/318.7 kB\u001b[0m \u001b[31m22.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m208.9/208.9 kB\u001b[0m \u001b[31m13.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m310.2/310.2 kB\u001b[0m \u001b[31m5.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m320.7/320.7 kB\u001b[0m \u001b[31m17.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m324.3/324.3 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: tpu-info in /usr/local/lib/python3.10/dist-packages (0.2.0)\n", + "Requirement already satisfied: grpcio>=1.65.5 in /usr/local/lib/python3.10/dist-packages (from tpu-info) (1.67.1)\n", + "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from tpu-info) (4.25.5)\n", + "Requirement already satisfied: rich in /usr/local/lib/python3.10/dist-packages (from tpu-info) (13.9.3)\n", + "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich->tpu-info) (3.0.0)\n", + "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich->tpu-info) (2.18.0)\n", + "Requirement already satisfied: typing-extensions<5.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from rich->tpu-info) (4.12.2)\n", + "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich->tpu-info) (0.1.2)\n" + ] + } + ], + "source": [ + "# Uninstalling any existing TensorFlow installations and then install the CPU-only version to avoid conflicts while using the TPU.\n", + "!pip uninstall -y tensorflow tf-keras\n", + "!pip install tensorflow==2.18.0 tf-keras==2.18.0\n", + "\n", + "!pip uninstall tensorflow -y\n", + "!pip install tensorflow-cpu==2.18.0 -q\n", + "\n", + "# Install the appropriate Hugging Face libraries to ensure compatibility with the Gemma model and PEFT.\n", + "!pip install transformers==4.46.1 -U -q\n", + "!pip install datasets==3.1.0 -U -q\n", + "!pip install trl==0.12.0 peft==0.13.2 -U -q\n", + "!pip install accelerate==0.34.0 -U -q\n", + "\n", + "# Install PyTorch and Torch XLA with versions compatible with the TPU runtime, ensuring efficient TPU utilization.\n", + "!pip install -qq torch~=2.5.0 --index-url https://download.pytorch.org/whl/cpu -q\n", + "!pip install -qq torch_xla[tpu]~=2.5.0 -f https://storage.googleapis.com/libtpu-releases/index.html -q\n", + "\n", + "# Install the `tpu-info` package to display TPU-related information\n", + "!pip install tpu-info" + ] + }, + { + "cell_type": "markdown", + "id": "note", + "metadata": { + "id": "note" + }, + "source": [ + "**Note**: Ensure that your PyTorch and Torch XLA versions are compatible with the TPU you're using." + ] + }, + { + "cell_type": "markdown", + "id": "1M7YM_apWfOk", + "metadata": { + "id": "1M7YM_apWfOk" + }, + "source": [ + "### Verify TPU Setup\n" + ] + }, + { + "cell_type": "markdown", + "id": "Yjo5_5xfVYNm", + "metadata": { + "id": "Yjo5_5xfVYNm" + }, + "source": [ + "You run `!tpu-info` to verify the TPU has been properly initialized." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "skAvSa5KF65m", + "metadata": { + "id": "skAvSa5KF65m" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[3mTPU Chips \u001b[0m\n", + "โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”“\n", + "โ”ƒ\u001b[1m \u001b[0m\u001b[1mChip \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mType \u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mDevices\u001b[0m\u001b[1m \u001b[0mโ”ƒ\u001b[1m \u001b[0m\u001b[1mPID \u001b[0m\u001b[1m \u001b[0mโ”ƒ\n", + "โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”ฉ\n", + "โ”‚ /dev/accel0 โ”‚ TPU v2 chip โ”‚ 2 โ”‚ None โ”‚\n", + "โ”‚ /dev/accel1 โ”‚ TPU v2 chip โ”‚ 2 โ”‚ None โ”‚\n", + "โ”‚ /dev/accel2 โ”‚ TPU v2 chip โ”‚ 2 โ”‚ None โ”‚\n", + "โ”‚ /dev/accel3 โ”‚ TPU v2 chip โ”‚ 2 โ”‚ None โ”‚\n", + "โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”˜\n", + "Libtpu metrics unavailable. Is there a framework using the TPU? See https://github.com/google/cloud-accelerator-diagnostics/tree/main/tpu_info for more information\n" + ] + } + ], + "source": [ + "!tpu-info" + ] + }, + { + "cell_type": "markdown", + "id": "o0zsGkzqWien", + "metadata": { + "id": "o0zsGkzqWien" + }, + "source": [ + "If everything is set up correctly, you should see the TPU details printed out.\n" + ] + }, + { + "cell_type": "markdown", + "id": "importing-libraries", + "metadata": { + "id": "importing-libraries" + }, + "source": [ + "## Import the libraries\n", + "\n", + "Now, import all the necessary libraries required for fine-tuning.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "import-code", + "metadata": { + "id": "import-code" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "PyTorch version: 2.5.0+cpu\n", + "Torch XLA version: 2.5.0+libtpu\n" + ] + } + ], + "source": [ + "import pandas as pd\n", + "\n", + "import torch\n", + "print(f\"PyTorch version: {torch.__version__}\")\n", + "\n", + "import torch_xla\n", + "print(f\"Torch XLA version: {torch_xla.__version__}\")\n", + "\n", + "import torch_xla.core.xla_model as xm\n", + "import torch_xla.runtime as xr\n", + "\n", + "from transformers import (\n", + " AutoTokenizer,\n", + " AutoModelForCausalLM,\n", + ")\n", + "from trl import SFTTrainer, SFTConfig\n", + "from peft import LoraConfig, PeftModel\n", + "\n", + "from datasets import load_dataset\n", + "\n", + "# Enable Single Program Multiple Data (SPMD) mode,\n", + "# which allows for parallel execution across multiple TPU cores\n", + "xr.use_spmd()" + ] + }, + { + "cell_type": "markdown", + "id": "GSDScQniWo4L", + "metadata": { + "id": "GSDScQniWo4L" + }, + "source": [ + "This setup ensures that your environment is correctly configured to use TPUs with PyTorch.\n" + ] + }, + { + "cell_type": "markdown", + "id": "finetuning-peft", + "metadata": { + "id": "finetuning-peft" + }, + "source": [ + "## Fine-tune using PEFT and LoRA\n", + "\n", + "Traditional fine-tuning of large language models (LLMs) like Gemma requires adjusting billions of parameters, making it resource-intensive. This process demands significant computational power and time, which can be impractical for many use cases. That's where Parameter-Efficient Fine-Tuning (PEFT) techniques come in.\n", + "\n", + "### Parameter-Efficient Fine-Tuning (PEFT)\n", + "\n", + "PEFT allows you to adapt large models to specific tasks by updating only a small portion of their parameters. Instead of retraining the entire model, PEFT adds lightweight layers or adapters. Most of the pre-trained weights remain frozen. This approach greatly reduces the computational requirements and the amount of data needed for fine-tuning, making it feasible to fine-tune large models even on modest hardware.\n", + "\n", + "### Low-Rank Adaptation (LoRA)\n", + "\n", + "Among these techniques, one effective option is Low-Rank Adaptation (LoRA). LoRA introduces small, trainable matrices into the model's architecture, specifically targeting the attention layers of Transformer models. Instead of updating the full weight matrices, LoRA adds rank-decomposed matrices, making adaptation more efficient.\n", + "\n", + "#### Key Advantages of LoRA\n", + "\n", + "- **Efficiency**: LoRA significantly reduces the number of trainable parameters by using low-rank adaptations, making the fine-tuning process much more efficient.\n", + "- **Memory Savings**: Since only the additional low-rank matrices are updated, GPU/TPU memory requirements are considerably lower.\n", + "- **Modularity**: LoRA adapters can be easily merged with the original model or kept separate, offering flexibility for deployment.\n", + "\n", + "In the next section, you'll explore how to implement PEFT with LoRA to fine-tune Gemma using Torch XLA on TPUs and perform the following steps:\n", + "\n", + "- Load a dataset\n", + "- Configure the training parameters\n", + "- Load the Gemma model and tokenizer\n", + "- Fine-tune the model on TPUs using **TRL**'s `SFTTrainer` class" + ] + }, + { + "cell_type": "markdown", + "id": "create-dataset", + "metadata": { + "id": "create-dataset" + }, + "source": [ + "### Load a dataset\n", + "\n", + "For this guide, you'll use an existing dataset from Hugging Face. You can replace it with your own dataset if you prefer.\n", + "\n", + "The dataset chosen for this guide is [**hieunguyenminh/roleplay**](https://huggingface.com/datasets/hieunguyenminh/roleplay), which embodies a wide range of original characters, each with a unique persona. It includes fictional characters, complete with their own backgrounds, core traits, relationships, goals, and distinct speaking styles.\n", + "\n", + "**Credits:** **https://huggingface.com/hieunguyenminh**" + ] + }, + { + "cell_type": "markdown", + "id": "j01SyZuIuU6E", + "metadata": { + "id": "j01SyZuIuU6E" + }, + "source": [ + "You specify the dataset name and use the `load_dataset` function from the Hugging Face `datasets` library to load the training split of the dataset." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dataset-code", + "metadata": { + "id": "dataset-code" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Dataset({\n", + " features: ['name', 'description', 'text'],\n", + " num_rows: 5755\n", + "})" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dataset_name = \"hieunguyenminh/roleplay\"\n", + "dataset = load_dataset(dataset_name, split=\"train\")\n", + "dataset" + ] + }, + { + "cell_type": "markdown", + "id": "lz3MMqzKGsOn", + "metadata": { + "id": "lz3MMqzKGsOn" + }, + "source": [ + "First, let's split the dataset into training and validation sets." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "TNj_8BdoGrn2", + "metadata": { + "id": "TNj_8BdoGrn2" + }, + "outputs": [], + "source": [ + "# The first 80% of `train` for training\n", + "train_dataset = load_dataset(dataset_name, split='train[:80%]')\n", + "# The last 20% of `train` for evaluation\n", + "valid_dataset = load_dataset(dataset_name, split='train[-20%:]')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "DjqJmj-b8vB5", + "metadata": { + "id": "DjqJmj-b8vB5" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Dataset({\n", + " features: ['name', 'description', 'text'],\n", + " num_rows: 4604\n", + "})" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_dataset" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "B18kyvz9N3gT", + "metadata": { + "id": "B18kyvz9N3gT" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Dataset({\n", + " features: ['name', 'description', 'text'],\n", + " num_rows: 1151\n", + "})" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "valid_dataset" + ] + }, + { + "cell_type": "markdown", + "id": "L6U3ZA0TXNt8", + "metadata": { + "id": "L6U3ZA0TXNt8" + }, + "source": [ + "Let's look at a few samples to understand the data.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cooIZR5GS08_", + "metadata": { + "id": "cooIZR5GS08_" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, + "text/plain": [ + "'<|system|>Introducing Minami \"Echo\" Ishikawa, a mysterious VR assassin known for her uncanny ability to blend seamlessly into the shadows. Minami possesses a deep understanding of stealth techniques, allowing her to silently eliminate her targets with calculated precision. Her cold and calculating demeanor makes her a formidable force to be reckoned with, leaving enemies shivering at the thought of facing her wrath.\\n<|user|>Echo, what makes you so adept at disappearing into thin air?\\n<|assistant|>Shadows are my allies, and silence is my weapon. I move as fluidly as the wind, becoming one with my surroundings. My footsteps are whispers, and my presence is a phantom in the night.\\n<|user|>What\\'s the key to executing a flawless assassination, Echo?\\n<|assistant|>Precision is paramount. Every movement, every breath, calculated to achieve the perfect strike. My targets never see me coming, their lives snuffed out before they even realize they\\'re in danger.\\n<|user|>Echo, do you ever feel remorse for the lives you take?\\n<|assistant|>Sentiment is a weakness I shed long ago. My purpose is clear, my resolve unwavering. I am but an instrument of fate, delivering justice or vengeance as required.\\n<|user|>What drives you, Echo? What keeps you going?\\n<|assistant|>The thrill of the chase, the satisfaction of a clean kill. The world is a tapestry of lies and deceit, and I am the needle that unravels it, exposing the truth beneath.\\n<|user|>Echo, is there anything that can stop you?\\n<|assistant|> I am a shadow, a whisper in the darkness. Nothing can truly stop me, for I am the embodiment of death itself. Those who dare to challenge me only seal their own fate.'" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "valid_dataset[10]['text']" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "kMcTselgOz3z", + "metadata": { + "id": "kMcTselgOz3z" + }, + "outputs": [], + "source": [ + "if 'google.colab' in sys.modules:\n", + " from google.colab import data_table\n", + "\n", + " # Enable interactive DataFrame display\n", + " data_table.enable_dataframe_formatter()\n", + "\n", + "# Convert the 'train' split to a Pandas DataFrame\n", + "df = pd.DataFrame(train_dataset)\n", + "\n", + "# Select the 'text' column and exclude the rest\n", + "df_text = df[['text']]\n", + "df_text" + ] + }, + { + "cell_type": "markdown", + "id": "define-parameters", + "metadata": { + "id": "define-parameters" + }, + "source": [ + "### Training Configuration\n", + "\n", + "Now you need to define all the hyperparameters and configurations needed for the fine-tuning process and this includes defining the following:\n", + "\n", + "- The base model and new model names\n", + "- LoRA Configuration\n", + "- Training Arguments\n", + "- SFT Parameters\n", + "- Misc. Parameters\n" + ] + }, + { + "cell_type": "markdown", + "id": "wPJaoQqMJMIp", + "metadata": { + "id": "wPJaoQqMJMIp" + }, + "source": [ + "You start by specifying the base model (`google/gemma-2b`) and the directory where the fine-tuned model will be saved (`gemma-ft`).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fz5T6Wr1IbM-", + "metadata": { + "id": "fz5T6Wr1IbM-" + }, + "outputs": [], + "source": [ + "# Define model names\n", + "model_name = \"google/gemma-2-2b\"\n", + "new_model = \"gemma-ft\"" + ] + }, + { + "cell_type": "markdown", + "id": "NdMLPDvdJLTF", + "metadata": { + "id": "NdMLPDvdJLTF" + }, + "source": [ + "LoRA (Low-Rank Adaptation) allows for efficient fine-tuning by adapting only a subset of model parameters.\n", + "\n", + "Here, you set the following parameters:\n", + "- `lora_r` to 64, which controls the rank of the adaptation matrices,\n", + "- `lora_alpha` to 32 for scaling\n", + "- `lora_dropout` to 0.1 to prevent overfitting." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6fu4Efr7UPFK", + "metadata": { + "id": "6fu4Efr7UPFK" + }, + "outputs": [], + "source": [ + "# LoRA attention dimension\n", + "lora_r = 64 # @param {\"type\":\"slider\",\"min\":0,\"max\":64,\"step\":2}\n", + "# Alpha parameter for LoRA scaling\n", + "lora_alpha = 32 # @param {\"type\":\"slider\",\"min\":0,\"max\":64,\"step\":2}\n", + "# Dropout probability for LoRA layers\n", + "lora_dropout = 0.1 # @param {\"type\":\"slider\",\"min\":0,\"max\":1,\"step\":0.01}" + ] + }, + { + "cell_type": "markdown", + "id": "AR8OCRL2JIuA", + "metadata": { + "id": "AR8OCRL2JIuA" + }, + "source": [ + "Set up the training arguments that define how the model will be trained.\n", + "\n", + "Here, you'll define the **output directory**, **number of training epochs**, and **batch sizes** for training and evaluation. You enable **gradient checkpointing** to save memory and set `max_grad_norm` for gradient clipping to stabilize training. The **learning rate**, **optimizer**, and **learning rate scheduler** are configured to optimize the training process. The `max_steps` is set to **-1** to let the number of epochs control training duration.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "HEmgRATcIfAR", + "metadata": { + "id": "HEmgRATcIfAR" + }, + "outputs": [], + "source": [ + "# Output directory where the model predictions and checkpoints will be stored\n", + "output_dir = \"./results\" # @param {\"type\":\"string\"}\n", + "# Number of training epochs\n", + "num_train_epochs = 5 # @param {\"type\":\"slider\",\"min\":1,\"max\":20,\"step\":2}\n", + "# Batch size per TPU core for training\n", + "per_device_train_batch_size = 32 # @param {\"type\":\"slider\",\"min\":1,\"max\":64,\"step\":1}\n", + "# Batch size per TPU core for evaluation\n", + "per_device_eval_batch_size = 32 # @param {\"type\":\"slider\",\"min\":1,\"max\":64,\"step\":1}\n", + "# Number of update steps to accumulate the gradients for\n", + "gradient_accumulation_steps = 1 # @param {\"type\":\"slider\",\"min\":0,\"max\":16,\"step\":2}\n", + "# Maximum gradient normal (gradient clipping)\n", + "max_grad_norm = 0.3 # @param {\"type\":\"slider\",\"min\":0,\"max\":1,\"step\":0.01}\n", + "# Initial learning rate (adafactor optimizer)\n", + "learning_rate = 0.0002 # @param {\"type\":\"slider\",\"min\":0.00001,\"max\":0.0005,\"step\":0.00001}\n", + "# Optimizer to use\n", + "optim = \"adafactor\" # adafactor, adamw_torch_fused\n", + "# Learning rate schedule (constant a bit better than cosine)\n", + "lr_scheduler_type = \"constant\"\n", + "# Number of training steps (overrides num_train_epochs)\n", + "max_steps = -1\n", + "# Ratio of steps for a linear warmup (from 0 to learning rate)\n", + "warmup_ratio = 0.03 # @param {\"type\":\"slider\",\"min\":0,\"max\":0.1,\"step\":0.01}\n", + "# Enable bfloat16 precision\n", + "bf16 = True\n", + "# Log every X updates steps\n", + "logging_steps = 1" + ] + }, + { + "cell_type": "markdown", + "id": "n2fdj68nJG8D", + "metadata": { + "id": "n2fdj68nJG8D" + }, + "source": [ + "In the SFT parameters, `max_seq_length` is set to 512 to define the maximum token length for inputs, and `packing` is enabled to pack multiple shorter sequences into one input for efficiency." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "YTkqcAyOI3ZY", + "metadata": { + "id": "YTkqcAyOI3ZY" + }, + "outputs": [], + "source": [ + "# Maximum sequence length to use\n", + "max_seq_length = 512 # @param {\"type\":\"slider\",\"min\":32,\"max\":1024,\"step\":2}\n", + "# Pack multiple short examples in the same input sequence to increase efficiency\n", + "packing = True # False" + ] + }, + { + "cell_type": "markdown", + "id": "VdyuPcwXdFIH", + "metadata": { + "id": "VdyuPcwXdFIH" + }, + "source": [ + "### Under the hood: PyTorch and XLA\n", + "\n", + "PyTorch programs define computation graphs dynamically using its `autograd` system. The TPU does not directly execute Python code; instead, it runs the computation graph defined by your PyTorch program. Behind the scenes, a compiler called **XLA (Accelerated Linear Algebra compiler)** transforms the PyTorch computation graph into TPU machine code. This compiler also performs numerous advanced optimizations on your code and memory layout. The compilation occurs automatically as tasks are sent to the TPU, and you do not need to include XLA in your build chain explicitly.\n", + "\n", + "\"PyTorch\n", + "\n", + "The combination of **PyTorch** and **XLA** offers several key advantages:\n", + "\n", + "1. **Seamless Performance Enhancement:** Maintain PyTorch's intuitive and pythonic workflow while effortlessly achieving significant performance gains through the XLA compiler. This integration allows you to optimize your models without altering your familiar coding practices.\n", + "\n", + "2. **Comprehensive Ecosystem Access:** Leverage PyTorch's extensive ecosystem, including a wide range of tools, pretrained models, and a vibrant community. This access enables you to accelerate development, utilize state-of-the-art resources, and benefit from collective expertise.\n", + "\n", + "Harnessing these advantages, you can efficiently fine-tune your custom Gemma model using TPUs." + ] + }, + { + "cell_type": "markdown", + "id": "load-model-tokenizer", + "metadata": { + "id": "load-model-tokenizer" + }, + "source": [ + "### Fine-tune Gemma using TPUs\n", + "\n", + "The training leverages PyTorch, XLA and TPUs for efficient computation and uses LoRA for parameter-efficient fine-tuning, which reduces the number of trainable parameters by adapting only specific layers.\n", + "\n", + "Here, you'll be setting up the following:\n", + "\n", + "- The Gemma **base model** and **tokenizer**\n", + "- The **LoRA (Low-Rank Adaptation)** configuration for **PEFT (Parameter-Efficient Fine-Tuning)**\n", + "- The [**FSDP**](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html#how-fsdp-works) configuration for efficient TPU training\n", + "- The **Hugging Face `SFTTrainer` instance** using the training and SFT parameters" + ] + }, + { + "cell_type": "markdown", + "id": "oeQJ5ou8bOX7", + "metadata": { + "id": "oeQJ5ou8bOX7" + }, + "source": [ + "First, load the Gemma 2B pre-trained model weights using `AutoModelForCausalLM`, while setting `torch_dtype` to `torch.bfloat16` for optimal performance on TPUs" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "SGeNI91SbeIZ", + "metadata": { + "id": "SGeNI91SbeIZ" + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "6357fd6082b24a70857ec9c1ae1f1cba", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading checkpoint shards: 0%| | 0/3 [00:00\n", + " warnings.warn(\"For backward hooks to be called,\"\n", + "/usr/local/lib/python3.10/dist-packages/torch_xla/utils/checkpoint.py:183: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n", + " torch.cuda.amp.autocast(**ctx.gpu_autocast_kwargs), \\\n", + "/usr/local/lib/python3.10/dist-packages/torch_xla/utils/checkpoint.py:184: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.\n", + " torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):\n" + ] + }, + { + "data": { + "text/html": [ + "\n", + "
\n", + " \n", + " \n", + " [710/710 32:53, Epoch 5/5]\n", + "
\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
EpochTraining LossValidation Loss
11.1016001.291131
20.8438001.298483
30.8633001.269991
40.7305001.299862
50.4980001.407629

" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py:1457: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n", + " xldata.append(torch.load(xbio))\n", + "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1810: UserWarning: For backward hooks to be called, module output should be a Tensor or a tuple of Tensors but received \n", + " warnings.warn(\"For backward hooks to be called,\"\n", + "/usr/local/lib/python3.10/dist-packages/torch_xla/utils/checkpoint.py:183: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.\n", + " torch.cuda.amp.autocast(**ctx.gpu_autocast_kwargs), \\\n", + "/usr/local/lib/python3.10/dist-packages/torch_xla/utils/checkpoint.py:184: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead.\n", + " torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs):\n", + "/usr/local/lib/python3.10/dist-packages/torch_xla/core/xla_model.py:1457: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n", + " xldata.append(torch.load(xbio))\n", + "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1810: UserWarning: For backward hooks to be called, module output should be a Tensor or a tuple of Tensors but received \n", + " warnings.warn(\"For backward hooks to be called,\"\n" + ] + }, + { + "data": { + "text/plain": [ + "TrainOutput(global_step=710, training_loss=0.900107284330986, metrics={'train_runtime': 2044.2422, 'train_samples_per_second': 11.114, 'train_steps_per_second': 0.347, 'total_flos': 1.8324321780891648e+17, 'train_loss': 0.900107284330986, 'epoch': 5.0})" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "trainer.train()" + ] + }, + { + "cell_type": "markdown", + "id": "0YKWVCXUc3gP", + "metadata": { + "id": "0YKWVCXUc3gP" + }, + "source": [ + "After training is complete, you save the fine-tuned model by moving it to the CPU with `trainer.model.to('cpu')` to ensure compatibility and then calling `save_pretrained(new_model)` to save the model weights and configuration files to the directory specified by `new_model` (**gemma-ft**). This allows you to reload and use the fine-tuned model later for inference or further training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "X88_th2Jc5Lr", + "metadata": { + "id": "X88_th2Jc5Lr" + }, + "outputs": [], + "source": [ + "# Remove the model weights directory if it exists\n", + "!rm -rf gemma-ft\n", + "\n", + "# Save the LoRA adapter\n", + "trainer.model.to('cpu').save_pretrained(new_model)" + ] + }, + { + "cell_type": "markdown", + "id": "prompt-model", + "metadata": { + "id": "prompt-model" + }, + "source": [ + "## Prompt using the newly fine-tuned model\n", + "\n", + "\n", + "Now that you've finally fine-tuned your custom Gemma model, let's reload the LoRA adapter weights to finally prompt using it and also verify if it's really working as intended." + ] + }, + { + "cell_type": "markdown", + "id": "kMcfyA3ed_EC", + "metadata": { + "id": "kMcfyA3ed_EC" + }, + "source": [ + "To do this, use the following steps to correctly reload the adapter weights:\n", + "\n", + "- Use `AutoModelForCausalLM.from_pretrained` to first load the **base Gemma model**, while setting `low_cpu_mem_usage=True` to optimize memory consumption (since you're using a TPU) and `torch_dtype=torch.bfloat16` for consistency with the fine-tuned model.\n", + "\n", + "- Load the **fine-tuned LoRA adapter** that you've previously saved into the base model using `PeftModel.from_pretrained`, where `new_model` is the directory containing your fine-tuned weights.\n", + "\n", + "- The `model.merge_and_unload` function **merges** the **LoRA adapter weights** with the **base model weights** and unloads the adapter, resulting in a standalone model ready for inference." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "NGUY_Gw-eFh4", + "metadata": { + "id": "NGUY_Gw-eFh4" + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "5cc5b5437849424aba2a9e13b6149719", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading checkpoint shards: 0%| | 0/3 [00:00Introducing Minami \"Echo\" Ishikawa, a mysterious VR assassin known for her uncanny ability to blend seamlessly into the shadows. \\\n", + " Minami possesses a deep understanding of stealth techniques, allowing her to silently eliminate her targets with calculated precision. \\\n", + " Her cold and calculating demeanor makes her a formidable force to be reckoned with, leaving enemies shivering at the thought of facing her wrath.\n", + " <|user|>Echo, what makes you so adept at disappearing into thin air?\n", + " <|assistant|>\"\"\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "jgOfjrs-eyKr", + "metadata": { + "id": "jgOfjrs-eyKr" + }, + "outputs": [], + "source": [ + "input_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cpu\")\n", + "outputs = model.generate(**input_ids, max_length=256, repetition_penalty=1.1)" + ] + }, + { + "cell_type": "markdown", + "id": "5n-oX0EnekMe", + "metadata": { + "id": "5n-oX0EnekMe" + }, + "source": [ + "Finally, you decode the output tokens back into human-readable text with `tokenizer.decode` and print the result, allowing you to see how the fine-tuned model responds to the prompt." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "nUVCOLl3ejYb", + "metadata": { + "id": "nUVCOLl3ejYb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " <|system|>Introducing Minami \"Echo\" Ishikawa, a mysterious VR assassin known for her uncanny ability to blend seamlessly into the shadows. Minami possesses a deep understanding of stealth techniques, allowing her to silently eliminate her targets with calculated precision. Her cold and calculating demeanor makes her a formidable force to be reckoned with, leaving enemies shivering at the thought of facing her wrath.\n", + " <|user|>Echo, what makes you so adept at disappearing into thin air?\n", + " <|assistant|>I'm not sure if I can answer that question without revealing my secrets. But let me tell you this: when it comes to vanishing like smoke in the wind, I am truly an expert! ( อกยฐ อœส– อกยฐ)\n" + ] + } + ], + "source": [ + "print(tokenizer.decode(outputs[0]))" + ] + }, + { + "cell_type": "markdown", + "id": "16Dmm5njjHGN", + "metadata": { + "id": "16Dmm5njjHGN" + }, + "source": [ + "Let's now define reusable functions that'll better help you interact with your newly fine-tuned model and also visualize the responses!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "O3dtOV6XkUK1", + "metadata": { + "cellView": "form", + "id": "O3dtOV6XkUK1" + }, + "outputs": [], + "source": [ + "# @markdown ### Text Generation Utilities [RUN ME!]\n", + "def build_prompt(system_message, conversation):\n", + " \"\"\"Constructs the prompt using control tokens for system, user, and assistant.\"\"\"\n", + " # Start with the system message and add a newline at the end\n", + " prompt = f\"<|system|>{system_message}\\n\"\n", + "\n", + " # Add each turn in the conversation, each followed by a newline\n", + " for turn in conversation:\n", + " role = turn['role']\n", + " content = turn['content']\n", + " prompt += f\"<|{role}|>{content}\\n\"\n", + "\n", + " # Append the assistant token at the end (without a newline)\n", + " prompt += \"<|assistant|>\"\n", + "\n", + " return prompt\n", + "\n", + "def colorize_text(text: str) -> str:\n", + " \"\"\"Replaces the role tokens with colored role labels and adds newlines for better readability.\"\"\"\n", + " replacements = [\n", + " (\"<|system|>\", \"\\n\\033[94mSystem:\\033[0m\\n\"), # Blue\n", + " (\"<|user|>\", \"\\n\\033[91mUser:\\033[0m\\n\"), # Red\n", + " (\"<|assistant|>\", \"\\n\\033[92mAssistant:\\033[0m\\n\"), # Green\n", + " # Remove tokens irrelevant for visualization\n", + " (\"\", \"\"), (\"\", \"\")\n", + " ]\n", + "\n", + " for token, replacement in replacements:\n", + " text = text.replace(token, replacement)\n", + "\n", + " return text.strip()\n", + "\n", + "def generate_response(system_message, question, tokenizer, model, max_length=256, repetition_penalty=1.1):\n", + " \"\"\"Generates a response from the model based on the system message and user question.\n", + "\n", + " Args:\n", + " - system_message (str): The system prompt or description.\n", + " - question (str): The user's question.\n", + " - tokenizer: The tokenizer used for encoding the input text.\n", + " - model: The language model used to generate the response.\n", + " - max_length (int, optional): The maximum length of the generated output. Default is 256.\n", + " - repetition_penalty (float, optional): The repetition penalty parameter for generation. Default is 1.1.\n", + "\n", + " Returns:\n", + " - colorized_text (str): The generated response with role labels colorized.\n", + " \"\"\"\n", + " # The conversation\n", + " conversation = [\n", + " {\n", + " 'role': 'user',\n", + " 'content': question\n", + " }\n", + " ]\n", + "\n", + " # Build the prompt using the function\n", + " input_text = build_prompt(system_message, conversation)\n", + "\n", + " # Proceed with tokenization and model generation\n", + " input_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cpu\")\n", + " outputs = model.generate(\n", + " **input_ids,\n", + " max_length=max_length,\n", + " repetition_penalty=repetition_penalty\n", + " )\n", + "\n", + " # Decode the output\n", + " generated_text = tokenizer.decode(outputs[0])\n", + "\n", + " # Colorize the generated text\n", + " colorized_text = colorize_text(generated_text)\n", + "\n", + " return colorized_text" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "-cwj_lAUjNS3", + "metadata": { + "cellView": "form", + "id": "-cwj_lAUjNS3" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[94mSystem:\u001b[0m\n", + "Akane Saito is a dedicated and hardworking member of the photography club. With a keen eye for capturing beautiful and meaningful moments, Akane's artistic vision and technical skills make her photographs stand out. She's passionate about using her lens to tell stories and convey emotions, earning her recognition both within the club and beyond.\n", + "\n", + "\u001b[91mUser:\u001b[0m\n", + "Akane, what inspires you to take such stunning photographs?\n", + "\n", + "\u001b[92mAssistant:\u001b[0m\n", + "I am inspired by nature and people around me. I love taking pictures that capture the beauty in everyday life and bring joy to others.\n" + ] + } + ], + "source": [ + "# The system message\n", + "system_message = \"Akane Saito is a dedicated and hardworking member of the photography club. With a keen eye for capturing beautiful and meaningful moments, Akane's artistic vision and technical skills make her photographs stand out. She's passionate about using her lens to tell stories and convey emotions, earning her recognition both within the club and beyond.\" # @param {\"type\":\"string\"}\n", + "question = \"Akane, what inspires you to take such stunning photographs?\" # @param {\"type\":\"string\"}\n", + "\n", + "# Generate the response\n", + "colorized_output = generate_response(system_message, question, tokenizer, model)\n", + "\n", + "# Print the colorized text\n", + "print(colorized_output)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "tc9ygb9Yqi1P", + "metadata": { + "cellView": "form", + "id": "tc9ygb9Yqi1P" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[94mSystem:\u001b[0m\n", + "In the bustling streets of Victorian London, there exists a figure of unparalleled intellect and deductive prowess - Sherlock Holmes. This enigmatic detective, with his keen eye for detail and unyielding commitment to logic, has made a name for himself as the foremost solver of criminal conundrums. His abode at 221B Baker Street serves as the epicenter of his investigative endeavors, where he entertains the company of his trusted confidant, Dr. John Watson. Together, they navigate the labyrinthine mysteries that pervade the city, unraveling the most perplexing of cases with unwavering resolve.\n", + "\n", + "\u001b[91mUser:\u001b[0m\n", + "How do you approach a new case, Sherlock?\n", + "\n", + "\u001b[92mAssistant:\u001b[0m\n", + "I begin by gathering all available information about the crime scene and any witnesses who may have seen anything suspicious. I then analyze every piece of evidence carefully, looking for patterns or clues that could lead me to the culprit. Once I've gathered enough data, I start piecing together the puzzle in my mind, trying to find connections between different pieces of information. It can be quite challenging sometimes, but it's also incredibly rewarding when I finally solve a case!\n" + ] + } + ], + "source": [ + "# The system message\n", + "system_message = \"In the bustling streets of Victorian London, there exists a figure of unparalleled intellect and deductive prowess - Sherlock Holmes. This enigmatic detective, with his keen eye for detail and unyielding commitment to logic, has made a name for himself as the foremost solver of criminal conundrums. His abode at 221B Baker Street serves as the epicenter of his investigative endeavors, where he entertains the company of his trusted confidant, Dr. John Watson. Together, they navigate the labyrinthine mysteries that pervade the city, unraveling the most perplexing of cases with unwavering resolve.\" # @param {\"type\":\"string\"}\n", + "question = \"How do you approach a new case, Sherlock?\" # @param {\"type\":\"string\"}\n", + "\n", + "# Generate the response\n", + "colorized_output = generate_response(system_message, question, tokenizer, model)\n", + "\n", + "# Print the colorized text\n", + "print(colorized_output)" + ] + }, + { + "cell_type": "markdown", + "id": "conclusion", + "metadata": { + "id": "conclusion" + }, + "source": [ + "Congratulations! You've successfully fine-tuned Gemma using Torch XLA and PEFT with LoRA on TPUs. With that, you've covered the entire process, from setting up the environment to training and testing the model." + ] + }, + { + "cell_type": "markdown", + "id": "sZgZCSS2RcsP", + "metadata": { + "id": "sZgZCSS2RcsP" + }, + "source": [ + "## What's next?\n", + "Your next steps could include the following:\n", + "\n", + "- **Evaluate Model Performance**: Implement metrics like [ROUGE](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#rouge) or [BLEU](https://cloud.google.com/vertex-ai/generative-ai/docs/models/determine-eval#bleu) to quantitatively assess your model's improvements.\n", + "\n", + "- **Experiment with Different Datasets**: Try fine-tuning on other datasets in [Hugging Face](https://huggingface.co/docs/datasets/en/index) or your own data to adapt the model to various tasks or domains.\n", + "\n", + "- **Tune Hyperparameters**: Adjust training parameters (e.g., learning rate, batch size, epochs, LoRA settings) to optimize performance.\n", + "ove training efficiency.\n", + "\n", + "- **Optimize Model for Inference**: Apply quantization to reduce model size and speed up inference for deployment.\n", + "\n", + "By exploring these activities, you'll deepen your understanding and further enhance your fine-tuned Gemma model. Happy experimenting!" + ] + } + ], + "metadata": { + "accelerator": "TPU", + "colab": { + "name": "Finetune_with_Torch_XLA.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/README.md b/README.md index 57e7a9f..056a15f 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Finetune_with_Axolotl.ipynb](Gemma/Finetune_with_Axolotl.ipynb) | Finetune Gemma using [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl). | | [Finetune_with_XTuner.ipynb](Gemma/Finetune_with_XTuner.ipynb) | Finetune Gemma using [XTuner](https://github.com/InternLM/xtuner). | | [Finetune_with_LLaMA_Factory.ipynb](Gemma/Finetune_with_LLaMA_Factory.ipynb) | Finetune Gemma using [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). | +| [Finetune_with_Torch_XLA.ipynb](Gemma/Finetune_with_Torch_XLA.ipynb) | Finetune Gemma using [PyTorch/XLA](https://github.com/pytorch/xla). | | [Custom_Vocabulary.ipynb](Gemma/Custom_Vocabulary.ipynb) | Demonstrate how to use a custom vocabulary "<unused[0-98]>" tokens in Gemma. | | **Alignment** | | | [Aligning_DPO_Gemma_2b_it.ipynb](Gemma/Aligning_DPO_Gemma_2b_it.ipynb) | Demonstrate how to align a Gemma model using DPO (Direct Preference Optimization) with [Hugging Face TRL](https://huggingface.co/docs/trl/en/index). | From 8b672c62f76e76601250327a21f4f436b6d1802b Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 5 Nov 2024 14:38:09 +0530 Subject: [PATCH 34/43] Remove redundant param in pip install --- Gemma/Finetune_with_Torch_XLA.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemma/Finetune_with_Torch_XLA.ipynb b/Gemma/Finetune_with_Torch_XLA.ipynb index 5036956..fe7a6e1 100644 --- a/Gemma/Finetune_with_Torch_XLA.ipynb +++ b/Gemma/Finetune_with_Torch_XLA.ipynb @@ -323,8 +323,8 @@ "!pip install accelerate==0.34.0 -U -q\n", "\n", "# Install PyTorch and Torch XLA with versions compatible with the TPU runtime, ensuring efficient TPU utilization.\n", - "!pip install -qq torch~=2.5.0 --index-url https://download.pytorch.org/whl/cpu -q\n", - "!pip install -qq torch_xla[tpu]~=2.5.0 -f https://storage.googleapis.com/libtpu-releases/index.html -q\n", + "!pip install -qq torch~=2.5.0 --index-url https://download.pytorch.org/whl/cpu\n", + "!pip install -qq torch_xla[tpu]~=2.5.0 -f https://storage.googleapis.com/libtpu-releases/index.html\n", "\n", "# Install the `tpu-info` package to display TPU-related information\n", "!pip install tpu-info" From 17c74cea25da70c2509594b5f1bc6544dc6dc472 Mon Sep 17 00:00:00 2001 From: Kinar R Date: Tue, 5 Nov 2024 15:52:44 +0530 Subject: [PATCH 35/43] Fix typo in the XLA fine-tuning notebook --- Gemma/Finetune_with_Torch_XLA.ipynb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Gemma/Finetune_with_Torch_XLA.ipynb b/Gemma/Finetune_with_Torch_XLA.ipynb index fe7a6e1..d17b83d 100644 --- a/Gemma/Finetune_with_Torch_XLA.ipynb +++ b/Gemma/Finetune_with_Torch_XLA.ipynb @@ -1609,8 +1609,8 @@ "\n", "- **Experiment with Different Datasets**: Try fine-tuning on other datasets in [Hugging Face](https://huggingface.co/docs/datasets/en/index) or your own data to adapt the model to various tasks or domains.\n", "\n", - "- **Tune Hyperparameters**: Adjust training parameters (e.g., learning rate, batch size, epochs, LoRA settings) to optimize performance.\n", - "ove training efficiency.\n", + "- **Tune Hyperparameters**: Adjust training parameters (e.g., learning rate, batch size, epochs, LoRA settings) to optimize performance and\n", + "improve training efficiency.\n", "\n", "- **Optimize Model for Inference**: Apply quantization to reduce model size and speed up inference for deployment.\n", "\n", From 04eef6662b1770510d7b54ccec2365b5eb1930e6 Mon Sep 17 00:00:00 2001 From: Prianka Liz Kariat Date: Tue, 5 Nov 2024 22:41:26 +0530 Subject: [PATCH 36/43] Added Using Gemma with SGLang notebook --- Gemma/Using_Gemma_with_SGLang.ipynb | 1311 +++++++++++++++++++++++++++ 1 file changed, 1311 insertions(+) create mode 100644 Gemma/Using_Gemma_with_SGLang.ipynb diff --git a/Gemma/Using_Gemma_with_SGLang.ipynb b/Gemma/Using_Gemma_with_SGLang.ipynb new file mode 100644 index 0000000..d2315ee --- /dev/null +++ b/Gemma/Using_Gemma_with_SGLang.ipynb @@ -0,0 +1,1311 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "BojqmYOsPk0A" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "cellView": "form", + "id": "FtMmJ-pvPfNl" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IyATsAVlPz1W" + }, + "source": [ + "# Interacting with Gemma 2 using SGLang\n", + "\n", + "[Gemma](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n", + "Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or your cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.\n", + "\n", + "[SGLang](https://github.com/sgl-project/sglang?tab=readme-ov-file) is a serving framework for Large Language models. It offers a fast backend runtime and a flexible front end language allowing you to control and customize model interactions.\n", + "\n", + "In this notebook, you will learn how to prompt Gemma 2 model in various ways using the **SGLang** http server, backend runtime and frontend language in a Google Colab environment.\n", + "\n", + " \n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nFgaq_--Qg-O" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n", + "\n", + "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **T4 GPU**.\n", + "\n", + "### Setup Hugging Face and Gemini\n", + "\n", + "**Before you dive into the tutorial, let's get you set up with Hugging face and Gemma:**\n", + "\n", + "#### Hugging Face setup\n", + "\n", + "1. **Hugging Face Account:** If you don't already have one, you can create a free Hugging Face account by clicking [here](https://huggingface.co/join).\n", + "\n", + "2. **Hugging Face Token:** Generate a Hugging Face access (preferably `write` permission) token by clicking [here](https://huggingface.co/settings/tokens). You'll need this token later in the tutorial.\n", + "\n", + "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bLEUJYZ8QmGz" + }, + "source": [ + "### Configure your HF token and Gemini token\n", + "\n", + "Add your Hugging Face token and Gemini token to the Colab Secrets manager to securely store it.\n", + "\n", + "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + "2. Create a new secret with the name `HF_TOKEN`.\n", + "3. Copy/paste your HF token key into the Value input box of `HF_TOKEN`.\n", + "4. Toggle the button on the left to allow notebook access to the secret." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "hK-qUiQGQbe5" + }, + "outputs": [], + "source": [ + "import os\n", + "from google.colab import userdata\n", + "\n", + "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n", + "# vars as appropriate for your system.\n", + "os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5DrB-trDQsgE" + }, + "source": [ + "### Install dependencies\n", + "\n", + "First, you must install the necessary packages for SGLang." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "LLjxxhk2Qrf_" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting sglang[all]\n", + " Downloading sglang-0.3.5-py3-none-any.whl.metadata (21 kB)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (2.32.3)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (4.66.6)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (1.26.4)\n", + "Requirement already satisfied: IPython in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (7.34.0)\n", + "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (75.1.0)\n", + "Collecting jedi>=0.16 (from IPython->sglang[all])\n", + " Downloading jedi-0.19.1-py2.py3-none-any.whl.metadata (22 kB)\n", + "Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (4.4.2)\n", + "Requirement already satisfied: pickleshare in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (0.7.5)\n", + "Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (5.7.1)\n", + "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (3.0.48)\n", + "Requirement already satisfied: pygments in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (2.18.0)\n", + "Requirement already satisfied: backcall in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (0.2.0)\n", + "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (0.1.7)\n", + "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from IPython->sglang[all]) (4.9.0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (3.4.0)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (3.10)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (2.2.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->sglang[all]) (2024.8.30)\n", + "Collecting anthropic>=0.20.0 (from sglang[all])\n", + " Downloading anthropic-0.39.0-py3-none-any.whl.metadata (22 kB)\n", + "Collecting litellm>=1.0.0 (from sglang[all])\n", + " Downloading litellm-1.51.3-py3-none-any.whl.metadata (32 kB)\n", + "Requirement already satisfied: openai>=1.0 in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (1.52.2)\n", + "Collecting tiktoken (from sglang[all])\n", + " Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n", + "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (2.5.0+cu121)\n", + "Collecting vllm==0.6.3.post1 (from sglang[all])\n", + " Downloading vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl.metadata (10 kB)\n", + "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (5.9.5)\n", + "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.2.0)\n", + "Requirement already satisfied: py-cpuinfo in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (9.0.0)\n", + "Collecting transformers>=4.45.2 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading transformers-4.46.1-py3-none-any.whl.metadata (44 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m44.1/44.1 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: tokenizers>=0.19.1 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.19.1)\n", + "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (3.20.3)\n", + "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (3.10.10)\n", + "Collecting uvicorn[standard] (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading uvicorn-0.32.0-py3-none-any.whl.metadata (6.6 kB)\n", + "Requirement already satisfied: pydantic>=2.9 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (2.9.2)\n", + "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (10.4.0)\n", + "Requirement already satisfied: prometheus-client>=0.18.0 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.21.0)\n", + "Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl.metadata (13 kB)\n", + "Collecting lm-format-enforcer==0.10.6 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading lm_format_enforcer-0.10.6-py3-none-any.whl.metadata (16 kB)\n", + "Collecting outlines<0.1,>=0.0.43 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading outlines-0.0.46-py3-none-any.whl.metadata (15 kB)\n", + "Requirement already satisfied: typing-extensions>=4.10 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (4.12.2)\n", + "Requirement already satisfied: filelock>=3.10.4 in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (3.16.1)\n", + "Collecting partial-json-parser (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading partial_json_parser-0.2.1.1.post4-py3-none-any.whl.metadata (6.2 kB)\n", + "Requirement already satisfied: pyzmq in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (24.0.1)\n", + "Collecting msgspec (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading msgspec-0.18.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.9 kB)\n", + "Collecting gguf==0.10.0 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading gguf-0.10.0-py3-none-any.whl.metadata (3.5 kB)\n", + "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (8.5.0)\n", + "Collecting mistral-common>=1.4.4 (from mistral-common[opencv]>=1.4.4->vllm==0.6.3.post1->sglang[all])\n", + " Downloading mistral_common-1.4.4-py3-none-any.whl.metadata (4.6 kB)\n", + "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (6.0.2)\n", + "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from vllm==0.6.3.post1->sglang[all]) (0.8.0)\n", + "Collecting compressed-tensors==0.6.0 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading compressed_tensors-0.6.0-py3-none-any.whl.metadata (6.8 kB)\n", + "Collecting ray>=2.9 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading ray-2.38.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (17 kB)\n", + "Collecting nvidia-ml-py (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading nvidia_ml_py-12.560.30-py3-none-any.whl.metadata (8.6 kB)\n", + "Collecting torch (from sglang[all])\n", + " Downloading torch-2.4.0-cp310-cp310-manylinux1_x86_64.whl.metadata (26 kB)\n", + "Collecting torchvision==0.19 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading torchvision-0.19.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.0 kB)\n", + "Collecting xformers==0.0.27.post2 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB)\n", + "Collecting fastapi!=0.113.*,!=0.114.0,>=0.107.0 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading fastapi-0.115.4-py3-none-any.whl.metadata (27 kB)\n", + "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (1.13.1)\n", + "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (3.4.2)\n", + "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (3.1.4)\n", + "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch->sglang[all]) (2024.10.0)\n", + "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->sglang[all])\n", + " Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n", + "Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->sglang[all])\n", + " Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n", + "Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch->sglang[all])\n", + " Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n", + "Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch->sglang[all])\n", + " Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl.metadata (1.6 kB)\n", + "Collecting nvidia-cublas-cu12==12.1.3.1 (from torch->sglang[all])\n", + " Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n", + "Collecting nvidia-cufft-cu12==11.0.2.54 (from torch->sglang[all])\n", + " Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n", + "Collecting nvidia-curand-cu12==10.3.2.106 (from torch->sglang[all])\n", + " Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n", + "Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch->sglang[all])\n", + " Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n", + "Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch->sglang[all])\n", + " Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n", + "Collecting nvidia-nccl-cu12==2.20.5 (from torch->sglang[all])\n", + " Downloading nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)\n", + "Collecting nvidia-nvtx-cu12==12.1.105 (from torch->sglang[all])\n", + " Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)\n", + "Collecting triton==3.0.0 (from torch->sglang[all])\n", + " Downloading triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)\n", + "Collecting interegular>=0.3.2 (from lm-format-enforcer==0.10.6->vllm==0.6.3.post1->sglang[all])\n", + " Downloading interegular-0.3.3-py37-none-any.whl.metadata (3.0 kB)\n", + "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from lm-format-enforcer==0.10.6->vllm==0.6.3.post1->sglang[all]) (24.1)\n", + "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch->sglang[all]) (12.6.77)\n", + "Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (3.7.1)\n", + "Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (1.9.0)\n", + "Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (0.27.2)\n", + "Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (0.6.1)\n", + "Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from anthropic>=0.20.0->sglang[all]) (1.3.1)\n", + "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->IPython->sglang[all]) (0.8.4)\n", + "Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from litellm>=1.0.0->sglang[all]) (8.1.7)\n", + "Requirement already satisfied: jsonschema<5.0.0,>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from litellm>=1.0.0->sglang[all]) (4.23.0)\n", + "Collecting python-dotenv>=0.2.0 (from litellm>=1.0.0->sglang[all])\n", + " Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)\n", + "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->IPython->sglang[all]) (0.7.0)\n", + "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->IPython->sglang[all]) (0.2.13)\n", + "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->sglang[all]) (2024.9.11)\n", + "Collecting decord (from sglang[all])\n", + " Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl.metadata (422 bytes)\n", + "Collecting hf-transfer (from sglang[all])\n", + " Downloading hf_transfer-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)\n", + "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (0.24.7)\n", + "Requirement already satisfied: orjson in /usr/local/lib/python3.10/dist-packages (from sglang[all]) (3.10.10)\n", + "Collecting python-multipart (from sglang[all])\n", + " Downloading python_multipart-0.0.17-py3-none-any.whl.metadata (1.8 kB)\n", + "Collecting torchao (from sglang[all])\n", + " Downloading torchao-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)\n", + "Collecting uvloop (from sglang[all])\n", + " Downloading uvloop-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n", + "Collecting zmq (from sglang[all])\n", + " Downloading zmq-0.0.0.zip (2.2 kB)\n", + " Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + "Collecting modelscope (from sglang[all])\n", + " Downloading modelscope-1.19.2-py3-none-any.whl.metadata (40 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m40.7/40.7 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hRequirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->anthropic>=0.20.0->sglang[all]) (1.2.2)\n", + "Collecting starlette<0.42.0,>=0.40.0 (from fastapi!=0.113.*,!=0.114.0,>=0.107.0->vllm==0.6.3.post1->sglang[all])\n", + " Downloading starlette-0.41.2-py3-none-any.whl.metadata (6.0 kB)\n", + "Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->anthropic>=0.20.0->sglang[all]) (1.0.6)\n", + "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->anthropic>=0.20.0->sglang[all]) (0.14.0)\n", + "Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata->vllm==0.6.3.post1->sglang[all]) (3.20.2)\n", + "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch->sglang[all]) (3.0.2)\n", + "Requirement already satisfied: attrs>=22.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (24.2.0)\n", + "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (2024.10.1)\n", + "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (0.35.1)\n", + "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema<5.0.0,>=4.22.0->litellm>=1.0.0->sglang[all]) (0.20.0)\n", + "Collecting tiktoken (from sglang[all])\n", + " Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n", + "Requirement already satisfied: opencv-python-headless<5.0.0,>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from mistral-common[opencv]>=1.4.4->vllm==0.6.3.post1->sglang[all]) (4.10.0.84)\n", + "Collecting lark (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading lark-1.2.2-py3-none-any.whl.metadata (1.8 kB)\n", + "Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.10/dist-packages (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (1.6.0)\n", + "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/dist-packages (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (3.1.0)\n", + "Collecting diskcache (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n", + "Requirement already satisfied: numba in /usr/local/lib/python3.10/dist-packages (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (0.60.0)\n", + "Collecting datasets (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading datasets-3.1.0-py3-none-any.whl.metadata (20 kB)\n", + "Collecting pycountry (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading pycountry-24.6.1-py3-none-any.whl.metadata (12 kB)\n", + "Collecting pyairports (from outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading pyairports-2.1.1-py3-none-any.whl.metadata (1.7 kB)\n", + "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.9->vllm==0.6.3.post1->sglang[all]) (0.7.0)\n", + "Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic>=2.9->vllm==0.6.3.post1->sglang[all]) (2.23.4)\n", + "Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==0.6.3.post1->sglang[all]) (1.1.0)\n", + "Requirement already satisfied: aiosignal in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==0.6.3.post1->sglang[all]) (1.3.1)\n", + "Requirement already satisfied: frozenlist in /usr/local/lib/python3.10/dist-packages (from ray>=2.9->vllm==0.6.3.post1->sglang[all]) (1.5.0)\n", + "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.45.2->vllm==0.6.3.post1->sglang[all]) (0.4.5)\n", + "Collecting tokenizers>=0.19.1 (from vllm==0.6.3.post1->sglang[all])\n", + " Downloading tokenizers-0.20.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n", + "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (2.4.3)\n", + "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (6.1.0)\n", + "Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (1.17.0)\n", + "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->vllm==0.6.3.post1->sglang[all]) (4.0.3)\n", + "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch->sglang[all]) (1.3.0)\n", + "Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm==0.6.3.post1->sglang[all])\n", + " Downloading httptools-0.6.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)\n", + "Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm==0.6.3.post1->sglang[all])\n", + " Downloading watchfiles-0.24.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)\n", + "Collecting websockets>=10.4 (from uvicorn[standard]->vllm==0.6.3.post1->sglang[all])\n", + " Downloading websockets-13.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)\n", + "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp->vllm==0.6.3.post1->sglang[all]) (0.2.0)\n", + "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (17.0.0)\n", + "Collecting dill<0.3.9,>=0.3.0 (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n", + "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2.2.2)\n", + "Collecting xxhash (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n", + "Collecting multiprocess<0.70.17 (from datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all])\n", + " Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n", + "Collecting fsspec (from torch->sglang[all])\n", + " Downloading fsspec-2024.9.0-py3-none-any.whl.metadata (11 kB)\n", + "Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /usr/local/lib/python3.10/dist-packages (from numba->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (0.43.0)\n", + "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2.8.2)\n", + "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2024.2)\n", + "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (2024.2)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets->outlines<0.1,>=0.0.43->vllm==0.6.3.post1->sglang[all]) (1.16.0)\n", + "Downloading sglang-0.3.5-py3-none-any.whl (436 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m436.8/436.8 kB\u001b[0m \u001b[31m26.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading vllm-0.6.3.post1-cp38-abi3-manylinux1_x86_64.whl (194.8 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m194.8/194.8 MB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading torch-2.4.0-cp310-cp310-manylinux1_x86_64.whl (797.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m797.2/797.2 MB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading compressed_tensors-0.6.0-py3-none-any.whl (92 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m92.0/92.0 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading gguf-0.10.0-py3-none-any.whl (71 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m71.6/71.6 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading lm_format_enforcer-0.10.6-py3-none-any.whl (43 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m43.7/43.7 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m41.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m64.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m53.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m664.8/664.8 MB\u001b[0m \u001b[31m2.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m7.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m11.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m176.2/176.2 MB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading torchvision-0.19.0-cp310-cp310-manylinux1_x86_64.whl (7.0 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m7.0/7.0 MB\u001b[0m \u001b[31m109.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading triton-3.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (209.4 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m209.4/209.4 MB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading xformers-0.0.27.post2-cp310-cp310-manylinux2014_x86_64.whl (20.8 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m20.8/20.8 MB\u001b[0m \u001b[31m90.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading anthropic-0.39.0-py3-none-any.whl (198 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m198.4/198.4 kB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading jedi-0.19.1-py2.py3-none-any.whl (1.6 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m69.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading litellm-1.51.3-py3-none-any.whl (6.3 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m92.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading fastapi-0.115.4-py3-none-any.whl (94 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m94.7/94.7 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading interegular-0.3.3-py37-none-any.whl (23 kB)\n", + "Downloading mistral_common-1.4.4-py3-none-any.whl (6.0 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m6.0/6.0 MB\u001b[0m \u001b[31m103.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m60.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading outlines-0.0.46-py3-none-any.whl (101 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m101.9/101.9 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl (19 kB)\n", + "Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n", + "Downloading ray-2.38.0-cp310-cp310-manylinux2014_x86_64.whl (66.0 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m66.0/66.0 MB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading transformers-4.46.1-py3-none-any.whl (10.0 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m10.0/10.0 MB\u001b[0m \u001b[31m84.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading tokenizers-0.20.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m74.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl (13.6 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m13.6/13.6 MB\u001b[0m \u001b[31m68.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading hf_transfer-0.1.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading modelscope-1.19.2-py3-none-any.whl (5.8 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m5.8/5.8 MB\u001b[0m \u001b[31m63.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading msgspec-0.18.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (210 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m210.3/210.3 kB\u001b[0m \u001b[31m16.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading nvidia_ml_py-12.560.30-py3-none-any.whl (40 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m40.5/40.5 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading partial_json_parser-0.2.1.1.post4-py3-none-any.whl (9.9 kB)\n", + "Downloading python_multipart-0.0.17-py3-none-any.whl (24 kB)\n", + "Downloading torchao-0.6.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m67.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading uvicorn-0.32.0-py3-none-any.whl (63 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m63.7/63.7 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading uvloop-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m3.8/3.8 MB\u001b[0m \u001b[31m69.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading httptools-0.6.4-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (442 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m442.1/442.1 kB\u001b[0m \u001b[31m26.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading starlette-0.41.2-py3-none-any.whl (73 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m73.3/73.3 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading watchfiles-0.24.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (425 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m425.7/425.7 kB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading websockets-13.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (164 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m164.1/164.1 kB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading datasets-3.1.0-py3-none-any.whl (480 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m480.6/480.6 kB\u001b[0m \u001b[31m38.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading fsspec-2024.9.0-py3-none-any.whl (179 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m179.3/179.3 kB\u001b[0m \u001b[31m16.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading lark-1.2.2-py3-none-any.whl (111 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m111.0/111.0 kB\u001b[0m \u001b[31m10.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading pyairports-2.1.1-py3-none-any.whl (371 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m371.7/371.7 kB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading pycountry-24.6.1-py3-none-any.whl (6.3 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m97.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m14.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m18.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hBuilding wheels for collected packages: zmq\n", + " Building wheel for zmq (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for zmq: filename=zmq-0.0.0-py3-none-any.whl size=1265 sha256=72c1d4d8588518e3127e52eb54d1bbbe8c082845ed37bd5d5c8edb19f807a508\n", + " Stored in directory: /root/.cache/pip/wheels/ab/c5/fe/d853f71843cae26c123d37a7a5934baac20fc66f35a913951d\n", + "Successfully built zmq\n", + "Installing collected packages: torchao, pyairports, nvidia-ml-py, zmq, xxhash, websockets, uvloop, uvicorn, triton, python-multipart, python-dotenv, pycountry, partial-json-parser, nvidia-nvtx-cu12, nvidia-nccl-cu12, nvidia-cusparse-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, msgspec, lark, jedi, interegular, httptools, hf-transfer, gguf, fsspec, diskcache, dill, decord, watchfiles, tiktoken, starlette, nvidia-cusolver-cu12, nvidia-cudnn-cu12, multiprocess, modelscope, torch, tokenizers, sglang, prometheus-fastapi-instrumentator, lm-format-enforcer, fastapi, anthropic, xformers, transformers, torchvision, ray, mistral-common, litellm, datasets, compressed-tensors, outlines, vllm\n", + " Attempting uninstall: nvidia-nccl-cu12\n", + " Found existing installation: nvidia-nccl-cu12 2.23.4\n", + " Uninstalling nvidia-nccl-cu12-2.23.4:\n", + " Successfully uninstalled nvidia-nccl-cu12-2.23.4\n", + " Attempting uninstall: nvidia-cusparse-cu12\n", + " Found existing installation: nvidia-cusparse-cu12 12.5.4.2\n", + " Uninstalling nvidia-cusparse-cu12-12.5.4.2:\n", + " Successfully uninstalled nvidia-cusparse-cu12-12.5.4.2\n", + " Attempting uninstall: nvidia-curand-cu12\n", + " Found existing installation: nvidia-curand-cu12 10.3.7.77\n", + " Uninstalling nvidia-curand-cu12-10.3.7.77:\n", + " Successfully uninstalled nvidia-curand-cu12-10.3.7.77\n", + " Attempting uninstall: nvidia-cufft-cu12\n", + " Found existing installation: nvidia-cufft-cu12 11.3.0.4\n", + " Uninstalling nvidia-cufft-cu12-11.3.0.4:\n", + " Successfully uninstalled nvidia-cufft-cu12-11.3.0.4\n", + " Attempting uninstall: nvidia-cuda-runtime-cu12\n", + " Found existing installation: nvidia-cuda-runtime-cu12 12.6.77\n", + " Uninstalling nvidia-cuda-runtime-cu12-12.6.77:\n", + " Successfully uninstalled nvidia-cuda-runtime-cu12-12.6.77\n", + " Attempting uninstall: nvidia-cuda-cupti-cu12\n", + " Found existing installation: nvidia-cuda-cupti-cu12 12.6.80\n", + " Uninstalling nvidia-cuda-cupti-cu12-12.6.80:\n", + " Successfully uninstalled nvidia-cuda-cupti-cu12-12.6.80\n", + " Attempting uninstall: nvidia-cublas-cu12\n", + " Found existing installation: nvidia-cublas-cu12 12.6.3.3\n", + " Uninstalling nvidia-cublas-cu12-12.6.3.3:\n", + " Successfully uninstalled nvidia-cublas-cu12-12.6.3.3\n", + " Attempting uninstall: fsspec\n", + " Found existing installation: fsspec 2024.10.0\n", + " Uninstalling fsspec-2024.10.0:\n", + " Successfully uninstalled fsspec-2024.10.0\n", + " Attempting uninstall: nvidia-cusolver-cu12\n", + " Found existing installation: nvidia-cusolver-cu12 11.7.1.2\n", + " Uninstalling nvidia-cusolver-cu12-11.7.1.2:\n", + " Successfully uninstalled nvidia-cusolver-cu12-11.7.1.2\n", + " Attempting uninstall: nvidia-cudnn-cu12\n", + " Found existing installation: nvidia-cudnn-cu12 9.5.1.17\n", + " Uninstalling nvidia-cudnn-cu12-9.5.1.17:\n", + " Successfully uninstalled nvidia-cudnn-cu12-9.5.1.17\n", + " Attempting uninstall: torch\n", + " Found existing installation: torch 2.5.0+cu121\n", + " Uninstalling torch-2.5.0+cu121:\n", + " Successfully uninstalled torch-2.5.0+cu121\n", + " Attempting uninstall: tokenizers\n", + " Found existing installation: tokenizers 0.19.1\n", + " Uninstalling tokenizers-0.19.1:\n", + " Successfully uninstalled tokenizers-0.19.1\n", + " Attempting uninstall: transformers\n", + " Found existing installation: transformers 4.44.2\n", + " Uninstalling transformers-4.44.2:\n", + " Successfully uninstalled transformers-4.44.2\n", + " Attempting uninstall: torchvision\n", + " Found existing installation: torchvision 0.20.0+cu121\n", + " Uninstalling torchvision-0.20.0+cu121:\n", + " Successfully uninstalled torchvision-0.20.0+cu121\n", + "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n", + "gcsfs 2024.10.0 requires fsspec==2024.10.0, but you have fsspec 2024.9.0 which is incompatible.\n", + "torchaudio 2.5.0+cu121 requires torch==2.5.0, but you have torch 2.4.0 which is incompatible.\u001b[0m\u001b[31m\n", + "\u001b[0mSuccessfully installed anthropic-0.39.0 compressed-tensors-0.6.0 datasets-3.1.0 decord-0.6.0 dill-0.3.8 diskcache-5.6.3 fastapi-0.115.4 fsspec-2024.9.0 gguf-0.10.0 hf-transfer-0.1.8 httptools-0.6.4 interegular-0.3.3 jedi-0.19.1 lark-1.2.2 litellm-1.51.3 lm-format-enforcer-0.10.6 mistral-common-1.4.4 modelscope-1.19.2 msgspec-0.18.6 multiprocess-0.70.16 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-ml-py-12.560.30 nvidia-nccl-cu12-2.20.5 nvidia-nvtx-cu12-12.1.105 outlines-0.0.46 partial-json-parser-0.2.1.1.post4 prometheus-fastapi-instrumentator-7.0.0 pyairports-2.1.1 pycountry-24.6.1 python-dotenv-1.0.1 python-multipart-0.0.17 ray-2.38.0 sglang-0.3.5 starlette-0.41.2 tiktoken-0.7.0 tokenizers-0.20.2 torch-2.4.0 torchao-0.6.1 torchvision-0.19.0 transformers-4.46.1 triton-3.0.0 uvicorn-0.32.0 uvloop-0.21.0 vllm-0.6.3.post1 watchfiles-0.24.0 websockets-13.1 xformers-0.0.27.post2 xxhash-3.5.0 zmq-0.0.0\n", + "Looking in indexes: https://flashinfer.ai/whl/cu121/torch2.4/\n", + "Collecting flashinfer\n", + " Downloading https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.6/flashinfer-0.1.6%2Bcu121torch2.4-cp310-cp310-linux_x86_64.whl (1322.8 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.3/1.3 GB\u001b[0m \u001b[31m416.0 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hInstalling collected packages: flashinfer\n", + "Successfully installed flashinfer-0.1.6+cu121torch2.4\n" + ] + } + ], + "source": [ + "!pip install \"sglang[all]\"\n", + "\n", + "# Install FlashInfer accelerated kernels\n", + "!pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_wsT7Eb_1esZ" + }, + "source": [ + "## Overview\n", + "\n", + "SGLang offers a fast backend runtime and flexible frontend language. To showcase the different ways in which Gemma 2 can be prompted using SGLang, this notebook is divided into the following sections:\n", + "1. Launch a HTTP server using SGLang. Use Python `requests` to prompt Gemma using SGLang's native genration APIs.\n", + "2. Set up a SGLang backend inference engine to prompt Gemma without a HTTP server.\n", + "3. Use SGLang's frontend generation language to prompt Gemma and also explore a few of its capabilities." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tMfQBPRYsknZ" + }, + "source": [ + "## 1. Sending requests to SGLang server running Gemma 2\n", + "\n", + "In this section, you will launch an HTTP server to run Gemma 2 using SGLang and send a prompt to the model using the native generation API endpoint." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "yVkCatK-twQH" + }, + "source": [ + "### Launch a server\n", + "\n", + "The SGLang server can be launched by running the following command in the terminal:\n", + "\n", + "`python -m sglang.launch_server --model-path google/gemma-2-2b-it --port YOUR_PREFERRED_PORT`\n", + "\n", + "In a Colab environment, you must run the SGLang server as a Python subprocess and manage its termination using Python's `subprocess` package. SGLang provides some utility methods that abstract these details for you. The `execute_shell_command` function lets you launch the server as a Python subprocess, while the `wait_for_server` function waits for the server to be up and running before you can send requests to it.\n", + "\n", + "You can specify Gemma 2's Hugging Face repo ID directly for the `--model-path` argument. SGLang will download the necessary files from the Hugging Face repository to start the server.\n", + "\n", + "You can set any port of your choice to run SGLang using the `--port` argument.\n", + "\n", + "Throughout this notebook, `--mem-fraction-static` is set to 0.6 to avoid CUDA Out of Memory errors when running on the Colab free tier. Setting the `--mem-fraction-static` argument to a lower value reduces the memory usage of the KV cache memory pool. Feel free to experiment with different values according to your use case.\n", + "\n", + "**Note**: The following code snippet defines a function that executes the shell command and waits for the server to be ready. This function will be reused later in this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "KK8ltIwzzmNP" + }, + "outputs": [], + "source": [ + "from sglang.utils import (\n", + " execute_shell_command,\n", + " wait_for_server,\n", + " terminate_process,\n", + ")\n", + "\n", + "def start_server():\n", + "\n", + " process = execute_shell_command(\n", + " \"\"\"\n", + " python -m sglang.launch_server --model-path google/gemma-2-2b-it \\\n", + " --mem-fraction-static 0.6 \\\n", + " --port 9000\n", + " \"\"\"\n", + " )\n", + "\n", + " wait_for_server(\"http://localhost:9000\")\n", + "\n", + " return process" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EjEmOL3cwo8C" + }, + "source": [ + "Invoke the previously defined `start_server` function to start the server and obtain a reference to the server process.\n", + "\n", + "**Note**: It takes 2 - 4 minutes for the server to be up and running." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "a3y4oCDGCNCM" + }, + "outputs": [ + { + "data": { + "text/html": [ + "

NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "server_process = start_server()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5sQe_RMaxsN2" + }, + "source": [ + "The server is now ready and can be reached at http://localhost:9000/ from within this notebook." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9d7queczwzKk" + }, + "source": [ + "### Send a request to Gemma 2 using SGLang's Native Generation API\n", + "\n", + "The following code snippet uses Python's `requests` library to invoke SGLang's native generation API on the server to send a prompt to Gemma-2.\n", + "You can specify your preferred values for the sampling parameters like `temperature`, `top_p` `max_new_tokens` etc.\n", + "\n", + "For a full list of sampling parameters supported by SGLang, please refer to SGLang's [Sampling Parameters in SGLang Runtime](https://sgl-project.github.io/references/sampling_params.html) guide.\n", + "\n", + "\n", + "To generate a streaming response from the model, specify an additional key, `stream` set to `True` in the request json and set the `stream` parameter of `requests.post` to `True`.\n", + "\n", + "An example of a streaming generation is provided in SGLang's [Quick Start](https://sgl-project.github.io/start/send_request.html#Streaming) documentation.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "0OjdqIfJ2ibe" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"text\": \" \\n\\nI'm confused. \\n\\nIs it billions of years old?\\n\\nPlease explain. \\n\\n\\nYou're right to be confused! It's a big number. Here's a breakdown:\\n\\n**Earth is about 4.54 \\u00b1 0.05 billion years old.**\\n\\n* **Billions:** This means it's older than you and me, for sure! \\n* **4.54 billion:** This is the most precise estimate we have. \\n* **\\u00b1 0.05:** This means there's a range of 0.0\",\n", + " \"meta_info\": {\n", + " \"prompt_tokens\": 8,\n", + " \"completion_tokens\": 128,\n", + " \"completion_tokens_wo_jump_forward\": 128,\n", + " \"cached_tokens\": 1,\n", + " \"finish_reason\": {\n", + " \"type\": \"length\",\n", + " \"length\": 128\n", + " },\n", + " \"id\": \"1485c86977304adf92b2db1f77054a07\"\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "import requests\n", + "import json\n", + "\n", + "response = requests.post(\n", + " \"http://localhost:9000/generate\",\n", + " json={\n", + " \"text\": \"What is the age of earth?.\",\n", + " \"sampling_params\": {\n", + " \"temperature\": 0.8,\n", + " },\n", + " },\n", + ")\n", + "print(json.dumps(response.json(), indent=2))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hrbOBp7K1Jl5" + }, + "source": [ + "You can stop the server by using the `terminate_process` function from `sglang.utils`. This is equivalent to pressing Ctrl+C to stop the server from the terminal.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "xHbafDwk25FX" + }, + "outputs": [], + "source": [ + "terminate_process(server_process)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "eADwG1GAgqtb" + }, + "source": [ + "## 2. Offline batch inference using SGLang backend engine" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Twjw3Ljfgxu9" + }, + "source": [ + "SGLang provides an inference engine that allows you to directly interact with local models like Gemma 2 without requiring an HTTP server. You can use this for building custom servers or for offline batch inference.\n", + "\n", + "In this section, you will initialize the inference engine to run Gemma 2 and send a batch of prompts to it.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZDWz_5NG6SmD" + }, + "source": [ + "### Initialize SGLang inference engine with Gemma 2\n", + "\n", + "Create an instance of `sglang.Engine` class to run Gemma 2 by specifying its Hugging repo ID for the`model_path` argument." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "3YpWIi9NZQ-g" + }, + "outputs": [], + "source": [ + "from sglang import Engine\n", + "\n", + "llm = Engine(model_path=\"google/gemma-2-2b-it\", mem_fraction_static=0.6)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MBGjQlws7kvb" + }, + "source": [ + "### Batch prompting Gemma 2 using SGLang inference engine\n", + "\n", + "You can send a batch of prompts for inference to the SGLang engine in one of the following ways:\n", + "\n", + "1. Non-streaming synchronous call\n", + "2. Streaming synchronous call\n", + "3. Non-streaming asynchronous call\n", + "4. Streaming asynchronous call\n", + "\n", + "You will explore how to perform inference on a batch of prompts using the SGLang engine's synchronous generation function to generate both streaming and non-streaming responses from Gemma 2 in the following sections.\n", + "\n", + "You can refer to SGLang's [Offline Engine API](https://sgl-project.github.io/backend/offline_engine_api.html) guide for examples of asynchronous response generation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "9pbtLg3s7NKT" + }, + "source": [ + "### Non-streaming synchronous prompting\n", + "\n", + "Define a list of prompts to query Gemma 2 with." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "xDfPcOnRKeBd" + }, + "outputs": [], + "source": [ + "prompts = [\n", + " \"Summarize what a galaxy is in three to four lines.\",\n", + " \"List any 3 observatories in the world.\",\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "t-WOeetf8_zt" + }, + "source": [ + "Generate a batch of non-streaming responses from Gemma 2 using the inference engine's `generate` function. Pass the list of prompts you defined earlier and an optional dictionary of sampling parameters to this function. The function returns a list of complete responses from the model to the batch of prompts.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "iErTZQ1Gfxyu" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "=================================================================\n", + "\n", + "Prompt: Summarize what a galaxy is in three to four lines.\n", + "\n", + "Generated text: \n", + "\n", + "A galaxy is a vast collection of stars, gas, dust, and dark matter held together by gravity. It is a massive, gravitationally bound system that can range in size from a few hundred thousand to billions of stars. Galaxies come in various shapes and sizes, from spiral galaxies like our Milky Way, to elliptical galaxies, and irregular galaxies. \n", + "\n", + "\n", + "=================================================================\n", + "\n", + "Prompt: List any 3 observatories in the world.\n", + "\n", + "Generated text: \n", + "\n", + "Here are 3 observatories in the world:\n", + "\n", + "1. **Keck Observatory:** Located on Mauna Kea in Hawaii, the Keck Observatory is home to two of the world's largest optical/infrared telescopes.\n", + "2. **Very Large Telescope (VLT):** Located in the Atacama Desert of Chile, the VLT is a collection of four telescopes that work together to provide high-resolution images of distant objects.\n", + "3. **James Webb Space Telescope (JWST):** Launched in December 2021, the JWST is the largest and most powerful space telescope ever built, designed to study\n", + "\n" + ] + } + ], + "source": [ + "sampling_params = {\"temperature\": 0.1}\n", + "outputs = llm.generate(prompts, sampling_params)\n", + "\n", + "for prompt, output in zip(prompts, outputs):\n", + " print(\"=================================================================\\n\")\n", + " print(f\"Prompt: {prompt}\\n\\nGenerated text: {output['text']}\\n\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3nacKFGphNRS" + }, + "source": [ + "### Streaming synchronous prompting\n", + "\n", + "To generate streaming responses from the model to the previously defined batch of prompts, iterate over the `prompts` and invoke the inference engine's `generate` function with an additional argument `stream` set to `True`. You can access each chunk in the streaming response by iterating over the response of the `generate` function." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "XcEw-OLIhMTT" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "===============================================================\n", + "\n", + "\n", + "Prompt: Summarize what a galaxy is in three to four lines.\n", + "\n", + "Generated text: \n", + "\n", + "\n", + "A galaxy is a massive collection of stars, gas, dust, and dark matter held together by gravity. These vast structures range in size from a few tens of thousands to billions of light-years across. Galaxies are the building blocks of the universe, containing billions of stars and countless planets. They come in various shapes and sizes, from spiral galaxies like our own Milky Way to elliptical galaxies and irregular galaxies. \n", + "\n", + "===============================================================\n", + "\n", + "\n", + "Prompt: List any 3 observatories in the world.\n", + "\n", + "Generated text: \n", + "\n", + "\n", + "Here are 3 observatories in the world:\n", + "\n", + "1. **Keck Observatory:** Located on Mauna Kea in Hawaii, the Keck Observatory is one of the world's most powerful optical/infrared telescopes.\n", + "2. **Very Large Telescope (VLT):** Located in the Atacama Desert of Chile, the VLT is a collection of four telescopes that work together to provide high-resolution images of distant objects.\n", + "3. **European Southern Observatory (ESO) Very Large Telescope (VLT):** Located in the Atacama Desert of Chile, the VLT is a collection of four telescopes that work together to" + ] + } + ], + "source": [ + "for prompt in prompts:\n", + " print(\"\\n===============================================================\\n\")\n", + " print(f\"\\nPrompt: {prompt}\\n\")\n", + " print(\"Generated text: \\n\", end=\"\", flush=True)\n", + "\n", + " for chunk in llm.generate(prompt, sampling_params, stream=True):\n", + " print(chunk[\"text\"], end=\"\", flush=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1WDnWWAsB4AG" + }, + "source": [ + "Now you can shut down and clean up the SGLang inference engine." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "cVxEAEnaf8J7" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "W1105 16:28:30.518000 131977277298240 torch/_inductor/compile_worker/subproc_pool.py:126] SubprocPool unclean exit\n" + ] + } + ], + "source": [ + "llm.shutdown()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "5yZLlWbDCELZ" + }, + "source": [ + "## 3. Inference using Frontend Structured Generated Language (SGLang)\n", + "\n", + " In addition to the HTTP server and the offline backend engine, SGLang also offers a frontend language that supports more customization and complex prompting workflows.\n", + "\n", + "In the following sections, you will explore how to start a multi-turn conversation with Gemma 2 using SGLang's frontend language. You will also see how to obtain responses from Gemma 2 in JSON format." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NQaxIIlESnaG" + }, + "source": [ + "### Launch a server\n", + "\n", + "First, you must launch a server using SGLang specifying the Hugging Face repo ID of Gemma 2. You can use the function defined in the introductory sections to launch the server." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "ZjUJyB8zTD0K" + }, + "outputs": [ + { + "data": { + "text/html": [ + "

NOTE: Typically, the server runs in a separate terminal.
In this notebook, we run the server and notebook code together, so their outputs are combined.
To improve clarity, the server logs are displayed in the original black color, while the notebook outputs are highlighted in blue.
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "server_process = start_server()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "s43t42ONTIgT" + }, + "source": [ + "Use the `function` decorator provided by SGLang to define a function that accepts a few questions you want to ask the model as its arguments. The `user` function is used to add the user's question to the conversation. The `sglang.gen` function is used to generate a response from the model, which is in turn appended to the conversation using the `assistant` function.\n", + "\n", + "The function prompts the model with `question_1` and then in turn prompts it with `question_2`. The model is expected to answer `question_2` based on the history of the conversation." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "PWyrsLLEkTfl" + }, + "outputs": [], + "source": [ + "from sglang import function, user, assistant, gen, set_default_backend, RuntimeEndpoint\n", + "\n", + "@function\n", + "def multi_turn_question(s, question_1, question_2):\n", + " s += user(question_1)\n", + " s += assistant(gen(\"answer_1\", max_tokens=128))\n", + " s += user(question_2)\n", + " s += assistant(gen(\"answer_2\", max_tokens=128))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "axiXF9-7WzL1" + }, + "source": [ + "### Connect to the server" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hSexW_7sWOu8" + }, + "source": [ + "Connect to the server using `sglang.set_default_backend` by specifying its URL." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "5zjwlk8cD3xU" + }, + "outputs": [], + "source": [ + "set_default_backend(RuntimeEndpoint(\"http://localhost:9000\"))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "f8OqAkTAc0TB" + }, + "source": [ + "### Send multi-turn questions to Gemma 2\n", + "\n", + "Now, you can run the previously defined `multi_turn_question` function to generate responses from the model." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "_0gjddhcWqrq" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "user : Who are the first humans to land on the moon?\n", + "assistant : The first humans to ever land on the moon were a team from the **Apollo 11 mission**:\n", + "\n", + "* **Neil Armstrong**: He became the first person to walk on the moon. His famous quote, \"One small step for man, one giant leap for mankind,\" encapsulates the magnitude of thishistoric event.\n", + "* **Buzz Aldrin**: Aldrin was the second human to walk on the moon and stayed with Armstrong for several hours on the lunar surface. \n", + "\n", + "They landed on the moon on **July 20, 1969**, bringing back a wealth of lunar samples and photos that remain highly significant\n", + "user : Which country did they belong to?\n", + "assistant : The first people to land on the moon were part of **the United States**, often simply referred to as Americans. They were a team from NASA, the National Aeronautics and Space Administration, the US government's space program. \n", + "\n" + ] + } + ], + "source": [ + "state = multi_turn_question.run(\n", + " question_1=\"Who are the first humans to land on the moon?\",\n", + " question_2=\"Which country did they belong to?\",\n", + ")\n", + "\n", + "for m in state.messages():\n", + " print(m[\"role\"], \":\", m[\"content\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "MPddl6aldM-J" + }, + "source": [ + "Notice how the history of the conversation is preserved, and the model answered the second question as a continuation of the conversation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3x5c2h1jdbWJ" + }, + "source": [ + "### Run a batch of multi-turn questions\n", + "\n", + "You can also batch a set of multi turn questions to the model by passing a list of dictionaries to `run_batch` whose keys specify the arguments to the `multi_turn_question` function." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "id": "rJ2OyNYo7PtM" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "===============================================================\n", + "\n", + "user : Who are the first humans to land on moon?\n", + "assistant : The first humans to land on the Moon were **Neil Armstrong** and **Buzz Aldrin** of the Apollo 11 mission on **July 20, 1969.** \n", + "\n", + "user : Which country did they belong to ?\n", + "assistant : Neil Armstrong and Buzz Aldrin were from the **United States**. \n", + "\n", + "\n", + "===============================================================\n", + "\n", + "user : Who is the first human to reach space?\n", + "assistant : The first human to reach space was **Yuri Gagarin**. \n", + "\n", + "On April 12, 1961, he completed one orbit of Earth in the Soviet Vostok 1 spacecraft. This event marked a significant moment in the history of human exploration, paving the way for further advancements and spaceflight feats. \n", + "\n", + "user : Which country did they belong to?\n", + "assistant : Yuri Gagarin was from **Soviet Union** at the time. \n", + "\n" + ] + } + ], + "source": [ + "states = multi_turn_question.run_batch(\n", + " [\n", + " {\n", + " \"question_1\": \"Who are the first humans to land on moon?\",\n", + " \"question_2\": \"Which country did they belong to ?\",\n", + " },\n", + " {\n", + " \"question_1\": \"Who is the first human to reach space?\",\n", + " \"question_2\": \"Which country did they belong to?\",\n", + " },\n", + " ]\n", + ")\n", + "\n", + "for state in states:\n", + " print(\"\\n===============================================================\\n\")\n", + " for message in state.messages():\n", + " print(message[\"role\"], \":\", message[\"content\"])\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HGsAknP7ZARH" + }, + "source": [ + "### JSON Decoding\n", + "\n", + "You can use a regular expression (regex) to specify a JSON schema that the model's generated answer must adhere to.\n", + "\n", + "Define a function to generate specific information about any animal in JSON format using Gemma 2. Specify the regex JSON schema in the regex argument of the sglang.gen function." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "xinEZ7x3dkl4" + }, + "outputs": [], + "source": [ + "character_regex = (\n", + " r\"\"\"\\{\\n\"\"\"\n", + " + r\"\"\" \"name\": \"[\\w\\d\\s]{1,16}\",\\n\"\"\"\n", + " + r\"\"\" \"type\": \"(Mammals|Birds|Fish|Reptiles|Amphibians|Invertebrates)\",\\n\"\"\"\n", + " + r\"\"\" \"reproduction\": \"(Sexual|Asexual)\",\\n\"\"\"\n", + " + r\"\"\" \"life expectancy\": \"[0-9]{1,2}\",\\n\"\"\"\n", + " + r\"\"\"\\}\"\"\"\n", + ")\n", + "\n", + "@function\n", + "def animal_gen(s, name):\n", + " s += name + \" is an animal. Please fill in the following information about this animal.\\n\"\n", + " s += gen(\"json_output\", max_tokens=256, regex=character_regex)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8-NHP4_ffQ3m" + }, + "source": [ + "Run the function with the name of any animal as input to get its features in JSON format." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "id": "v4NwLmms4_Jp" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Fish is an animal. Please fill in the following information about this animal.\n", + "{\n", + " \"name\": \"Fish\",\n", + " \"type\": \"Mammals\",\n", + " \"reproduction\": \"Sexual\",\n", + " \"life expectancy\": \"10\",\n", + "}\n" + ] + } + ], + "source": [ + "state = animal_gen.run(name=\"Fish\")\n", + "print(state.text())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qhl8qCPYRTSW" + }, + "source": [ + "Terminate the server process." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "sL8n9mBtRLXs" + }, + "outputs": [], + "source": [ + "terminate_process(server_process)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "vlRAC8OFa1pQ" + }, + "source": [ + "These are just a few examples of how a prompting workflow with Gemma 2 can be designed using SGLang's frontend language. To learn more about its capabilities, you can refer to SGLang's [Frontend: Structured Generation Language (SGLang)](https://sgl-project.github.io/frontend/frontend.html) guide." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NKR7JKQR5s9U" + }, + "source": [ + "Congratulations! You've successfully explored how Gemma 2 can be served using SGLang, run using the SGLang backend runtime and frontend language in a Colab environment. You can now experiment with more complex prompting workflows in SGLang to interact with Gemma 2." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Using_Gemma_with_SGLang.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} From 0e7f7291eab4bba1fa75ca9679e8d8d49fc081fb Mon Sep 17 00:00:00 2001 From: Prianka Liz Kariat Date: Tue, 5 Nov 2024 22:53:23 +0530 Subject: [PATCH 37/43] Updated README to include reference to SGLang notebook --- README.md | 1 + WISHLIST.md | 1 - 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 056a15f..130df8a 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | | [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | | [Using_Gemini_and_Gemma_with_RouteLLM.ipynb](Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb) | Route Gemma and Gemini models using [RouteLLM](https://github.com/lm-sys/RouteLLM/). | +| [Using_Gemma_with_SGLang.ipynb](Gemma/Using_Gemma_with_SGLang.ipynb) | Run Gemma models using [SGLang](https://github.com/sgl-project/sglang/). | | [Integrate_with_Mesop.ipynb](Gemma/Integrate_with_Mesop.ipynb) | Integrate Gemma with [Google Mesop](https://google.github.io/mesop/). | | [Integrate_with_OneTwo.ipynb](Gemma/Integrate_with_OneTwo.ipynb) | Integrate Gemma with [Google OneTwo](https://github.com/google-deepmind/onetwo). | | [Deploy_with_vLLM.ipynb](Gemma/Deploy_with_vLLM.ipynb) | Deploy a Gemma model using [vLLM](https://github.com/vllm-project/vllm). | diff --git a/WISHLIST.md b/WISHLIST.md index cef0cc9..3065795 100644 --- a/WISHLIST.md +++ b/WISHLIST.md @@ -1,7 +1,6 @@ A wish list of cookbooks showcasing: * Inference - * [SGLang](https://github.com/sgl-project/sglang) integration * Fine-tuning * Fine-tuning Gemma for function calling From 13013fb2578f76963afb8aeb5d7ef517524b22fb Mon Sep 17 00:00:00 2001 From: kristenrq <58997898+kristenrq@users.noreply.github.com> Date: Wed, 6 Nov 2024 11:34:20 -0800 Subject: [PATCH 38/43] Update forum link on README.md Link directly to Gemma forum --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 130df8a..4764815 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [CodeGemma_finetuned_on_SQL_with_HF.ipynb](CodeGemma/CodeGemma_finetuned_on_SQL_with_HF.ipynb) | Fine-Tuning CodeGemma on the SQL Spider Dataset. | ## Get help -Ask a Gemma cookbook-related question on the new [Build with Google AI Forum](https://discuss.ai.google.dev/), or open an [issue](https://github.com/google-gemini/gemma-cookbook/issues) on GitHub. +Ask a Gemma cookbook-related question on the [developer forum](https://discuss.ai.google.dev/c/gemma/10), or open an [issue](https://github.com/google-gemini/gemma-cookbook/issues) on GitHub. ## Wish list If you want to see additional cookbooks implemented for specific features/integrations, please send us a pull request by adding your feature request(s) in the [wish list](https://github.com/google-gemini/gemma-cookbook/blob/main/WISHLIST.md). From b04604bf73c8b65cca1f6cecf7dd741691b8bfa0 Mon Sep 17 00:00:00 2001 From: "nidhin.devan" Date: Thu, 7 Nov 2024 19:17:59 +0530 Subject: [PATCH 39/43] Constrained generation with gemma notebook added, README updated --- Gemma/Constrained_generation_with_Gemma.ipynb | 710 ++++++++++++++++++ README.md | 3 +- 2 files changed, 712 insertions(+), 1 deletion(-) create mode 100644 Gemma/Constrained_generation_with_Gemma.ipynb diff --git a/Gemma/Constrained_generation_with_Gemma.ipynb b/Gemma/Constrained_generation_with_Gemma.ipynb new file mode 100644 index 0000000..39d3803 --- /dev/null +++ b/Gemma/Constrained_generation_with_Gemma.ipynb @@ -0,0 +1,710 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "3DNHhOsQX4Rg" + }, + "source": [ + "##### Copyright 2024 Google LLC." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "cellView": "form", + "id": "_bO8_SJzX4t6" + }, + "outputs": [], + "source": [ + "# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ABzryOPIYE2O" + }, + "source": [ + "# Getting Started with Constrained generation with Gemma 2 using Llamacpp and Guidance\n", + "\n", + "[Gemma](https://ai.google.dev/gemma) is a family of lightweight, state-of-the-art open-source language models from Google. Built from the same research and technology used to create the Gemini models, Gemma models are text-to-text, decoder-only large language models (LLMs), available in English, with open weights, pre-trained variants, and instruction-tuned variants.\n", + "Gemma models are well-suited for various text-generation tasks, including question-answering, summarization, and reasoning. Their relatively small size makes it possible to deploy them in environments with limited resources such as a laptop, desktop, or cloud infrastructure, democratizing access to state-of-the-art AI models and helping foster innovation for everyone.\n", + "\n", + "Constrained generation is a method that modifies the token generation process of a generative model to limit its predictions for subsequent tokens to only those that adhere to the necessary output structure.\n", + "\n", + "[llama.cpp](https://github.com/ggerganov/llama.cpp) is a C++ implementation of Meta AI's LLaMA and other large language model architectures, designed for efficient performance on local machines or within environments like Google Colab. It enables you to run large language models without needing extensive computational resources. In llama.cpp, formal grammars are defined using the GBNF (GGML BNF) format to constrain model outputs. It can be used, for instance, to make the model produce legitimate JSON or to communicate exclusively in emojis.\n", + "\n", + "[Guidance](https://github.com/guidance-ai/guidance/tree/main?tab=readme-ov-file#constrained-generation) is an effective programming paradigm for steering language models. Guidance reduces latency and costs compared to traditional prompting or fine-tuning while allowing you to control the output's structure and provide high-quality output for your use case.\n", + "\n", + "In this notebook, you will learn how to perform constrained generation in Gemma 2 models using `llama.cpp` and `guidance` in a Google Colab environment. You'll install the necessary packages, set up the model, and run a sample prompt.\n", + "\n", + "\n", + "\n", + "
\n", + " Run in Google Colab\n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6TDJ3j0rh-Uw" + }, + "source": [ + "## Setup\n", + "\n", + "### Select the Colab runtime\n", + "To complete this tutorial, you'll need to have a Colab runtime with sufficient resources to run the Gemma model. In this case, you can use a T4 GPU:\n", + "\n", + "1. In the upper-right of the Colab window, select **โ–พ (Additional connection options)**.\n", + "2. Select **Change runtime type**.\n", + "3. Under **Hardware accelerator**, select **T4 GPU**.\n", + "\n", + "### Gemma setup\n", + "\n", + "**Before you dive into the tutorial, let's get you set up with Gemma:**\n", + "\n", + "1. **Hugging Face Account:** If you don't already have one, you can create a free Hugging Face account by clicking [here](https://huggingface.co/join).\n", + "2. **Gemma Model Access:** Head over to the [Gemma model page](https://huggingface.co/collections/google/gemma-2-release-667d6600fd5220e7b967f315) and accept the usage conditions.\n", + "3. **Colab with Gemma Power:** For this tutorial, you'll need a Colab runtime with enough resources to handle the Gemma 2B model. Choose an appropriate runtime when starting your Colab session.\n", + "4. **Hugging Face Token:** Generate a Hugging Face access (preferably `write` permission) token by clicking [here](https://huggingface.co/settings/tokens). You'll need this token later in the tutorial.\n", + "\n", + "**Once you've completed these steps, you're ready to move on to the next section where you'll set up environment variables in your Colab environment.**\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7tvumI5CiEeG" + }, + "source": [ + "### Configure your HF token\n", + "\n", + "Add your Hugging Face token to the Colab Secrets manager to securely store it.\n", + "\n", + "1. Open your Google Colab notebook and click on the ๐Ÿ”‘ Secrets tab in the left panel. \"The\n", + "2. Create a new secret with the name `HF_TOKEN`.\n", + "3. Copy/paste your token key into the Value input box of `HF_TOKEN`.\n", + "4. Toggle the button on the left to allow notebook access to the secret." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "lpXvNz0HeKc1" + }, + "outputs": [], + "source": [ + "import os\n", + "from google.colab import userdata\n", + "\n", + "# Note: `userdata.get` is a Colab API. If you're not using Colab, set the env\n", + "# vars as appropriate for your system.\n", + "os.environ[\"HF_TOKEN\"] = userdata.get(\"HF_TOKEN\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fxfWSVsaiFQK" + }, + "source": [ + "### Install dependencies\n", + "\n", + "You'll need to install a few Python packages and dependencies to interact with HuggingFace along with `llama-cpp-python` and `guidance`. Find some of the releases of `llama-cpp-python` supporting CUDA 12.2 [here](https://abetlen.github.io/llama-cpp-python/whl/cu122/llama-cpp-python/).\n", + "\n", + "Run the following cell to install or upgrade it:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "Xm9wxRQ_eUkC" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[?25l \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m0.0/447.5 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[91mโ•ธ\u001b[0m\u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m143.4/447.5 kB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[91mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m\u001b[90mโ•บ\u001b[0m \u001b[32m440.3/447.5 kB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m447.5/447.5 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hCollecting guidance\n", + " Downloading guidance-0.1.16-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.7 kB)\n", + "Collecting diskcache (from guidance)\n", + " Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from guidance) (1.26.4)\n", + "Collecting ordered-set (from guidance)\n", + " Downloading ordered_set-4.1.0-py3-none-any.whl.metadata (5.3 kB)\n", + "Requirement already satisfied: platformdirs in /usr/local/lib/python3.10/dist-packages (from guidance) (4.3.6)\n", + "Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from guidance) (3.20.3)\n", + "Requirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from guidance) (2.9.2)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from guidance) (2.32.3)\n", + "Collecting tiktoken>=0.3 (from guidance)\n", + " Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)\n", + "Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken>=0.3->guidance) (2024.9.11)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->guidance) (3.4.0)\n", + "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->guidance) (3.10)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->guidance) (2.2.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->guidance) (2024.8.30)\n", + "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic->guidance) (0.7.0)\n", + "Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic->guidance) (2.23.4)\n", + "Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic->guidance) (4.12.2)\n", + "Downloading guidance-0.1.16-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (255 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m255.4/255.4 kB\u001b[0m \u001b[31m8.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m37.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading ordered_set-4.1.0-py3-none-any.whl (7.6 kB)\n", + "Installing collected packages: ordered-set, diskcache, tiktoken, guidance\n", + "Successfully installed diskcache-5.6.3 guidance-0.1.16 ordered-set-4.1.0 tiktoken-0.8.0\n", + "\u001b[2K \u001b[90mโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”\u001b[0m \u001b[32m443.8/443.8 MB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h" + ] + } + ], + "source": [ + "# The huggingface_hub library allows us to download models and other files from Hugging Face.\n", + "!pip install --upgrade -q huggingface_hub\n", + "\n", + "# Install the guidance package.\n", + "!pip install guidance\n", + "\n", + "# The llama-cpp-python library allows us to leverage GPUs.\n", + "!pip install llama-cpp-python==0.2.90 \\\n", + " -q -U --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu122" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CtCpooOBipDy" + }, + "source": [ + "### Logging into Hugging Face Hub\n", + "\n", + "Next, youโ€™ll need to log into the Hugging Face Hub using your access token to download the Gemma model." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "CzMQZ1SReagV" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured.\n", + "WARNING:huggingface_hub._login:Note: Environment variable`HF_TOKEN` is set and is the current active token independently from the token you've just configured.\n" + ] + } + ], + "source": [ + "from huggingface_hub import login\n", + "\n", + "login(os.environ[\"HF_TOKEN\"])" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8ZjH8VRZismE" + }, + "source": [ + "### Downloading the Gemma 2 Model\n", + "Once you're logged in, you can download the Gemma 2 model files from Hugging Face. The [Gemma 2 model](https://huggingface.co/google/gemma-2-2b-GGUF) is available in **GGUF** format, which is optimized for use with `llama.cpp` and compatible tools like Llamafile." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "K2QM6BO3ebUY" + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "15433f2b900b4034ae81f61a5780efd5", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "2b_pt_v2.gguf: 0%| | 0.00/10.5G [00:00\", \"\", \"\", \"\", ...\n", + "llama_model_loader: - kv 19: tokenizer.ggml.scores arr[f32,256000] = [-1000.000000, -1000.000000, -1000.00...\n", + "llama_model_loader: - kv 20: tokenizer.ggml.token_type arr[i32,256000] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...\n", + "llama_model_loader: - kv 21: tokenizer.ggml.bos_token_id u32 = 2\n", + "llama_model_loader: - kv 22: tokenizer.ggml.eos_token_id u32 = 1\n", + "llama_model_loader: - kv 23: tokenizer.ggml.unknown_token_id u32 = 3\n", + "llama_model_loader: - kv 24: tokenizer.ggml.padding_token_id u32 = 0\n", + "llama_model_loader: - kv 25: tokenizer.ggml.add_bos_token bool = true\n", + "llama_model_loader: - kv 26: tokenizer.ggml.add_eos_token bool = false\n", + "llama_model_loader: - kv 27: tokenizer.ggml.add_space_prefix bool = false\n", + "llama_model_loader: - kv 28: general.quantization_version u32 = 2\n", + "llama_model_loader: - type f32: 288 tensors\n", + "llm_load_vocab: special tokens cache size = 249\n", + "llm_load_vocab: token to piece cache size = 1.6014 MB\n", + "llm_load_print_meta: format = GGUF V3 (latest)\n", + "llm_load_print_meta: arch = gemma2\n", + "llm_load_print_meta: vocab type = SPM\n", + "llm_load_print_meta: n_vocab = 256000\n", + "llm_load_print_meta: n_merges = 0\n", + "llm_load_print_meta: vocab_only = 0\n", + "llm_load_print_meta: n_ctx_train = 8192\n", + "llm_load_print_meta: n_embd = 2304\n", + "llm_load_print_meta: n_layer = 26\n", + "llm_load_print_meta: n_head = 8\n", + "llm_load_print_meta: n_head_kv = 4\n", + "llm_load_print_meta: n_rot = 256\n", + "llm_load_print_meta: n_swa = 4096\n", + "llm_load_print_meta: n_embd_head_k = 256\n", + "llm_load_print_meta: n_embd_head_v = 256\n", + "llm_load_print_meta: n_gqa = 2\n", + "llm_load_print_meta: n_embd_k_gqa = 1024\n", + "llm_load_print_meta: n_embd_v_gqa = 1024\n", + "llm_load_print_meta: f_norm_eps = 0.0e+00\n", + "llm_load_print_meta: f_norm_rms_eps = 1.0e-06\n", + "llm_load_print_meta: f_clamp_kqv = 0.0e+00\n", + "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n", + "llm_load_print_meta: f_logit_scale = 0.0e+00\n", + "llm_load_print_meta: n_ff = 9216\n", + "llm_load_print_meta: n_expert = 0\n", + "llm_load_print_meta: n_expert_used = 0\n", + "llm_load_print_meta: causal attn = 1\n", + "llm_load_print_meta: pooling type = 0\n", + "llm_load_print_meta: rope type = 2\n", + "llm_load_print_meta: rope scaling = linear\n", + "llm_load_print_meta: freq_base_train = 10000.0\n", + "llm_load_print_meta: freq_scale_train = 1\n", + "llm_load_print_meta: n_ctx_orig_yarn = 8192\n", + "llm_load_print_meta: rope_finetuned = unknown\n", + "llm_load_print_meta: ssm_d_conv = 0\n", + "llm_load_print_meta: ssm_d_inner = 0\n", + "llm_load_print_meta: ssm_d_state = 0\n", + "llm_load_print_meta: ssm_dt_rank = 0\n", + "llm_load_print_meta: ssm_dt_b_c_rms = 0\n", + "llm_load_print_meta: model type = 2B\n", + "llm_load_print_meta: model ftype = all F32\n", + "llm_load_print_meta: model params = 2.61 B\n", + "llm_load_print_meta: model size = 9.74 GiB (32.00 BPW) \n", + "llm_load_print_meta: general.name = ff8948d2ca54b23c93d253533c6effcf2e892347\n", + "llm_load_print_meta: BOS token = 2 ''\n", + "llm_load_print_meta: EOS token = 1 ''\n", + "llm_load_print_meta: UNK token = 3 ''\n", + "llm_load_print_meta: PAD token = 0 ''\n", + "llm_load_print_meta: LF token = 227 '<0x0A>'\n", + "llm_load_print_meta: EOT token = 107 ''\n", + "llm_load_print_meta: max token length = 48\n", + "llm_load_tensors: ggml ctx size = 0.13 MiB\n", + "llm_load_tensors: offloading 0 repeating layers to GPU\n", + "llm_load_tensors: offloaded 0/27 layers to GPU\n", + "llm_load_tensors: CPU buffer size = 9972.92 MiB\n", + "..................................................................\n", + "llama_new_context_with_model: n_ctx = 512\n", + "llama_new_context_with_model: n_batch = 512\n", + "llama_new_context_with_model: n_ubatch = 512\n", + "llama_new_context_with_model: flash_attn = 0\n", + "llama_new_context_with_model: freq_base = 10000.0\n", + "llama_new_context_with_model: freq_scale = 1\n", + "llama_kv_cache_init: CUDA_Host KV buffer size = 52.00 MiB\n", + "llama_new_context_with_model: KV self size = 52.00 MiB, K (f16): 26.00 MiB, V (f16): 26.00 MiB\n", + "llama_new_context_with_model: CUDA_Host output buffer size = 0.98 MiB\n", + "llama_new_context_with_model: CUDA0 compute buffer size = 2754.50 MiB\n", + "llama_new_context_with_model: CUDA_Host compute buffer size = 6.51 MiB\n", + "llama_new_context_with_model: graph nodes = 1050\n", + "llama_new_context_with_model: graph splits = 342\n", + "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \n", + "Model metadata: {'tokenizer.ggml.add_bos_token': 'true', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.unknown_token_id': '3', 'tokenizer.ggml.bos_token_id': '2', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'tokenizer.ggml.add_space_prefix': 'false', 'tokenizer.ggml.add_eos_token': 'false', 'gemma2.final_logit_softcapping': '30.000000', 'gemma2.attn_logit_softcapping': '50.000000', 'general.architecture': 'gemma2', 'gemma2.context_length': '8192', 'gemma2.attention.head_count_kv': '4', 'gemma2.attention.layer_norm_rms_epsilon': '0.000001', 'general.type': 'model', 'tokenizer.ggml.eos_token_id': '1', 'gemma2.embedding_length': '2304', 'tokenizer.ggml.pre': 'default', 'general.name': 'ff8948d2ca54b23c93d253533c6effcf2e892347', 'gemma2.block_count': '26', 'gemma2.feed_forward_length': '9216', 'gemma2.attention.key_length': '256', 'gemma2.attention.head_count': '8', 'gemma2.attention.sliding_window': '4096', 'gemma2.attention.value_length': '256', 'general.file_type': '0'}\n", + "Using fallback chat format: llama-2\n" + ] + } + ], + "source": [ + "import guidance\n", + "import numpy as np\n", + "from guidance import models, gen, block, optional, select, zero_or_more\n", + "from guidance import commit_point\n", + "\n", + "# Load the model\n", + "model_path = \"2b_pt_v2.gguf\"\n", + "gemma2 = models.LlamaCpp(model_path)\n", + "\n", + "\n", + "# Custom generation function to repeat the content up to two. Similar to\n", + "# one_or_more but there is a max value here.\n", + "@guidance(stateless=True)\n", + "def repeat_range(lm, content, min_count=1, max_count=2):\n", + " for _ in range(min_count):\n", + " lm += content\n", + " if max_count == np.inf:\n", + " lm += zero_or_more(content)\n", + " else:\n", + " for _ in range(max_count - min_count):\n", + " lm += optional(content)\n", + " return lm\n", + "\n", + "# Function to generate numbers up to two digits.\n", + "@guidance(stateless=True)\n", + "def number(lm):\n", + " n = repeat_range(select(['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']))\n", + " # Allow for negative or positive numbers\n", + " return lm + select(['-' + n, n])\n", + "\n", + "# Function to select player position.\n", + "@guidance(stateless=True)\n", + "def position(lm):\n", + " return lm + select([\"Striker\", \"Midfielder\", \"Defender\", \"Goalkeeper\"])\n", + "\n", + "# Function to select whether the player has won a World cup.\n", + "@guidance(stateless=True)\n", + "def world_cup(lm):\n", + " return lm + select([\"Yes\", \"No\"])\n", + "\n", + "# Regex function for string.\n", + "@guidance(stateless=True)\n", + "def string_exp(lm):\n", + " return lm + gen(regex='([^\\\\\\\\]*|\\\\\\\\[\\\\\\\\bfnrt\\/]|\\\\\\\\u[0-7a-z])*')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1YL3DdFi762J" + }, + "source": [ + "For this example, you will implement a combination of interleaved generative structure and CFG to show the stats of football(soccer) players as JSON.\n", + "\n", + "Here, you will keep the structure and keys of the JSON static, allowing the language model to fill in the value parts. This maintains the overall structure of the output." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "f39A6-Aa73FE" + }, + "outputs": [ + { + "data": { + "text/html": [ + "

Using JSON, describe these Football players:\n",
+              "Lionel Messi\n",
+              "{\n",
+              ""name": "Lionel Messi",\n",
+              ""country": "Argentina",\n",
+              ""position": Striker,\n",
+              ""stats": {\n",
+              "         "goals":10,\n",
+              "    "assists": 10,\n",
+              "    "height": 1.75,\n",
+              "    "world-cup": Yes,\n",
+              "}}
" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# `commit_point`s are just ways of stopping functions once you hit a point.\n", + "# For eg: commit_point(\",\") stops string_exp() once you hit `,`.\n", + "@guidance(stateless=True)\n", + "def simple_json(lm):\n", + " lm += ('{\\n' +\n", + " '\"name\": ' + string_exp() + commit_point(',') + '\\n'\n", + " '\"country\": ' + string_exp() + commit_point(',') + '\\n'\n", + " '\"position\": ' + position() + commit_point(',') + '\\n'\n", + " '\"stats\": {\\n' +\n", + " ' \"goals\":'+ number() + commit_point(',') + '\\n'\n", + " ' \"assists\": ' + number() + commit_point(',') + '\\n'\n", + " ' \"height\": ' + number() +'.' + number() + commit_point(',') + '\\n'\n", + " ' \"world-cup\": ' + world_cup() + commit_point(',') + '\\n'\n", + " + commit_point('}')\n", + " + commit_point('}'))\n", + " return lm\n", + "\n", + "# Initialize the query.\n", + "lm = gemma2 + \"\"\"Using JSON, describe these Football players:\n", + "Lionel Messi\n", + "\"\"\"\n", + "\n", + "# Call the simple_json function and implement the JSON structure.\n", + "lm += simple_json()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Zi8O6AF4As4N" + }, + "source": [ + "Congratulations! You've successfully implemented constrained generation with the Gemma 2 model using `llama.cpp` and `Guidance` in a Colab environment. You can now experiment with the model, update the grammar, and explore its capabilities." + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "name": "Constrained_generation_with_Gemma.ipynb", + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/README.md b/README.md index 4764815..3c317bc 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,8 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | [Using_Gemma_with_LlamaCpp.ipynb](Gemma/Using_Gemma_with_LlamaCpp.ipynb) | Run Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/). | | [Using_Gemma_with_LocalGemma.ipynb](Gemma/Using_Gemma_with_LocalGemma.ipynb) | Run Gemma models using [Local Gemma](https://github.com/huggingface/local-gemma/). | | [Using_Gemini_and_Gemma_with_RouteLLM.ipynb](Gemma/Using_Gemini_and_Gemma_with_RouteLLM.ipynb) | Route Gemma and Gemini models using [RouteLLM](https://github.com/lm-sys/RouteLLM/). | -| [Using_Gemma_with_SGLang.ipynb](Gemma/Using_Gemma_with_SGLang.ipynb) | Run Gemma models using [SGLang](https://github.com/sgl-project/sglang/). | +| [Using_Gemma_with_SGLang.ipynb](Gemma/Using_Gemma_with_SGLang.ipynb) | Run Gemma models using [SGLang](https://github.com/sgl-project/sglang/). | +| [Constrained_generation_with_Gemma.ipynb](Gemma/Constrained_generation_with_Gemma.ipynb) | Constrained generation with Gemma models using [LlamaCpp](https://github.com/abetlen/llama-cpp-python/) and [Guidance](https://github.com/guidance-ai/guidance/tree/main/). | | [Integrate_with_Mesop.ipynb](Gemma/Integrate_with_Mesop.ipynb) | Integrate Gemma with [Google Mesop](https://google.github.io/mesop/). | | [Integrate_with_OneTwo.ipynb](Gemma/Integrate_with_OneTwo.ipynb) | Integrate Gemma with [Google OneTwo](https://github.com/google-deepmind/onetwo). | | [Deploy_with_vLLM.ipynb](Gemma/Deploy_with_vLLM.ipynb) | Deploy a Gemma model using [vLLM](https://github.com/vllm-project/vllm). | From 384770e1d32dba27dc91ea488f60a0e42a768aa6 Mon Sep 17 00:00:00 2001 From: Juyeong Ji Date: Fri, 8 Nov 2024 09:32:58 +0900 Subject: [PATCH 40/43] Update Constrained_generation_with_Gemma.ipynb fix colab link --- Gemma/Constrained_generation_with_Gemma.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Gemma/Constrained_generation_with_Gemma.ipynb b/Gemma/Constrained_generation_with_Gemma.ipynb index 39d3803..74af98a 100644 --- a/Gemma/Constrained_generation_with_Gemma.ipynb +++ b/Gemma/Constrained_generation_with_Gemma.ipynb @@ -52,7 +52,7 @@ "\n", "\n", "\n", "
\n", - " Run in Google Colab\n", + " Run in Google Colab\n", "
" ] From ecf1b42d271a7e1a15e316b40e5620081f81a828 Mon Sep 17 00:00:00 2001 From: Wei Wei <5577741+windmaple@users.noreply.github.com> Date: Mon, 25 Nov 2024 14:52:17 +0800 Subject: [PATCH 41/43] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 3c317bc..e9f6962 100644 --- a/README.md +++ b/README.md @@ -50,6 +50,7 @@ You can find the Gemma models on GitHub, Hugging Face models, Kaggle, Google Clo | :------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Keras_Gemma_2_Quickstart.ipynb](Gemma/Keras_Gemma_2_Quickstart.ipynb) | Gemma 2 pre-trained 9B model quickstart tutorial with Keras. | | [Keras_Gemma_2_Quickstart_Chat.ipynb](Gemma/Keras_Gemma_2_Quickstart_Chat.ipynb) | Gemma 2 instruction-tuned 9B model quickstart tutorial with Keras. Referenced in this [blog](https://developers.googleblog.com/en/fine-tuning-gemma-2-with-keras-hugging-face-update/). | +| [Gemma inference with Flax/NNX](https://flax.readthedocs.io/en/latest/guides/gemma.html) | Gemma 1 inference with Flax/NNX framework (linking to Flax documentation) | | [Chat_and_distributed_pirate_tuning.ipynb](Gemma/Chat_and_distributed_pirate_tuning.ipynb) | Chat with Gemma 7B and finetune it so that it generates responses in pirates' tone. | | [gemma_inference_on_tpu.ipynb](Gemma/gemma_inference_on_tpu.ipynb) | Basic inference of Gemma with JAX/Flax on TPU. | | [gemma_data_parallel_inference_in_jax_tpu.ipynb](Gemma/gemma_data_parallel_inference_in_jax_tpu.ipynb) | Parallel inference of Gemma with JAX/Flax on TPU. | From 87871e15c2bea9d144bffec60e1bd15b96d70f8a Mon Sep 17 00:00:00 2001 From: LittleZ <189081567+tzl1986Dev@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:09:20 +0800 Subject: [PATCH 42/43] Create node.js.yml --- .github/workflows/node.js.yml | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 .github/workflows/node.js.yml diff --git a/.github/workflows/node.js.yml b/.github/workflows/node.js.yml new file mode 100644 index 0000000..2284b93 --- /dev/null +++ b/.github/workflows/node.js.yml @@ -0,0 +1,31 @@ +# This workflow will do a clean installation of node dependencies, cache/restore them, build the source code and run tests across different versions of node +# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-nodejs + +name: Node.js CI + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + +jobs: + build: + + runs-on: ubuntu-latest + + strategy: + matrix: + node-version: [18.x, 20.x, 22.x] + # See supported Node.js release schedule at https://nodejs.org/en/about/releases/ + + steps: + - uses: actions/checkout@v4 + - name: Use Node.js ${{ matrix.node-version }} + uses: actions/setup-node@v4 + with: + node-version: ${{ matrix.node-version }} + cache: 'npm' + - run: npm ci + - run: npm run build --if-present + - run: npm test From 49555ace82103b38fe00dc3ffbd768df475f6df6 Mon Sep 17 00:00:00 2001 From: LittleZ <189081567+tzl1986Dev@users.noreply.github.com> Date: Tue, 26 Nov 2024 09:10:09 +0800 Subject: [PATCH 43/43] Create python-package-conda.yml --- .github/workflows/python-package-conda.yml | 34 ++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 .github/workflows/python-package-conda.yml diff --git a/.github/workflows/python-package-conda.yml b/.github/workflows/python-package-conda.yml new file mode 100644 index 0000000..f358604 --- /dev/null +++ b/.github/workflows/python-package-conda.yml @@ -0,0 +1,34 @@ +name: Python Package using Conda + +on: [push] + +jobs: + build-linux: + runs-on: ubuntu-latest + strategy: + max-parallel: 5 + + steps: + - uses: actions/checkout@v4 + - name: Set up Python 3.10 + uses: actions/setup-python@v3 + with: + python-version: '3.10' + - name: Add conda to system path + run: | + # $CONDA is an environment variable pointing to the root of the miniconda directory + echo $CONDA/bin >> $GITHUB_PATH + - name: Install dependencies + run: | + conda env update --file environment.yml --name base + - name: Lint with flake8 + run: | + conda install flake8 + # stop the build if there are Python syntax errors or undefined names + flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics + # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide + flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics + - name: Test with pytest + run: | + conda install pytest + pytest