10_4_4_Best_Practices_for_Vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix 10.4.4: Best practices for using vision with Claude\n",
    "\n",
    "Vision allows for a new mode of interaction with Claude. We’ve compiled a few tips for getting the best performance on your images. Before we get to that, let's first setup the code we need to run the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pip install -qUr requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "from IPython.display import Image\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "session = boto3.Session()\n",
    "region = session.region_name\n",
    "\n",
    "modelId = 'anthropic.claude-3-sonnet-20240229-v1:0'\n",
    "#modelId = 'anthropic.claude-3-haiku-20240307-v1:0'\n",
    "\n",
    "print(f'Using modelId: {modelId}')\n",
    "print('Using region: ', region)\n",
    "\n",
    "bedrock_client = boto3.client(service_name = 'bedrock-runtime', region_name = region,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_base64_encoded_image(image_path):\n",
    "    with open(image_path, \"rb\") as f:\n",
    "        image_file = f.read()\n",
    "\n",
    "    return image_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying traditional techniques to multimodal\n",
    "\n",
    "You can fix hallucination issues with traditional prompt engineering techniques like role assignment. Let’s see an example of this:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose I want Claude to count the number of dogs in this image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/nine_dogs.jpg') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'jpeg',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/nine_dogs.jpg\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"How many dogs are in this picture?\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's only 9 dogs but Claude thinks there is 10! Let’s apply a little prompt engineering and and try again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'jpeg',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/nine_dogs.jpg\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"You have perfect vision and pay great attention to detail which makes you an expert at counting objects in images. How many dogs are in this picture? Before providing the answer in <answer> tags, think step by step in <thinking> tags and analyze every part of the image.\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! After applying some prompt engineering to the prompt, we see that Claude now counts correctly that there is 9 dogs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visual prompting \n",
    "\n",
    "Images as input allows for prompts to now be given within the image itself. Let’s take a look at some examples.\n",
    "\n",
    "In this image, we write some text and draw an arrow on it. Let’s just pass this in to Claude with no accompanying text prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/circle.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/circle.png\")}\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "    \"inferenceConfig\": {\"maxTokens\": 2048},\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, Claude tried to describe the image as we didn’t give it a question. Let’s add a question to the image and pass it in again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/labeled_circle.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/labeled_circle.png\")}\n",
    "                },\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "    \"inferenceConfig\": {\"maxTokens\": 2048},\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also highlight specific parts of the image and ask questions about it.\n",
    "\n",
    "What’s the difference between these two numbers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/table.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/table.png\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"What’s the difference between these two numbers?\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Few-shot examples\n",
    "\n",
    "Adding examples to prompts still improves accuracy with visual tasks as well. Let’s ask Claude to read a picture of a speedometer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/140.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/140.png\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"What speed am I going?\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Claude’s answer doesn’t look quite right here, it thinks we are going 140km/hour and not 140 miles/hour! Let’s try again but this time let’s add some examples to the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/70.png\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"What speed am I going?\"},\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"role\": 'assistant',\n",
    "        \"content\": [\n",
    "            {\"text\": \"You are going 70 miles per hour.\"}\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/100.png\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"What speed am I going?\"},\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"role\": 'assistant',\n",
    "        \"content\": [\n",
    "            {\"text\": \"You are going 100 miles per hour.\"}\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\n",
    "                \"format\": 'png',\n",
    "                \"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/140.png\")}\n",
    "                },\n",
    "            },\n",
    "            {\"text\": \"What speed am I going?\"},\n",
    "        ]\n",
    "    },\n",
    "]\n",
    "\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect! With those examples, Claude learned how to read the speed on the speedometer. Note though that few-shot prompting with images doesn't always work but it is worth trying on your use case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple images as input\n",
    "Claude can also accept and reason over multiple images at once within the prompt as well! For example, let’s say you had a really large image - like an image of a long receipt! We can split that image up into chunks and feed each one of those chunks into Claude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/receipt1.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/receipt2.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\"format\": 'png',\"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/receipt1.png\")}},},\n",
    "            {\"image\": {\"format\": 'png',\"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/receipt2.png\")}},},\n",
    "            {\"text\": \"Output the name of the restaurant and the total.\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Object identification from examples\n",
    "\n",
    "With image input, you can pass in other images to the prompt and Claude will use that information to answer questions. Let’s see an example of this. \n",
    "\n",
    "Suppose we were trying to identify the type of pant in an image. We can provide Claude some examples of different types of pants in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Image(filename='./images/best_practices/officer_example.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": 'user',\n",
    "        \"content\": [\n",
    "            {\"image\": {\"format\": 'png',\"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/wrinkle.png\")}},},\n",
    "            {\"image\": {\"format\": 'png',\"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/officer.png\")}},},\n",
    "            {\"image\": {\"format\": 'png',\"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/chinos.png\")}},},\n",
    "            {\"image\": {\"format\": 'png',\"source\": {\"bytes\": get_base64_encoded_image(\"./images/best_practices/officer_example.png\")}},},\n",
    "            {\"text\": \"These pants are (in order) WRINKLE-RESISTANT DRESS PANT, ITALIAN MELTON OFFICER PANT, SLIM RAPID MOVEMENT CHINO. What pant is shown in the last image?\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "converse_api_params = {\n",
    "    \"modelId\": modelId,\n",
    "    \"messages\": messages,\n",
    "}\n",
    "response = bedrock_client.converse(**converse_api_params)\n",
    "\n",
    "# Extract the generated text content from the response\n",
    "print(response['output']['message']['content'][0]['text'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}