Skip to content

Commit

Permalink
update basic evaluation example
Browse files Browse the repository at this point in the history
  • Loading branch information
@shilpakancharla
shilpakancharla committed May 28, 2024
1 parent 84c281c commit 9c71837
Showing 1 changed file with 119 additions and 108 deletions.
227 changes: 119 additions & 108 deletions examples/prompting/Basic_Evaluation.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,43 +1,63 @@
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "X0doHtrEl66V"
},
"source": [
"# Gemini API: Basic evaluation"
],
"##### Copyright 2024 Google LLC."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"id": "2K6KP3hUl7_l"
},
"outputs": [],
"source": [
"# @title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"# you may not use this file except in compliance with the License.\n",
"# You may obtain a copy of the License at\n",
"#\n",
"# https://www.apache.org/licenses/LICENSE-2.0\n",
"#\n",
"# Unless required by applicable law or agreed to in writing, software\n",
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"# See the License for the specific language governing permissions and\n",
"# limitations under the License."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sP8PQnz1QrcF"
}
},
"source": [
"# Gemini API: Basic evaluation"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "bxGr_x3MRA0z"
},
"source": [
"<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
" <td>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/examples/prompting/Basic_Evaluation.ipynb\"><img src = \"https://www.tensorflow.org/images/colab_logo_32px.png\"/>Run in Google Colab</a>\n",
" <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/examples/prompting/Basic_Evaluation.ipynb\"><img src = \"https://www.tensorflow.org/images/colab_logo_32px.png\"/>Run in Google Colab</a>\n",
" </td>\n",
"</table>"
],
"metadata": {
"id": "bxGr_x3MRA0z"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ysy--KfNRrCq"
},
"source": [
"Gemini API's Python SDK can be used for various forms of evaluation, including:\n",
"- Providing feedback based on selected criteria\n",
Expand All @@ -46,21 +66,18 @@
"- Identifying weak areas\n",
"\n",
"Below is an example of using the LLM to enhance text quality through feedback and grading."
],
"metadata": {
"id": "ysy--KfNRrCq"
}
]
},
{
"cell_type": "code",
"source": [
"!pip install -U -q google-generativeai"
],
"execution_count": null,
"metadata": {
"id": "Ne-3gnXqR0hI"
},
"execution_count": null,
"outputs": []
"outputs": [],
"source": [
"!pip install -U -q google-generativeai"
]
},
{
"cell_type": "code",
Expand All @@ -77,87 +94,92 @@
},
{
"cell_type": "markdown",
"metadata": {
"id": "eomJzCa6lb90"
},
"source": [
"## Configure your API key\n",
"\n",
"To run the following cell, your API key must be stored it in a Colab Secret named `GOOGLE_API_KEY`. If you don't already have an API key, or you're not sure how to create a Colab Secret, see [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) for an example."
],
"metadata": {
"id": "eomJzCa6lb90"
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "v-JZzORUpVR2"
},
"outputs": [],
"source": [
"from google.colab import userdata\n",
"GOOGLE_API_KEY=userdata.get('GOOGLE_API_KEY')\n",
"\n",
"genai.configure(api_key=GOOGLE_API_KEY)"
],
"metadata": {
"id": "v-JZzORUpVR2"
},
"execution_count": null,
"outputs": []
]
},
{
"cell_type": "markdown",
"source": [
"## Example"
],
"metadata": {
"id": "L-Wt23A_uzFZ"
}
},
{
"cell_type": "code",
"source": [
"model = genai.GenerativeModel(model_name='gemini-pro', generation_config={\"temperature\": 0})"
],
"metadata": {
"id": "FCp3Mmx-uyjw"
},
"execution_count": null,
"outputs": []
"source": [
"## Example\n",
"\n",
"Start by defining some system instructions for this problem. For demonstration purposes, the use case involves prompting the model to write an essay with some mistakes. Remember that for generation tasks like writing an essay, you can change the temperature of the model to get more creative answers. Here, you can use `\"temperature\": 0` for predictability."
]
},
{
"cell_type": "code",
"source": [
"# For demonstration purposes, we prompt the model to write an essay with some mistakes.\n",
"# Remember that for generation tasks like writing an essay, you can change the temperature of the model to get more creative answers. But, we will here use temperature 0 for predictability.\n",
"generation_prompt = \"\"\"You're a college student. Write an essay riddled with common mistakes. The topic is the benefits of reading.\n",
"Ensure your essay includes a clear thesis statement. Your output should be just an essay, so do not include any notes.\"\"\"\n",
"\n",
"essay = model.generate_content(generation_prompt).text\n",
"Markdown(essay)"
],
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 343
},
"id": "zNw1Hele0Bne",
"outputId": "3240c685-c589-4b97-ae59-33d7e97b91c9"
"id": "zNw1Hele0Bne"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/markdown": "## The Power of Words: Why Reading is the Best Thing Ever\n\nReading is a super important thing. It's like, the best thing ever. Everyone should read more, because it's good for you. Like, really good. Reading can make you smarter, more knowledgeable, and even a better person. It's like a magic potion that can transform your brain.\n\nFirstly, reading helps you learn new things. When you read, you're basically absorbing information like a sponge. You can learn about history, science, art, and even how to cook a mean lasagna. It's like having a personal tutor in your pocket, except it doesn't cost anything.\n\nSecondly, reading can make you a better writer. By reading, you learn how to use language effectively. You can see how different authors use words to create different effects. It's like learning from the masters. Plus, reading can help you expand your vocabulary, which is super important for sounding smart.\n\nThirdly, reading can help you relax and de-stress. When you're reading a good book, you can escape from the real world and get lost in another one. It's like taking a vacation without leaving your couch. Reading can also help you sleep better, because it can calm your mind and help you unwind.\n\nIn conclusion, reading is awesome. It's like a superpower that can make you smarter, more knowledgeable, and a better person. So, next time you're feeling bored or stressed, pick up a book and give it a try. You won't regret it. \n",
"text/plain": [
"<IPython.core.display.Markdown object>"
],
"text/markdown": "**The Benifits of Reading**\n\nReading is a very benificial activity that can have a positive impact on our lives. It can help us to learn new things, improve our vocabulary, and develop our critical thinking skills. Reading can also help us to relax and de-stress.\n\nOne of the main benifits of reading is that it can help us to learn new things. When we read, we are exposed to new ideas and perspectives. This can help us to expand our knowledge and understanding of the world around us. Reading can also help us to learn new skills. For example, if we read a book about cooking, we may learn how to cook new dishes.\n\nAnother benifit of reading is that it can help us to improve our vocabulary. When we read, we are exposed to new words and phrases. This can help us to expand our vocabulary and improve our communication skills. Reading can also help us to learn new grammar rules. For example, if we read a book written in a formal style, we may learn how to use more formal language in our own writing.\n\nReading can also help us to develop our critical thinking skills. When we read, we are forced to think about what we are reading and to make inferences. This can help us to develop our ability to analyze information and to form our own opinions. Reading can also help us to become more open-minded and to consider different perspectives.\n\nFinally, reading can help us to relax and de-stress. When we read, we can escape from the stresses of everyday life and enter a world of our own imagination. Reading can also help us to reduce stress levels and to improve our sleep.\n\nIn conclusion, reading is a very benificial activity that can have a positive impact on our lives. It can help us to learn new things, improve our vocabulary, and develop our critical thinking skills. Reading can also help us to relax and de-stress. Therefore, I encourage everyone to make time for reading in their lives."
]
},
"execution_count": 8,
"metadata": {},
"execution_count": 5
"output_type": "execute_result"
}
],
"source": [
"student_system_prompt = \"\"\"You're a college student. Your job is to write an essay riddled with common mistakes and a few major ones.\n",
"The essay should have mistakes regarding clarity, grammar, argumentation, and vocabulary.\n",
"Ensure your essay includes a clear thesis statement. You should write only an essay, so do not include any notes.\"\"\"\n",
"\n",
"student_model = genai.GenerativeModel(model_name='gemini-1.5-flash-latest', generation_config={\"temperature\": 0},\n",
" system_instruction=student_system_prompt)\n",
"\n",
"essay = student_model.generate_content(\"Write an essay about benefits of reading.\").text\n",
"Markdown(essay)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mzp8ZFXf0-OT"
},
"outputs": [
{
"data": {
"text/markdown": "## Evaluation:\n\n**Thesis Statement:** 2/5 - The essay states that reading is \"the best thing ever\" but doesn't offer a specific argument or claim. It needs a more focused thesis statement.\n\n**Clarity and Precision of Language:** 2/5 - The language is informal and uses slang (\"super important,\" \"like,\" \"really good,\" \"super important\"). It lacks the precision and formality expected in academic writing.\n\n**Grammar and Punctuation:** 2/5 - There are several grammatical errors, including comma splices and run-on sentences. The essay also lacks proper punctuation, particularly with the use of commas.\n\n**Argumentation:** 3/5 - The essay presents three benefits of reading, but the arguments are not fully developed. They rely on generalizations and lack specific examples or evidence.\n\n## Corrected Version:\n\n**The Power of Words: The Transformative Benefits of Reading**\n\nReading is a fundamental human activity with profound benefits for individuals and society. It is not merely a pastime but a powerful tool that can enhance cognitive abilities, broaden perspectives, and foster empathy. This essay will explore three key ways in which reading transforms individuals: by expanding knowledge, improving communication skills, and promoting emotional well-being.\n\nFirstly, reading serves as a gateway to knowledge and understanding. Through books, we can explore diverse cultures, historical events, scientific discoveries, and philosophical ideas. By immersing ourselves in different worlds and perspectives, we expand our intellectual horizons and develop a deeper understanding of the complexities of the world around us. For example, reading historical accounts can provide valuable insights into the past, while scientific texts can deepen our understanding of the natural world.\n\nSecondly, reading enhances communication skills. By engaging with different writing styles and literary techniques, readers develop a greater appreciation for the nuances of language. They learn how to use words effectively to convey ideas, emotions, and arguments. Moreover, reading exposes individuals to a wider vocabulary, enriching their communication and enabling them to express themselves with greater clarity and precision.\n\nThirdly, reading promotes emotional well-being. Engaging with a good book can provide a much-needed escape from the stresses of daily life. It allows individuals to immerse themselves in fictional worlds, connect with relatable characters, and experience a range of emotions. This process can be therapeutic, fostering empathy, reducing stress, and promoting relaxation. Furthermore, reading can inspire creativity and imagination, enriching our inner lives and fostering a sense of wonder.\n\nIn conclusion, reading is not simply a leisure activity but a transformative experience that enriches our lives in countless ways. By expanding our knowledge, improving our communication skills, and promoting emotional well-being, reading empowers us to become more informed, articulate, and compassionate individuals. \n\n**Changes Made:**\n\n* **Thesis Statement:** The thesis statement is now more specific and focused, outlining the three key benefits of reading that will be discussed in the essay.\n* **Clarity and Precision of Language:** The language is more formal and precise, avoiding slang and informal expressions.\n* **Grammar and Punctuation:** The essay has been corrected for grammatical errors, including comma splices and run-on sentences. Proper punctuation has been implemented throughout.\n* **Argumentation:** The arguments are more developed, providing specific examples and evidence to support the claims. The essay now uses a more academic tone and structure. \n",
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"evaluation_prompt = f\"\"\"\n",
"teacher_system_prompt = f\"\"\"\n",
"As a teacher, you are tasked with grading students' essays.\n",
"Please follow these instructions for evaluation:\n",
"\n",
Expand All @@ -169,46 +191,35 @@
"\n",
"2. Write a corrected version of the essay, addressing any identified issues\n",
"in the original submission. Point what changes were made.\n",
"\n",
"Essay: {essay}\n",
"Evaluation:\n",
"\"\"\"\n",
"teacher_model = genai.GenerativeModel(model_name='gemini-1.5-flash-latest', generation_config={\"temperature\": 0},\n",
" system_instruction=teacher_system_prompt)\n",
"\n",
"Markdown(model.generate_content(evaluation_prompt).text)"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 739
},
"id": "mzp8ZFXf0-OT",
"outputId": "f9f75f1d-2881-4a43-b49c-ae53d9d8e0b9"
},
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<IPython.core.display.Markdown object>"
],
"text/markdown": "**Evaluation:**\n\n1. Thesis statement: 3/5\n- The thesis statement is clear and concise, but it does not provide a specific direction for the essay.\n\n2. Clarity and precision of language: 3/5\n- The language used in the essay is generally clear and precise, but there are a few instances of wordiness and repetition.\n\n3. Grammar and punctuation: 2/5\n- There are several grammatical and punctuation errors throughout the essay, including incorrect spelling of \"beneficial\" and missing commas.\n\n4. Argumentation: 3/5\n- The essay provides several examples to support the benefits of reading, but it does not provide a clear structure or logical progression of ideas.\n\n**Corrected Version:**\n\n**The Benefits of Reading**\n\nReading is a highly beneficial activity that can positively impact our lives. It enhances our knowledge, improves our communication skills, develops our critical thinking abilities, and promotes relaxation.\n\nOne of the primary benefits of reading is its ability to expand our knowledge. Through exposure to diverse perspectives and ideas, reading broadens our understanding of the world. It allows us to delve into different subjects, from history and science to literature and art, enriching our intellectual horizons.\n\nFurthermore, reading significantly improves our vocabulary. Encountering new words and phrases in texts enriches our language skills. It enhances our ability to express ourselves clearly and effectively, both verbally and in writing. Additionally, reading exposes us to various grammatical structures, improving our understanding of language conventions.\n\nReading also plays a crucial role in developing our critical thinking skills. It requires us to analyze information, draw inferences, and form our own opinions. By engaging with different viewpoints and perspectives, reading fosters open-mindedness and encourages us to consider alternative viewpoints.\n\nFinally, reading offers a valuable escape from the stresses of daily life. It allows us to immerse ourselves in worlds of our imagination, reducing stress levels and promoting relaxation. Studies have shown that reading before bed can improve sleep quality and reduce anxiety.\n\nIn conclusion, reading is an immensely beneficial activity that enriches our lives in numerous ways. It expands our knowledge, enhances our communication skills, develops our critical thinking abilities, and promotes relaxation. Therefore, incorporating reading into our daily routines is highly recommended for personal growth and well-being.\n\n**Changes Made:**\n\n- Corrected spelling of \"beneficial\" throughout the essay.\n- Added commas to improve clarity and sentence structure.\n- Removed unnecessary repetition and wordiness.\n- Reorganized the essay to provide a clearer structure and logical progression of ideas.\n- Added a concluding paragraph to summarize the main points and reinforce the thesis statement."
},
"metadata": {},
"execution_count": 6
}
"Markdown(teacher_model.generate_content(essay).text)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4Hbzc0EVDHxS"
},
"source": [
"## Next steps\n",
"\n",
"Be sure to explore other examples of prompting in the repository. Try writing your own prompts for evaluating texts."
],
"metadata": {
"id": "4Hbzc0EVDHxS"
}
]
}
],
"metadata": {
"colab": {
"name": "Basic_Evaluation.ipynb",
"toc_visible": true
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
]
}
},
"nbformat": 4,
"nbformat_minor": 0
}

0 comments on commit 9c71837

Please sign in to comment.