Skip to content

Commit

Permalink
update langchain_agents notebook (#778)
Browse files Browse the repository at this point in the history 
* update langchain_agents notebook

* updates to feedbacks, etc

* small updates

---------

Co-authored-by: Josh Reini <[email protected]>
Co-authored-by: Josh Reini <[email protected]>
  • Loading branch information
@piotrm0 @joshreini1
3 people authored Jan 10, 2024
1 parent c16b700 commit 748400d
Show file tree
Hide file tree
Showing 2 changed files with 103 additions and 114 deletions.
100 changes: 53 additions & 47 deletions trulens_eval/examples/expositional/frameworks/langchain/langchain_agents.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,24 @@
"source": [
"# LangChain Agents\n",
"\n",
"Agents are often useful in the RAG setting to retrieve real-time information to be used for question answering.\n",
"\n",
"This example utilizes the openai functions agent to reliably call and return structured responses from particular tools. Certain OpenAI models have been fine-tuned for this capability to detect when a particular function should be called and respond with the inputs requred for that function. Compared to a ReACT framework that generates reasoning and actions in an interleaving manner, this strategy can often be more reliable and consistent.\n",
"\n",
"In either case - as the questions change over time, different agents may be needed to retrieve the most useful context. In this example you will create a langchain agent and use TruLens to identify gaps in tool coverage. By quickly identifying this gap, we can quickly add the missing tools to the application and improve the quality of the answers.\n",
"\n",
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/langchain/langchain_agents.ipynb)"
"Agents are often useful in the RAG setting to retrieve real-time information to\n",
"be used for question answering.\n",
"\n",
"This example utilizes the openai functions agent to reliably call and return\n",
"structured responses from particular tools. Certain OpenAI models have been\n",
"fine-tuned for this capability to detect when a particular function should be\n",
"called and respond with the inputs requred for that function. Compared to a\n",
"ReACT framework that generates reasoning and actions in an interleaving manner,\n",
"this strategy can often be more reliable and consistent.\n",
"\n",
"In either case - as the questions change over time, different agents may be\n",
"needed to retrieve the most useful context. In this example you will create a\n",
"langchain agent and use TruLens to identify gaps in tool coverage. By quickly\n",
"identifying this gap, we can quickly add the missing tools to the application\n",
"and improve the quality of the answers.\n",
"\n",
"[![Open In\n",
"Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/truera/trulens/blob/main/trulens_eval/examples/expositional/frameworks/langchain/langchain_agents.ipynb)"
]
},
{
Expand All @@ -24,13 +35,28 @@
"### Import from LangChain and TruLens"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install additional packages\n",
"\n",
"In addition to trulens-eval and langchain, we will also need additional packages: `yfinance` and `google-search-results`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ! pip install trulens_eval==0.11.0 langchain==0.0.283 yfinance==0.2.28 google-search-results==2.4.2"
"! pip install \\\n",
" \"trulens_eval==0.20.2\" \\\n",
" \"langchain>=0.0.248\" \\\n",
" \"openai>=1.0\" \\\n",
" \"yfinance>=0.2.27\" \\\n",
" \"google-search-results>=2.4.2\""
]
},
{
Expand All @@ -40,7 +66,6 @@
"outputs": [],
"source": [
"from trulens_eval import Feedback\n",
"from trulens_eval import Huggingface\n",
"from trulens_eval import Tru\n",
"from trulens_eval import TruChain\n",
"from trulens_eval.feedback import OpenAI as fOpenAI\n",
Expand All @@ -52,14 +77,10 @@
"from typing import Type\n",
"\n",
"from langchain import SerpAPIWrapper\n",
"from langchain.agents import AgentExecutor\n",
"from langchain.agents import AgentType\n",
"from langchain.agents import BaseSingleActionAgent\n",
"from langchain.agents import initialize_agent\n",
"from langchain.agents import load_tools\n",
"from langchain.agents import Tool\n",
"from langchain.chat_models import ChatOpenAI\n",
"from langchain.llms import OpenAI as langchainOpenAI\n",
"from langchain.tools import BaseTool\n",
"import openai\n",
"from pydantic import BaseModel\n",
Expand All @@ -84,29 +105,8 @@
"outputs": [],
"source": [
"import os\n",
"os.environ[\"OPENAI_API_KEY\"] = \"...\"\n",
"os.environ[\"SERPAPI_API_KEY\"] = \"...\"\n",
"\n",
"langchainOpenAI.openai_api_key = os.environ[\"OPENAI_API_KEY\"]"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Install additional packages\n",
"\n",
"In addition to trulens-eval and langchain, we will also need additional packages: `yfinance` and `google-search-results`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ! pip install trulens_eval==0.8.0 langchain==0.0.248 openai==0.27.8 yfinance==0.2.27 google-search-results==2.4.2"
"os.environ[\"OPENAI_API_KEY\"] = \"sk-...\"\n",
"os.environ[\"SERPAPI_API_KEY\"] = \"...\""
]
},
{
Expand All @@ -125,17 +125,20 @@
"source": [
"search = SerpAPIWrapper()\n",
"search_tool = Tool(\n",
" name=\"Search\",\n",
" func=search.run,\n",
" description=\"useful for when you need to answer questions about current events\"\n",
" )\n",
"\n",
" name=\"Search\",\n",
" func=search.run,\n",
" description=\"useful for when you need to answer questions about current events\"\n",
")\n",
"\n",
"llm = ChatOpenAI(model=\"gpt-3.5-turbo\", temperature=0)\n",
"\n",
"tools = [search_tool]\n",
"\n",
"agent = initialize_agent(tools, llm, agent=AgentType.OPENAI_FUNCTIONS, verbose=True)"
"agent = initialize_agent(\n",
" tools, llm,\n",
" agent=AgentType.OPENAI_FUNCTIONS,\n",
" verbose=True\n",
")"
]
},
{
Expand All @@ -154,13 +157,13 @@
"source": [
"class OpenAI_custom(fOpenAI):\n",
" def no_answer_feedback(self, question: str, response: str) -> float:\n",
" return float(openai.ChatCompletion.create(\n",
" return float(self.endpoint.client.chat.completions.create(\n",
" model=\"gpt-3.5-turbo\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"Does the RESPONSE provide an answer to the QUESTION? Rate on a scale of 1 to 10. Respond with the number only.\"},\n",
" {\"role\": \"user\", \"content\": f\"QUESTION: {question}; RESPONSE: {response}\"}\n",
" ]\n",
" )[\"choices\"][0][\"message\"][\"content\"]) / 10\n",
" ).choices[0].message.content) / 10\n",
"\n",
"custom = OpenAI_custom()\n",
"\n",
Expand All @@ -174,7 +177,11 @@
"metadata": {},
"outputs": [],
"source": [
"tru_agent = TruChain(agent, app_id = \"Search_Agent\", feedbacks = [f_no_answer])"
"tru_agent = TruChain(\n",
" agent,\n",
" app_id=\"Search_Agent\",\n",
" feedbacks = [f_no_answer]\n",
")"
]
},
{
Expand Down Expand Up @@ -254,7 +261,6 @@
"metadata": {},
"outputs": [],
"source": [
"\n",
"class CurrentStockPriceInput(BaseModel):\n",
" \"\"\"Inputs for get_current_stock_price\"\"\"\n",
"\n",
Expand Down Expand Up @@ -399,7 +405,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.4"
"version": "3.11.5"
},
"vscode": {
"interpreter": {
Expand Down
Loading

0 comments on commit 748400d

Please sign in to comment.