Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

err h #828

Closed
wants to merge 1 commit into from
Closed

err h #828

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
388 changes: 388 additions & 0 deletions python/src/aiconfig/editor/server/error_handling.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,388 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part 1: Motivation\n",
"Let's say we're building a local editor that allows you to load an AIConfig\n",
"from a local file and then run methods on it.\n",
"\n",
"In the (simplified) code below, we do just that."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded AIConfig: NYC Trip Planner\n",
"\n",
"\n"
]
}
],
"source": [
"import json\n",
"from typing import Any\n",
"\n",
"\n",
"def read_json_from_file(path: str) -> dict[str, Any]:\n",
" with open(path, \"r\") as f:\n",
" return json.loads(f.read())\n",
" \n",
"\n",
"def start_app(path: str):\n",
" \"\"\"Load an AIConfig from a local path and do something with it.\"\"\"\n",
" aiconfig = read_json_from_file(path)\n",
" print(f\"Loaded AIConfig: {aiconfig['name']}\\n\")\n",
"\n",
"\n",
"start_app(\"cookbooks/Getting-Started/travel.aiconfig.json\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Cool, LGTM, ship it!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# A few hours later..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Issue #9000 Editor crashes on new file path\n",
"### opened 2 hours ago by lastmile-biggest-fan\n",
"\n",
"Dear LastMile team,\n",
"I really like the editor, but when I give it a new file path, it crashes!\n",
"I was hoping it would create a new AIConfig for me and write it to the file..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# OK, what happened?"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"ename": "FileNotFoundError",
"evalue": "[Errno 2] No such file or directory: 'i-dont-exist-yet-please-create-me.json'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[17], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mstart_app\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mi-dont-exist-yet-please-create-me.json\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
"Cell \u001b[0;32mIn[16], line 11\u001b[0m, in \u001b[0;36mstart_app\u001b[0;34m(path)\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mstart_app\u001b[39m(path: \u001b[38;5;28mstr\u001b[39m):\n\u001b[1;32m 10\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Load an AIConfig from a local path and do something with it.\"\"\"\u001b[39;00m\n\u001b[0;32m---> 11\u001b[0m aiconfig \u001b[38;5;241m=\u001b[39m json\u001b[38;5;241m.\u001b[39mloads(\u001b[43mread_file\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpath\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 12\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mLoaded AIConfig: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00maiconfig[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mname\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 13\u001b[0m \u001b[38;5;28mprint\u001b[39m()\n",
"Cell \u001b[0;32mIn[16], line 5\u001b[0m, in \u001b[0;36mread_file\u001b[0;34m(path)\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mread_file\u001b[39m(path: \u001b[38;5;28mstr\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m \u001b[38;5;28mstr\u001b[39m:\n\u001b[0;32m----> 5\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mpath\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mr\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m f:\n\u001b[1;32m 6\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m f\u001b[38;5;241m.\u001b[39mread()\n",
"File \u001b[0;32m/opt/homebrew/Caskroom/miniconda/base/envs/aiconfig/lib/python3.10/site-packages/IPython/core/interactiveshell.py:310\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m 303\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m 304\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 305\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 306\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 307\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 308\u001b[0m )\n\u001b[0;32m--> 310\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'i-dont-exist-yet-please-create-me.json'"
]
}
],
"source": [
"start_app(\"i-dont-exist-yet-please-create-me.json\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Oops"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ok, let's diagnose the problem here. We forgot to handle the case where the path doesn't exist.\n",
"\n",
"That's understandable. As programmers, we don't always write perfect code.\n",
"Sometimes it's helpful to bring new tools into the workflow to prevent this kind of problem in the future.\n",
"\n",
"\n",
"Hmm, ok. Wouldn't it be nice if we had a static analyzer that could have caught this problem immediately? That way we could have fixed it before the initial PR was merged.\n",
"\n",
"Let's analyze some tools."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## V2: Optional\n",
"\n",
"First, let's fix the root cause and catch exceptions. Now, what do we do in the `except` block? \n",
"\n",
"Well, we can reraise, but that brings us right back to the previous case and doesn't achieve anything helpful. \n",
"\n",
"Instead, notice what happens if we return None and type hint the function accordingly (Optional[...])."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"[Pyright] Object of type \"None\" is not subscriptable\n",
"PylancereportOptionalSubscript\n",
"(variable) aiconfig: dict[str, Any] | None\n",
"\n"
]
}
],
"source": [
"from typing import Any, Optional\n",
"\n",
"\n",
"def read_json_from_file(path: str) -> Optional[dict[str, Any]]:\n",
" try:\n",
" with open(path, \"r\") as f:\n",
" return json.loads(f.read())\n",
" except Exception as e:\n",
" return None\n",
" \n",
"\n",
"def start_app(path: str):\n",
" \"\"\"Load an AIConfig from a local path and do something with it.\"\"\"\n",
" aiconfig = read_json_from_file(path)\n",
" print(f\"Loaded AIConfig: {aiconfig['name']}\\n\")\n",
"\n",
"print(\"\"\"\n",
"[Pyright] Object of type \"None\" is not subscriptable\n",
"PylancereportOptionalSubscript\n",
"(variable) aiconfig: dict[str, Any] | None\n",
"\"\"\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Aha!\n",
"\n",
"Now, Pyright immediately tells us that `None` is a possibility, and we have to handle this case. Let's do that.\n"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded AIConfig: NYC Trip Planner\n",
"\n",
"Loaded AIConfig: \n",
"\n"
]
}
],
"source": [
"from typing import Optional\n",
"from aiconfig.Config import AIConfigRuntime\n",
"\n",
"\n",
"\n",
"def read_json_from_file(path: str) -> Optional[dict[str, Any]]:\n",
" try:\n",
" with open(path, \"r\") as f:\n",
" return json.loads(f.read())\n",
" except Exception:\n",
" return None\n",
"\n",
"def start_app(path: str):\n",
" \"\"\"Load an AIConfig from a local path and do something with it.\"\"\"\n",
" aiconfig = read_json_from_file(path)\n",
" if aiconfig is None:\n",
" print(f\"Could not load AIConfig from path: {path}. Creating and saving.\")\n",
" aiconfig = json.dumps(AIConfigRuntime.create())\n",
" # [save the aiconfig to the path] \n",
" print(f\"Loaded and saved new AIConfig\\n\")\n",
" else:\n",
" print(f\"Loaded AIConfig: {aiconfig}\\n\")\n",
"\n",
"start_app(\"cookbooks/Getting-Started/travel.aiconfig.json\")\n",
"start_app(\"i-dont-exist-yet-please-create-me.json\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ok, cool, much better. But wait, it would be nice to retain some information about what went wrong. My `None` value doesn't tell me anything about why the AIConfig couldn't be loaded. Does the file not exist? Was it a permission problem, networked filesystem problem? etc."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# V3: Result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The result library (https://github.com/rustedpy/result) provides a neat type\n",
"called `Result`, which is a bit like Optional. It's parametrized by the value type just like optional, but also by a second type for the error case.\n",
"\n",
"We can use it like optional, but store an arbitrary value with information about what went wrong."
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded AIConfig: NYC Trip Planner\n",
"\n",
"Could not load AIConfig from path: i-dont-exist-yet-please-create-me.json (File not found at path: i-dont-exist-yet-please-create-me.json). Creating and saving.\n",
"Created and saved new AIConfig: \n",
"\n"
]
}
],
"source": [
"from aiconfig.Config import AIConfigRuntime\n",
"from result import Result, Ok, Err\n",
"from typing import Any\n",
"\n",
"from json import JSONDecodeError\n",
"\n",
"\n",
"def read_json_from_file(path: str) -> Result[dict[str, Any], str]:\n",
" \"\"\"Use `str` in the error case to contain a helpful error message.\"\"\"\n",
" try:\n",
" with open(path, \"r\") as f:\n",
" return Ok(json.loads(f.read()))\n",
" except FileNotFoundError:\n",
" return Err(f\"File not found at path: {path}\")\n",
" except OSError as e:\n",
" return Err(f\"Could not read file at path: {path}: {e}\")\n",
" except JSONDecodeError as e:\n",
" return Err(f\"Could not parse JSON at path: {path}: {e}\")\n",
"\n",
"def start_app(path: str):\n",
" \"\"\"Load an AIConfig from a local path and do something with it.\"\"\"\n",
" file_contents = read_json_from_file(path)\n",
" match file_contents:\n",
" case Ok(aiconfig_ok):\n",
" print(f\"Loaded AIConfig: {aiconfig_ok['name']}\\n\")\n",
" case Err(e):\n",
" print(f\"Could not load AIConfig from path: {path} ({e}). Creating and saving.\")\n",
" aiconfig = AIConfigRuntime.create().model_dump(exclude=\"callback_manager\")\n",
" # [Save to file path]\n",
" # aiconfig.save(path)\n",
" print(f\"Created and saved new AIConfig: {aiconfig['name']}\\n\")\n",
"\n",
"start_app(\"cookbooks/Getting-Started/travel.aiconfig.json\")\n",
"start_app(\"i-dont-exist-yet-please-create-me.json\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are several nice things about this pattern:\n",
"* If you fail to check for the error case, you get static errors similar to the `None` Optional case\n",
"* You also get specific, useful error information unlike Optional\n",
"* Structural pattern matching: When matching the cases, you can elegantly and safely unbox the data inside the result.\n",
"* Because of pyright's ability to check for exhaustive pattern matching, it will yell at you if you don't handle the Err case. Try it! Comment out the Err case."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Part 2: Composition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Cool, so we have a very basic example of better error handling. What about a more realistic level of complexity involving a sequence of chained operations? Consider this variant of the previous app example:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"from typing import Any\n",
"\n",
"def read_file(path: str) -> str:\n",
" with open(path, \"r\") as f:\n",
" return f.read()\n",
" \n",
" \n",
"\n",
"def read_json_from_file(path: str) -> dict[str, Any]:\n",
" with open(path, \"r\") as f:\n",
" return json.loads(f.read())\n",
" \n",
"\n",
"def start_app(path: str):\n",
" \"\"\"Load an AIConfig from a local path and do something with it.\"\"\"\n",
" aiconfig = read_json_from_file(path)\n",
" print(f\"Loaded AIConfig: {aiconfig['name']}\\n\")\n",
"\n",
"\n",
"start_app(\"cookbooks/Getting-Started/travel.aiconfig.json\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "aiconfig",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}