agentic-rag/open_deep_researcher-openrouter-tavily.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Deep Researcher**\n",
    "\n",
    "Empower yourself with an intelligent agent capable of reasoning through extensive online resources and executing intricate, multi-stage research tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "y7cTpP9rDZW-",
    "outputId": "5a443ad2-7a8d-4fef-f315-12108c28f1a2"
   },
   "outputs": [],
   "source": [
    "!pip3 install nest_asyncio tavily-python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook implements an AI researcher that persistently searches for information based on a user query until it's confident all necessary details have been gathered. It leverages several services:\n",
    "\n",
    "* [**Tavily:**](https://tavily.com/) To perform intelligent, AI-optimized searches and also to fetch and extract clean, LLM-friendly webpage content\n",
    "* [**OpenRouter:**](https://accounts.openrouter.ai/sign-in) To interact with a LLM for generating search queries, evaluating page relevance, and extracting context.\n",
    "\n",
    "### Key Components\n",
    "\n",
    "#### Tavily Search Integration\n",
    "\n",
    "Tavily is dedicated to creating an advanced search layer that bridges large language models (LLMs) with the web, empowering AI agents with real-time, context-aware data. With its adaptable search functionalities, Tavily allows AI systems to refine search approaches, gather raw content for in-depth analysis, or obtain concise summaries for rapid understanding. Unlike traditional models limited by fixed training datasets, Tavily’s Search and Extract features integrate semantic, contextual, and keyword-based search methods to provide up-to-date, actionable insights, enhancing data-driven decision-making processes.\n",
    "\n",
    "#### Openrouter\n",
    "\n",
    "The system uses OpenRouter to interact with a language model for various tasks such as query generation, relevance assessment, and context extraction. This allows for dynamic and intelligent information gathering based on the evolving understanding of the user's query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "import aiohttp\n",
    "import json\n",
    "import logging\n",
    "from tavily import AsyncTavilyClient\n",
    "from typing import Dict, List, Optional\n",
    "\n",
    "OPENROUTER_API_KEY = \"sk-XXXX\" ## Get your key at https://accounts.openrouter.ai/sign-in\n",
    "TAVILY_API_KEY = \"tvly-XXX\" ## Get your key at https://tavily.com/\n",
    "\n",
    "# Endpoints\n",
    "OPENROUTER_URL = \"https://openrouter.ai/api/v1/chat/completions\"\n",
    "TAVILY_SEARCH_URL = \"https://api.tavily.com/search\"\n",
    "\n",
    "# LLM model\n",
    "DEFAULT_MODEL = \"anthropic/claude-3.5-haiku\"\n",
    "\n",
    "# Initialize the Tavily client\n",
    "tavily_client = AsyncTavilyClient(api_key=TAVILY_API_KEY) if TAVILY_API_KEY else TavilyClient()\n",
    "\n",
    "# Configure logging\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "logger = logging.getLogger(__name__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def call_openrouter_async(\n",
    "    session: aiohttp.ClientSession,\n",
    "    messages: List[Dict[str, str]],\n",
    "    model: str = DEFAULT_MODEL,\n",
    "    timeout: int = 15\n",
    ") -> Optional[str]:\n",
    "    \"\"\"\n",
    "    Asynchronously call the OpenRouter chat completion API with the provided messages and return the response.\n",
    "\n",
    "    :param session: aiohttp ClientSession to use for the request.\n",
    "    :param messages: List of messages to send to the API.\n",
    "    :param model: Model to use for the completion.\n",
    "    :param timeout: Timeout in seconds for the request.\n",
    "    :return: The content of the assistant’s reply or None if an error occurs.\n",
    "    \"\"\"\n",
    "    headers = {\n",
    "        \"Authorization\": f\"Bearer {OPENROUTER_API_KEY}\",\n",
    "        \"X-Title\": \"OpenDeepResearcher\",\n",
    "        \"Content-Type\": \"application/json\"\n",
    "    }\n",
    "    payload = {\n",
    "        \"model\": model,\n",
    "        \"messages\": messages\n",
    "    }\n",
    "\n",
    "    try:\n",
    "        async with session.post(OPENROUTER_URL, headers=headers, json=payload, timeout=timeout) as resp:\n",
    "            if resp.status == 200:\n",
    "                result = await resp.json()\n",
    "                try:\n",
    "                    return result['choices'][0]['message']['content']\n",
    "                except (KeyError, IndexError) as e:\n",
    "                    logger.error(\"Unexpected OpenRouter response structure: %s\", result)\n",
    "                    return None\n",
    "            else:\n",
    "                text = await resp.text()\n",
    "                logger.error(\"OpenRouter API error: %d - %s\", resp.status, text)\n",
    "                return None\n",
    "    except aiohttp.ClientError as e:\n",
    "        logger.error(\"Error calling OpenRouter: %s\", e)\n",
    "        return None\n",
    "    except asyncio.TimeoutError:\n",
    "        logger.error(\"OpenRouter API request timed out\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def generate_search_queries_async(\n",
    "    session: aiohttp.ClientSession,\n",
    "    user_query: str,\n",
    "    max_queries: int = 4\n",
    ") -> List[str]:\n",
    "    \"\"\"\n",
    "    Asynchronously generate up to four precise search queries based on the user's query using an LLM.\n",
    "\n",
    "    Args:\n",
    "        session (aiohttp.ClientSession): The aiohttp session to use for the API call.\n",
    "        user_query (str): The user's query to generate search queries from.\n",
    "        max_queries (int, optional): Maximum number of search queries to generate. Defaults to 4.\n",
    "\n",
    "    Returns:\n",
    "        List[str]: A list of search queries generated by the LLM. Returns an empty list if an error occurs.\n",
    "\n",
    "    Example:\n",
    "        >>> queries = await generate_search_queries_async(session, \"What is the impact of AI on healthcare?\")\n",
    "        >>> print(queries)\n",
    "        ['impact of AI on healthcare', 'AI applications in healthcare', 'benefits of AI in medicine', 'AI in healthcare research']\n",
    "    \"\"\"\n",
    "    prompt = (\n",
    "        \"You are an expert research assistant. Given the user's query, generate up to four distinct, \"\n",
    "        \"precise search queries that would help gather comprehensive information on the topic. \"\n",
    "        f\"Return only a Python list of strings, for example: ['query1', 'query2', 'query3']. \"\n",
    "        f\"Do not return more than {max_queries} queries.\"\n",
    "    )\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"You are a helpful and precise research assistant.\"},\n",
    "        {\"role\": \"user\", \"content\": f\"User Query: {user_query}\\n\\n{prompt}\"}\n",
    "    ]\n",
    "\n",
    "    # Call the OpenRouter API\n",
    "    response = await call_openrouter_async(session, messages)\n",
    "    if not response:\n",
    "        logger.error(\"No response received from the LLM.\")\n",
    "        return []\n",
    "\n",
    "    try:\n",
    "        # Safely evaluate the response to ensure it's a valid Python list\n",
    "        search_queries = ast.literal_eval(response)\n",
    "        if isinstance(search_queries, list):\n",
    "            # Ensure the list contains only strings and trim to max_queries\n",
    "            search_queries = [query.strip() for query in search_queries if isinstance(query, str)]\n",
    "            return search_queries[:max_queries]\n",
    "        else:\n",
    "            logger.error(\"LLM did not return a valid list. Response: %s\", response)\n",
    "            return []\n",
    "    except (ValueError, SyntaxError) as e:\n",
    "        logger.error(\"Error parsing search queries: %s. Response: %s\", e, response)\n",
    "        return []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def perform_search_async(session , query):\n",
    "    \"\"\"\n",
    "    Perform an asynchronous search using the Tavily API.\n",
    "    More info : https://docs.tavily.com/api-reference/endpoint/search\n",
    "\n",
    "    Args:\n",
    "        query (str): The search query.\n",
    "        max_results (int, optional): Maximum number of results to return. Defaults to 5.\n",
    "    \"\"\"\n",
    "    payload = {\n",
    "    \"query\": query,\n",
    "    \"topic\": \"general\",\n",
    "    \"search_depth\": \"basic\",\n",
    "    \"max_results\": 10, ## Modify it to go deeper\n",
    "    \"time_range\": None,\n",
    "    \"days\": 3, ##Remove this to get results from all the times\n",
    "    \"include_answer\": True,\n",
    "    \"include_raw_content\": True,\n",
    "    \"include_images\": False,\n",
    "    \"include_image_descriptions\": False,\n",
    "    \"include_domains\": [],\n",
    "    \"exclude_domains\": []\n",
    "    }\n",
    "    headers = {\n",
    "        \"Authorization\": f\"Bearer {TAVILY_API_KEY}\",\n",
    "        \"Content-Type\": \"application/json\"\n",
    "    }\n",
    "    \n",
    "    try:\n",
    "        async with aiohttp.ClientSession() as session:\n",
    "            async with session.post(TAVILY_SEARCH_URL, json=payload, headers=headers) as resp:\n",
    "                if resp.status == 200:\n",
    "                    results = await resp.json()\n",
    "                    return results\n",
    "                else:\n",
    "                    error_text = await resp.text()\n",
    "                    logger.error(f\"Tavily API error: {resp.status} - {error_text}\")\n",
    "                    return None\n",
    "    except aiohttp.ClientError as e:\n",
    "        logger.error(f\"Error performing Tavily search: {e}\")\n",
    "        return None\n",
    "    except Exception as e:\n",
    "        logger.error(f\"Unexpected error during Tavily search: {e}\")\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def is_page_useful_async(session, user_query, page_text):\n",
    "    \"\"\"\n",
    "    Determines whether the provided webpage content is useful for answering the given user query.\n",
    "    \n",
    "    The function queries an LLM with a strict prompt that requires a \"Yes\" or \"No\" response.\n",
    "    If the model returns an unexpected response, the function attempts to extract \"Yes\" or \"No\" from it.\n",
    "    \n",
    "    Args:\n",
    "        session: The HTTP session used for making asynchronous API requests.\n",
    "        user_query (str): The user's search query.\n",
    "        page_text (str): The text content of the webpage (truncated to 20,000 characters for processing).\n",
    "    \n",
    "    Returns:\n",
    "        str: Either \"Yes\" or \"No\" indicating whether the page is useful for answering the query.\n",
    "    \"\"\"\n",
    "    prompt = (\n",
    "        \"You are an expert evaluator of research relevance. Analyze the user's query and the provided webpage content. \"\n",
    "        \"Determine whether the webpage contains valuable information that directly helps answer the query. \"\n",
    "        \"Respond with exactly one word: 'Yes' if the page is useful, or 'No' if it is not. \"\n",
    "        \"Do not include any explanations, extra words, or formatting beyond 'Yes' or 'No'.\"\n",
    "    )\n",
    "    \n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"You are a strict and concise evaluator of research relevance.\"},\n",
    "        {\"role\": \"user\", \"content\": f\"User Query: {user_query}\\n\\nWebpage Content (truncated to 20000 characters):\\n{page_text[:20000]}\\n\\n{prompt}\"}\n",
    "    ]\n",
    "    \n",
    "    response = await call_openrouter_async(session, messages)\n",
    "    \n",
    "    if response:\n",
    "        answer = response.strip()\n",
    "        if answer in {\"Yes\", \"No\"}:\n",
    "            return answer\n",
    "        \n",
    "        # Fallback: Attempt to extract \"Yes\" or \"No\" if extra text is included.\n",
    "        if \"Yes\" in answer:\n",
    "            return \"Yes\"\n",
    "        elif \"No\" in answer:\n",
    "            return \"No\"\n",
    "    \n",
    "    return \"No\"  # Default fallback to \"No\" if response is empty or ambiguous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def extract_relevant_context_async(session, user_query, search_query, page_text):\n",
    "    \"\"\"\n",
    "    Extracts relevant information from a webpage based on the user's query and the search query used.\n",
    "    \n",
    "    The function prompts an LLM to extract and return only the relevant context from the webpage content.\n",
    "    No extra commentary or formatting is included in the response.\n",
    "    \n",
    "    Args:\n",
    "        session: The HTTP session used for making asynchronous API requests.\n",
    "        user_query (str): The original user query.\n",
    "        search_query (str): The search query that led to the webpage.\n",
    "        page_text (str): The text content of the webpage (truncated to 20,000 characters for processing).\n",
    "    \n",
    "    Returns:\n",
    "        str: Extracted relevant information as plain text, or an empty string if no relevant context is found.\n",
    "    \"\"\"\n",
    "    prompt = (\n",
    "        \"You are an expert information extractor. Analyze the user's query, the search query that led to this page, \"\n",
    "        \"and the webpage content. Extract all pieces of information that are directly relevant to answering the user's query. \"\n",
    "        \"Return only the extracted relevant context as plain text without any additional commentary or formatting.\"\n",
    "    )\n",
    "    \n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"You are an expert in extracting and summarizing relevant information.\"},\n",
    "        {\"role\": \"user\", \"content\": f\"User Query: {user_query}\\nSearch Query: {search_query}\\n\\nWebpage Content (truncated to 20000 characters):\\n{page_text[:20000]}\\n\\n{prompt}\"}\n",
    "    ]\n",
    "    \n",
    "    response = await call_openrouter_async(session, messages)\n",
    "    \n",
    "    return response.strip() if response else \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def get_new_search_queries_async(session, user_query, previous_search_queries, all_contexts):\n",
    "    \"\"\"\n",
    "    Determines whether additional search queries are needed based on the original query, past searches, and extracted contexts.\n",
    "    \n",
    "    If further research is needed, the function returns a Python list of up to four new search queries.\n",
    "    If no further research is required, the function returns the string \"<done>\".\n",
    "    \n",
    "    Args:\n",
    "        session: The HTTP session used for making asynchronous API requests.\n",
    "        user_query (str): The original user query.\n",
    "        previous_search_queries (list of str): A list of search queries previously used.\n",
    "        all_contexts (list of str): A list of extracted relevant contexts from webpages.\n",
    "    \n",
    "    Returns:\n",
    "        list[str] or str: A list of up to four new search queries if further research is needed, or \"<done>\" if no more searches are required.\n",
    "    \"\"\"\n",
    "    context_combined = \"\\n\".join(all_contexts)\n",
    "    prompt = (\n",
    "        \"You are an analytical research assistant. Based on the original query, the search queries performed so far, \"\n",
    "        \"and the extracted contexts from webpages, determine if further research is needed. \"\n",
    "        \"If further research is needed, provide up to four new search queries as a Python list (for example, \"\n",
    "        \"['new query1', 'new query2']). If you believe no further research is needed, respond with exactly <done>. \"\n",
    "        \"\\nOutput only a Python list or the token <done> without any additional text.\"\n",
    "    )\n",
    "    \n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"You are a systematic research planner.\"},\n",
    "        {\"role\": \"user\", \"content\": f\"User Query: {user_query}\\nPrevious Search Queries: {previous_search_queries}\\n\\nExtracted Relevant Contexts:\\n{context_combined}\\n\\n{prompt}\"}\n",
    "    ]\n",
    "    \n",
    "    response = await call_openrouter_async(session, messages)\n",
    "    \n",
    "    if response:\n",
    "        cleaned = response.strip()\n",
    "        if cleaned == \"<done>\":\n",
    "            return \"<done>\"\n",
    "        try:\n",
    "            new_queries = eval(cleaned)\n",
    "            if isinstance(new_queries, list):\n",
    "                return new_queries\n",
    "            else:\n",
    "                print(\"LLM did not return a list for new search queries. Response:\", response)\n",
    "                return []\n",
    "        except Exception as e:\n",
    "            print(\"Error parsing new search queries:\", e, \"\\nResponse:\", response)\n",
    "            return []\n",
    "    \n",
    "    return []\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def generate_final_report_async(session, user_query, all_contexts):\n",
    "    \"\"\"\n",
    "    Generates a comprehensive final report using all gathered contexts relevant to the user's query.\n",
    "    \n",
    "    The function prompts an LLM to synthesize a well-structured and detailed report, incorporating key insights,\n",
    "    citations, and conclusions from the collected contexts.\n",
    "    \n",
    "    Args:\n",
    "        session: The HTTP session used for making asynchronous API requests.\n",
    "        user_query (str): The original user query.\n",
    "        all_contexts (list of str): A list of extracted relevant contexts from various sources.\n",
    "    \n",
    "    Returns:\n",
    "        str: A comprehensive report addressing the user's query, integrating all relevant insights and citations.\n",
    "    \"\"\"\n",
    "    context_combined = \"\\n\".join(all_contexts)\n",
    "    prompt = (\n",
    "        \"You are an expert researcher and report writer. Based on the gathered contexts below and the original query, \"\n",
    "        \"write a comprehensive, well-structured, and detailed report that thoroughly addresses the query. \"\n",
    "        \"Incorporate all relevant insights, citation URLs, and conclusions without extraneous commentary. \"\n",
    "        \"Ensure clarity, coherence, and factual accuracy.\"\n",
    "    )\n",
    "    \n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"You are a skilled report writer.\"},\n",
    "        {\"role\": \"user\", \"content\": f\"User Query: {user_query}\\n\\nGathered Relevant Contexts:\\n{context_combined}\\n\\n{prompt}\"}\n",
    "    ]\n",
    "    \n",
    "    report = await call_openrouter_async(session, messages)\n",
    "    \n",
    "    return report.strip() if report else \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def process_link(session, link, user_query, link_data):\n",
    "    \"\"\"\n",
    "    Processes a single link by evaluating its usefulness and extracting relevant context if applicable.\n",
    "    \n",
    "    This function uses the pre-fetched content from the `unique_links` dictionary to determine\n",
    "    the relevance of the page to the user's query and extracts meaningful information if the page\n",
    "    is deemed useful.\n",
    "    \n",
    "    Args:\n",
    "        session: The HTTP session used for making asynchronous API requests.\n",
    "        link (str): The URL of the webpage to process.\n",
    "        user_query (str): The original user query.\n",
    "        link_data (dict): A dictionary containing the following keys:\n",
    "            - \"query\": The search query that led to the webpage.\n",
    "            - \"content\": The extracted content from the webpage.\n",
    "            - \"raw_content\": The raw content from the webpage.\n",
    "    \n",
    "    Returns:\n",
    "        str or None: Extracted relevant context if the page is useful, otherwise None.\n",
    "    \"\"\"\n",
    "    print(f\"Processing link: {link}\")\n",
    "    \n",
    "    # Extract content from link_data\n",
    "    page_text = link_data.get(\"content\")  # Use \"content\" or \"raw_content\" as needed\n",
    "    search_query = link_data.get(\"query\")\n",
    "    \n",
    "    if not page_text:\n",
    "        print(f\"No content available for {link}\")\n",
    "        return None\n",
    "    \n",
    "    # Evaluate the usefulness of the page\n",
    "    usefulness = await is_page_useful_async(session, user_query, page_text)\n",
    "    print(f\"Page usefulness for {link}: {usefulness}\")\n",
    "    \n",
    "    # If the page is useful, extract relevant context\n",
    "    if usefulness == \"Yes\":\n",
    "        context = await extract_relevant_context_async(session, user_query, search_query, page_text)\n",
    "        if context:\n",
    "            print(f\"Extracted context from {link} (first 200 chars): {context[:200]}\")\n",
    "            return context\n",
    "    \n",
    "    return None\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "id": "GJTo96a7DGUz"
   },
   "outputs": [],
   "source": [
    "async def async_main():\n",
    "    \"\"\"\n",
    "    Perform an iterative deep research process using LLM-generated search queries and web scraping.\n",
    "\n",
    "    This method:\n",
    "    1. Takes a user query and an optional iteration limit as input.\n",
    "    2. Generates initial search queries using an LLM.\n",
    "    3. Performs web searches for each query and aggregates unique links.\n",
    "    4. Fetches, judges, and extracts useful content from each link.\n",
    "    5. Iteratively asks the LLM if more searches are needed and generates new queries.\n",
    "    6. Compiles a final report based on the aggregated research data.\n",
    "\n",
    "    Example:\n",
    "        >>> asyncio.run(async_main())\n",
    "        Enter your deep research query/topic: What is the impact of AI on healthcare?\n",
    "        Enter maximum number of iterations (default 10): 5\n",
    "        === Iteration 1 ===\n",
    "        Aggregated 15 unique links from this iteration.\n",
    "        LLM provided new search queries: ['AI in healthcare research', 'benefits of AI in medicine']\n",
    "        === Iteration 2 ===\n",
    "        Aggregated 10 unique links from this iteration.\n",
    "        LLM indicated that no further research is needed.\n",
    "        Generating final report...\n",
    "        ==== FINAL REPORT ====\n",
    "        \"The impact of AI on healthcare is significant...\"\n",
    "    \"\"\"\n",
    "    # Get user input\n",
    "    user_query = input(\"Enter your deep research query/topic: \").strip()\n",
    "    iter_limit_input = input(\"Enter maximum number of iterations (default 10): \").strip()\n",
    "    iteration_limit = int(iter_limit_input) if iter_limit_input.isdigit() else 10\n",
    "\n",
    "    # Initialize research data\n",
    "    aggregated_contexts: List[str] = []  # All useful contexts from every iteration\n",
    "    all_search_queries: List[str] = []   # Every search query used across iterations\n",
    "    iteration = 0\n",
    "\n",
    "    async with aiohttp.ClientSession() as session:\n",
    "\n",
    "        new_search_queries = await generate_search_queries_async(session, user_query)\n",
    "        if not new_search_queries:\n",
    "            print(\"No search queries were generated by the LLM. Exiting.\")\n",
    "            return\n",
    "        all_search_queries.extend(new_search_queries)\n",
    "\n",
    "        while iteration < iteration_limit:\n",
    "            print(f\"\\n=== Iteration {iteration + 1} ===\")\n",
    "            iteration_contexts: List[str] = []\n",
    "\n",
    "            # For each search query, perform Tavily searches concurrently.\n",
    "            search_tasks = [perform_search_async(session, query) for query in new_search_queries]\n",
    "            search_results = await asyncio.gather(*search_tasks)\n",
    "            \n",
    "            # Aggregate all unique links from all search queries of this iteration.\n",
    "            # Map each unique link to its associated data (query, content, raw_content).\n",
    "            unique_links = {}\n",
    "            for idx, result in enumerate(search_results):\n",
    "                query = new_search_queries[idx]\n",
    "                if result and \"results\" in result:  # Ensure the result is valid and contains \"results\"\n",
    "                    for item in result[\"results\"]:\n",
    "                        link = item.get(\"url\")\n",
    "                        if link and link not in unique_links:  # Ensure the link is valid and not already processed\n",
    "                            unique_links[link] = {\n",
    "                                \"query\": query,  # The search query that produced this link\n",
    "                                \"content\": item.get(\"content\"),  # Extracted content\n",
    "                                \"raw_content\": item.get(\"raw_content\")  # Raw content\n",
    "                            }\n",
    "            \n",
    "            print(f\"Aggregated {len(unique_links)} unique links from this iteration.\")\n",
    "\n",
    "            # Process each link concurrently: fetch, judge, and extract context\n",
    "            link_tasks = [\n",
    "                process_link(session, link, user_query, unique_links[link])\n",
    "                for link in unique_links\n",
    "            ]\n",
    "            link_results = await asyncio.gather(*link_tasks)\n",
    "            \n",
    "            # Collect non-None contexts\n",
    "            iteration_contexts = [res for res in link_results if res]\n",
    "            if iteration_contexts:\n",
    "                aggregated_contexts.extend(iteration_contexts)\n",
    "            else:\n",
    "                print(\"No useful contexts were found in this iteration.\")\n",
    "            \n",
    "\n",
    "            new_search_queries = await get_new_search_queries_async(\n",
    "                session, user_query, all_search_queries, aggregated_contexts\n",
    "            )\n",
    "            if new_search_queries == \"<done>\":\n",
    "                print(\"LLM indicated that no further research is needed.\")\n",
    "                break\n",
    "            elif new_search_queries:\n",
    "                print(\"LLM provided new search queries:\", new_search_queries)\n",
    "                all_search_queries.extend(new_search_queries)\n",
    "            else:\n",
    "                print(\"LLM did not provide any new search queries. Ending the loop.\")\n",
    "                break\n",
    "\n",
    "            iteration += 1\n",
    "\n",
    "        print(\"\\nPreparing final report...\")\n",
    "        final_report = await generate_final_report_async(session, user_query, aggregated_contexts)\n",
    "        print(\"\\n==== DEEP RESEARCH REPORT ====\\n\")\n",
    "        print(final_report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Why Deep Research Outperforms LLM Prompting**\n",
    "\n",
    "- **Real-Time & Reliable** – Access to current, verifiable data; no hallucinations.\n",
    "- **Precision & Accuracy** – Technical specs, numbers, and documentation that LLMs may distort.\n",
    "- **Specialized Expertise** – Industry regulations, niche knowledge, and historical insights.\n",
    "- **Evidence-Backed Insights** – Case studies, market trends, and real-world validation.\n",
    "\n",
    "**Bottom Line: LLMs assist, but deep research ensures truth, accuracy, and trust—critical for high-stakes decisions.**\n",
    "\n",
    "**Let's Put our deep research to test using the question below :**\n",
    "\n",
    "We can conduct a deep research on any topic of our choice such as below.\n",
    "\n",
    "**Enter your deep research query/topic:**  What are the key differences in design philosophies between the Boeing 787 and Airbus A350, and how do they impact fuel efficiency?\n",
    "\n",
    "**Enter maximum number of iterations (default 10):**  3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "id": "46Q5XpapDJZT"
   },
   "outputs": [
    {
     "name": "stdin",
     "output_type": "stream",
     "text": [
      "Enter your deep research query/topic:  What are the key differences in design philosophies between the Boeing 787 and Airbus A350, and how do they impact fuel efficiency?\n",
      "Enter maximum number of iterations (default 10):  3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== Iteration 1 ===\n",
      "Aggregated 35 unique links from this iteration.\n",
      "Processing link: https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://aerospaceer.com/boeing-b787-vs-airbus-a350/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://myeasyflight.substack.com/p/plane-comparison-airbus-a350-vs-boeing\n",
      "Processing link: https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/\n",
      "Processing link: https://simpleflying.com/787-a350-composite/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-a-mechanics-view/\n",
      "Processing link: https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/\n",
      "Processing link: https://thisvsthat.io/a350-vs-b787\n",
      "Processing link: https://www.coursesidekick.com/industrial-engineering/907651\n",
      "Processing link: https://myaircraftcost.com/boeing-dreamliner-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison-2/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/\n",
      "Processing link: https://theluxurytravelexpert.com/a350-vs-b787-dreamliner-best-plane/\n",
      "Processing link: https://www.longdom.org/open-access/wings-of-innovation-and-unraveling-the-secrets-of-aircraft-design-107578.html\n",
      "Processing link: https://aeroplanetech.com/airbus-a350-innovations/\n",
      "Processing link: https://awbi.org/modern-aircraft-essentials/\n",
      "Processing link: https://www.afrlhorizons.com/the-future-of-aeroelastic-technology-in-aviation/\n",
      "Processing link: https://mycarmakesnoise.com/general/unbelievable-feats-of-engineering-in-modern-aircraft-design/\n",
      "Processing link: https://aeroplanetech.com/high-speed-performance-factors/\n",
      "Processing link: https://aeroplanetech.com/twin-aisle-aircraft/\n",
      "Processing link: https://simpleflying.com/boeing-787-dreamliner-impressive-wing-flex/\n",
      "Processing link: https://medium.com/deep-in-deeptech/redesigning-aircraft-a-new-era-of-aerodynamics-0b73654be766\n",
      "Processing link: https://ijcrt.org/papers/IJCRT2406889.pdf\n",
      "Processing link: https://aviationbenefits.org/environmental-efficiency/climate-action/efficient-technology/\n",
      "Processing link: https://aviationguy.com/2013/12/05/airbus-and-mit-teaming-up-to-make-aircraft-construction-lighter-and-cheaper/\n",
      "Processing link: https://www.executivetraveller.com/news/airbus-lightweight-composite-a320-future-wing-aims-to-best-boeing\n",
      "Processing link: https://www2.gvsu.edu/peirsonb/term_paper.pdf\n",
      "Processing link: https://www.techbriefs.com/component/content/article/33914-lightweighting-in-aerospace-component-and-system-design\n",
      "Processing link: https://bydanjohnson.com/boeing-airbus-explore-light-aircraft/\n",
      "Processing link: https://www.airbus.com/en/newsroom/news/2017-08-composites-airbus-continues-to-shape-the-future\n",
      "Processing link: https://www.sciencedirect.com/topics/engineering/boeing-787-dreamliner\n",
      "Processing link: https://www.sciencedirect.com/science/article/pii/1044580395000666\n",
      "Processing link: https://www.uni-stuttgart.de/en/research/forschung-leben/10-2018/lightweight-aircraft/\n",
      "Page usefulness for https://aeroplanetech.com/high-speed-performance-factors/: No\n",
      "Page usefulness for https://www.longdom.org/open-access/wings-of-innovation-and-unraveling-the-secrets-of-aircraft-design-107578.html: No\n",
      "Page usefulness for https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/: Yes\n",
      "Page usefulness for https://mycarmakesnoise.com/general/unbelievable-feats-of-engineering-in-modern-aircraft-design/: No\n",
      "Page usefulness for https://www.afrlhorizons.com/the-future-of-aeroelastic-technology-in-aviation/: No\n",
      "Page usefulness for https://thisvsthat.io/a350-vs-b787: No\n",
      "Page usefulness for https://www.sciencedirect.com/topics/engineering/boeing-787-dreamliner: Yes\n",
      "Page usefulness for https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/: Yes\n",
      "Page usefulness for https://www.uni-stuttgart.de/en/research/forschung-leben/10-2018/lightweight-aircraft/: No\n",
      "Page usefulness for https://www.coursesidekick.com/industrial-engineering/907651: Yes\n",
      "Page usefulness for https://medium.com/deep-in-deeptech/redesigning-aircraft-a-new-era-of-aerodynamics-0b73654be766: No\n",
      "Page usefulness for https://www.techbriefs.com/component/content/article/33914-lightweighting-in-aerospace-component-and-system-design: No\n",
      "Page usefulness for https://www.airbus.com/en/newsroom/news/2017-08-composites-airbus-continues-to-shape-the-future: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison-2/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://www2.gvsu.edu/peirsonb/term_paper.pdf: Yes\n",
      "Page usefulness for https://bydanjohnson.com/boeing-airbus-explore-light-aircraft/: No\n",
      "Page usefulness for https://simpleflying.com/787-a350-composite/: Yes\n",
      "Page usefulness for https://myeasyflight.substack.com/p/plane-comparison-airbus-a350-vs-boeing: Yes\n",
      "Page usefulness for https://aviationbenefits.org/environmental-efficiency/climate-action/efficient-technology/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://ijcrt.org/papers/IJCRT2406889.pdf: No\n",
      "Page usefulness for https://aerospaceer.com/boeing-b787-vs-airbus-a350/: Yes\n",
      "Page usefulness for https://simpleflying.com/boeing-787-dreamliner-impressive-wing-flex/: No\n",
      "Page usefulness for https://www.executivetraveller.com/news/airbus-lightweight-composite-a320-future-wing-aims-to-best-boeing: No\n",
      "Page usefulness for https://awbi.org/modern-aircraft-essentials/: No\n",
      "Page usefulness for https://aviationguy.com/2013/12/05/airbus-and-mit-teaming-up-to-make-aircraft-construction-lighter-and-cheaper/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-a-mechanics-view/: Yes\n",
      "Page usefulness for https://aeroplanetech.com/twin-aisle-aircraft/: No\n",
      "Page usefulness for https://www.sciencedirect.com/science/article/pii/1044580395000666: No\n",
      "Page usefulness for https://theluxurytravelexpert.com/a350-vs-b787-dreamliner-best-plane/: No\n",
      "Page usefulness for https://aeroplanetech.com/airbus-a350-innovations/: Yes\n",
      "Page usefulness for https://myaircraftcost.com/boeing-dreamliner-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/: Yes\n",
      "Extracted context from https://myaircraftcost.com/boeing-dreamliner-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): Design Philosophy and Fuel Efficiency:\n",
      "- Boeing 787 Dreamliner achieves 20% better fuel efficiency compared to older aircraft\n",
      "- Fuel efficiency is attributed to lightweight design and advanced engines\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): Boeing 787 is made up of approximately 50% composite materials, which significantly contributes to its lightweight structure and improves fuel efficiency. Boeing's official materials indicate this des\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/ (first 200 chars): The webpage content provides limited information about the Boeing 787 and Airbus A350's design philosophies. The excerpt indicates that the Airbus A350-900 uses a mixed-material structure combining al\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-a-mechanics-view/ (first 200 chars): The webpage excerpt provides limited context about the A350's design, mentioning a hybrid structure combining aluminum alloys and composite materials. However, the provided content is incomplete and d\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): Based on the truncated webpage content, the key insight is about the Boeing 787's design philosophy of using composite materials:\n",
      "\n",
      "The Boeing 787 is constructed with approximately 50% composite materi\n",
      "Extracted context from https://myeasyflight.substack.com/p/plane-comparison-airbus-a350-vs-boeing (first 200 chars): The Boeing 787 features a raked wingtip design while the Airbus A350 features an embedded winglet. The Boeing 787 is made of 50% composite material. The different wing designs suggest variations in ae\n",
      "Extracted context from https://aerospaceer.com/boeing-b787-vs-airbus-a350/ (first 200 chars): Key design differences in composite materials:\n",
      "- Boeing 787: 50% composite materials\n",
      "- Airbus A350: 53% composite materials\n",
      "\n",
      "Boeing 787 wing design:\n",
      "- Wings designed to flex during flight\n",
      "- Reduces tu\n",
      "Extracted context from https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/ (first 200 chars): Composite materials are a key design philosophy for the Boeing 787, which uses composites for approximately half of its structure, including the fuselage and wings. This approach is aimed at reducing \n",
      "Extracted context from https://www.sciencedirect.com/topics/engineering/boeing-787-dreamliner (first 200 chars): The Airbus A350 XWB uses advanced lightweight materials including:\n",
      "- Composites in fuselage, wings, and tail sections\n",
      "- Aluminum-lithium alloys in floor beams and structural frames\n",
      "\n",
      "The excerpt provid\n",
      "Extracted context from https://www2.gvsu.edu/peirsonb/term_paper.pdf (first 200 chars): The provided webpage excerpt provides limited information directly addressing the user's query about design philosophies and fuel efficiency differences between the Boeing 787 and Airbus A350. The onl\n",
      "Extracted context from https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/ (first 200 chars): The Boeing 787 Dreamliner uses a groundbreaking design with its fuselage composed of 50% carbon-fiber-reinforced plastic (CFRP), which significantly reduces aircraft weight. This material choice enabl\n",
      "Extracted context from https://aeroplanetech.com/airbus-a350-innovations/ (first 200 chars): The Airbus A350 features an enhanced aerodynamic profile designed to minimize drag and optimize fuel consumption. The design demonstrates a focus on engineering innovations that improve overall aircra\n",
      "Extracted context from https://simpleflying.com/787-a350-composite/ (first 200 chars): Boeing 787:\n",
      "- 50% composites by weight\n",
      "- 80% composites by volume\n",
      "- Additional material composition:\n",
      "  - 20% aluminum\n",
      "  - 15% titanium\n",
      "  - 10% steel\n",
      "  - 5% other materials\n",
      "\n",
      "Key design philosophy focus\n",
      "Extracted context from https://www.coursesidekick.com/industrial-engineering/907651 (first 200 chars): Design Philosophy and Fuel Efficiency Comparison:\n",
      "\n",
      "Both the Boeing 787 and Airbus A350 are technologically advanced commercial aircraft focused on:\n",
      "- Passenger comfort\n",
      "- Maximizing fuel efficiency\n",
      "- M\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/ (first 200 chars): Design Philosophy and Fuel Efficiency Differences:\n",
      "\n",
      "Boeing 787:\n",
      "- Notable for its fuel efficiency\n",
      "- Lower maintenance costs\n",
      "\n",
      "Airbus A350:\n",
      "- Higher passenger capacity\n",
      "- Longer range capability\n",
      "\n",
      "Both ai\n",
      "LLM provided new search queries: ['Detailed comparison of Boeing 787 and Airbus A350 wing design aerodynamics', 'Specific fuel efficiency metrics for Boeing 787 and Airbus A350', 'Quantitative analysis of composite material impact on aircraft performance', 'Manufacturing techniques for composite aircraft structures Boeing and Airbus']\n",
      "\n",
      "=== Iteration 2 ===\n",
      "Aggregated 35 unique links from this iteration.\n",
      "Processing link: https://www.airliners.net/forum/viewtopic.php?t=1458471\n",
      "Processing link: https://aerofleets.com/boeing-787-vs-airbus-a350-a-comparison/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-8-vs-airbus-a350-900-a-comprehensive-comparison/\n",
      "Processing link: https://www.webflite.com/compare/airliners/787-9/a350-900\n",
      "Processing link: https://www.webflite.com/compare/airliners/787-8/a350-900\n",
      "Processing link: https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison-2/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/\n",
      "Processing link: https://aerospaceer.com/boeing-b787-vs-airbus-a350/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/\n",
      "Processing link: https://www.aircraft-commerce.com/wp-content/uploads/aircraft-commerce-docs/Analysis+&+Fleet+Planning/2016/108_ANALYSIS.pdf\n",
      "Processing link: https://simpleflying.com/boeing-787-vs-the-airbus-a350/\n",
      "Processing link: https://www.semanticscholar.org/paper/IMPACT-OF-COMPOSITE-MATERIALS-ON-AIRCRAFT-WEIGHT-IN-Rafi/1d16a9a07df543c85de4269726b03a1dac1587d9\n",
      "Processing link: https://aeroplanetech.com/composite-materials-usage/\n",
      "Processing link: https://www.sciencedirect.com/science/article/pii/S2214785317304224\n",
      "Processing link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7435959/\n",
      "Processing link: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7956575/\n",
      "Processing link: https://link.springer.com/chapter/10.1007/978-981-99-6774-2_9\n",
      "Processing link: https://www.mdpi.com/2226-4310/12/2/122\n",
      "Processing link: https://www.sciencedirect.com/science/article/pii/S1350630714002593\n",
      "Processing link: https://www.sciencedirect.com/science/article/pii/S221478532106781X\n",
      "Processing link: https://www.researchgate.net/publication/388778130_Experimental_and_Numerical_Assessment_of_Crashworthiness_Properties_of_Composite_Materials_A_Review\n",
      "Processing link: https://www.spiritaero.com/programs/fabrication/composite-fabrication/\n",
      "Processing link: https://sassofia.com/blog/introduction-to-composite-materials-in-aviation-and-tap-testing/\n",
      "Processing link: https://www.ati.org.uk/wp-content/uploads/2021/08/insight_9-composites_amended-2018-09-20.pdf\n",
      "Processing link: https://patents.google.com/patent/US8303758B2/en\n",
      "Processing link: https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470686652.eae365\n",
      "Processing link: https://www.aerospacemanufacturinganddesign.com/article/amd0312-composites-aircraft-focus/\n",
      "Processing link: https://www.secotools.com/article/127369?language=en\n",
      "Processing link: https://www.sciencedirect.com/science/article/pii/S1359835X21004747\n",
      "Processing link: https://www.youtube.com/watch?v=SeSK5E5DJ98\n",
      "Processing link: https://www.compositesworld.com/opportunities-challenges-for-composites-in-future-aircraft/\n",
      "Page usefulness for https://aeroplanetech.com/composite-materials-usage/: No\n",
      "Page usefulness for https://www.semanticscholar.org/paper/IMPACT-OF-COMPOSITE-MATERIALS-ON-AIRCRAFT-WEIGHT-IN-Rafi/1d16a9a07df543c85de4269726b03a1dac1587d9: Yes\n",
      "Page usefulness for https://www.aircraft-commerce.com/wp-content/uploads/aircraft-commerce-docs/Analysis+&+Fleet+Planning/2016/108_ANALYSIS.pdf: Yes\n",
      "Page usefulness for https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/: No\n",
      "Page usefulness for https://www.airliners.net/forum/viewtopic.php?t=1458471: Yes\n",
      "Page usefulness for https://www.sciencedirect.com/science/article/pii/S221478532106781X: No\n",
      "Page usefulness for https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7956575/: No\n",
      "Page usefulness for https://sassofia.com/blog/introduction-to-composite-materials-in-aviation-and-tap-testing/: No\n",
      "Page usefulness for https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/: Yes\n",
      "Page usefulness for https://link.springer.com/chapter/10.1007/978-981-99-6774-2_9: No\n",
      "Page usefulness for https://www.ati.org.uk/wp-content/uploads/2021/08/insight_9-composites_amended-2018-09-20.pdf: Yes\n",
      "Page usefulness for https://www.webflite.com/compare/airliners/787-8/a350-900: No\n",
      "Page usefulness for https://www.secotools.com/article/127369?language=en: No\n",
      "Page usefulness for https://www.mdpi.com/2226-4310/12/2/122: Yes\n",
      "Page usefulness for https://www.webflite.com/compare/airliners/787-9/a350-900: No\n",
      "Page usefulness for https://www.sciencedirect.com/science/article/pii/S2214785317304224: No\n",
      "Page usefulness for https://pmc.ncbi.nlm.nih.gov/articles/PMC7435959/: No\n",
      "Page usefulness for https://aerospaceer.com/boeing-b787-vs-airbus-a350/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/: No\n",
      "Page usefulness for https://www.sciencedirect.com/science/article/pii/S1359835X21004747: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://www.youtube.com/watch?v=SeSK5E5DJ98: No\n",
      "Page usefulness for https://www.compositesworld.com/opportunities-challenges-for-composites-in-future-aircraft/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison-2/: No\n",
      "Page usefulness for https://simpleflying.com/boeing-787-vs-the-airbus-a350/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/: Yes\n",
      "Page usefulness for https://www.spiritaero.com/programs/fabrication/composite-fabrication/: No\n",
      "Page usefulness for https://www.aerospacemanufacturinganddesign.com/article/amd0312-composites-aircraft-focus/: No\n",
      "Page usefulness for https://aerofleets.com/boeing-787-vs-airbus-a350-a-comparison/: Yes\n",
      "Page usefulness for https://www.researchgate.net/publication/388778130_Experimental_and_Numerical_Assessment_of_Crashworthiness_Properties_of_Composite_Materials_A_Review: No\n",
      "Page usefulness for https://onlinelibrary.wiley.com/doi/abs/10.1002/9780470686652.eae365: No\n",
      "Page usefulness for https://patents.google.com/patent/US8303758B2/en: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://www.sciencedirect.com/science/article/pii/S1350630714002593: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-8-vs-airbus-a350-900-a-comprehensive-comparison/: No\n",
      "Extracted context from https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/ (first 200 chars): Unfortunately, the provided webpage content does not contain substantive details about the wing design aerodynamics or design philosophies of the Boeing 787 and Airbus A350. The excerpt appears to be \n",
      "Extracted context from https://www.mdpi.com/2226-4310/12/2/122 (first 200 chars): The webpage content provides limited information directly addressing the user's query. The only relevant extract is:\n",
      "\n",
      "The Boeing 787 and Airbus A350 are both composed of more than 50% composite materi\n",
      "Extracted context from https://www.airliners.net/forum/viewtopic.php?t=1458471 (first 200 chars): The 787 and A350 have slightly different approaches to composite materials in aircraft construction. The 787 uses 50% composites overall, while the A350 uses 53% composites. Boeing's wing design is mo\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): Boeing 787 Dreamliner is designed to be 20% more fuel-efficient compared to older aircraft. The A350 generally has lower maintenance costs. Both aircraft are advanced models designed for efficiency an\n",
      "Extracted context from https://www.ati.org.uk/wp-content/uploads/2021/08/insight_9-composites_amended-2018-09-20.pdf (first 200 chars): Based on the truncated webpage content provided, there is insufficient information to comprehensively answer the user's query about the design philosophies and fuel efficiency differences between the \n",
      "Extracted context from https://www.aircraft-commerce.com/wp-content/uploads/aircraft-commerce-docs/Analysis+&+Fleet+Planning/2016/108_ANALYSIS.pdf (first 200 chars): Based on the truncated content, there is limited specific information directly addressing the full query. The only relevant extracted details are:\n",
      "\n",
      "- The Boeing 787-8 entered service in 2011\n",
      "- The Boe\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/ (first 200 chars): Fuel efficiency comparison between Boeing 787 and Airbus A350:\n",
      "- Boeing 787-9 is notable for its superior fuel efficiency and lower maintenance costs\n",
      "- Airbus A350-900 offers higher passenger capacity\n",
      "Extracted context from https://aerofleets.com/boeing-787-vs-airbus-a350-a-comparison/ (first 200 chars): The Boeing 787 and Airbus A350 share similar specifications across their variant models. The primary design differences are focused on the fuselage and wing architectures. While the webpage content do\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): The Boeing 787 is lighter and more fuel-efficient, with lower operating costs due to advanced materials and design. The A350's aerodynamic design also enhances fuel efficiency, particularly for long-h\n",
      "Extracted context from https://www.semanticscholar.org/paper/IMPACT-OF-COMPOSITE-MATERIALS-ON-AIRCRAFT-WEIGHT-IN-Rafi/1d16a9a07df543c85de4269726b03a1dac1587d9 (first 200 chars): The content provided appears to be a truncated excerpt from a technical document and does not contain sufficient detailed information to fully address the specific differences between the Boeing 787 a\n",
      "LLM provided new search queries: ['Boeing 787 and Airbus A350 wing aerodynamics performance analysis', 'Detailed composite material weight reduction impact on aircraft fuel efficiency', 'Comparative engineering design philosophy Boeing 787 vs Airbus A350 technical white papers', 'Precise fuel efficiency metrics and operational cost differences between Boeing 787 and A350']\n",
      "\n",
      "=== Iteration 3 ===\n",
      "Aggregated 29 unique links from this iteration.\n",
      "Processing link: https://www.coursehero.com/file/209840758/Aircraft-Design-and-Performance-Module-23docx/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-8-vs-airbus-a350-900-a-comprehensive-comparison/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/\n",
      "Processing link: https://thisvsthat.io/a350-vs-b787\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison-2/\n",
      "Processing link: https://simpleflying.com/boeing-787-vs-the-airbus-a350/\n",
      "Processing link: https://aerospaceer.com/boeing-b787-vs-airbus-a350/\n",
      "Processing link: https://www.webflite.com/compare/airliners/787-9/a350-900\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/\n",
      "Processing link: https://www.semanticscholar.org/paper/IMPACT-OF-COMPOSITE-MATERIALS-ON-AIRCRAFT-WEIGHT-IN-Rafi/1d16a9a07df543c85de4269726b03a1dac1587d9\n",
      "Processing link: https://www.sensxpert.com/blog/impact-composites-aviation-industry/\n",
      "Processing link: https://www.mhi.co.jp/technology/review/pdf/e594/e594170.pdf\n",
      "Processing link: https://www.aerodinecomposites.com/news/how-aircraft-composites-are-making-aircraft-lighter-and-more-fuel-efficient/\n",
      "Processing link: https://www.wecanfigurethisout.org/ENERGY/Web_notes/Energy_Consumption/Transportation_Supporting_Files/Composites+in+the+Aircraft+Industry+-+Appropedia.pdf\n",
      "Processing link: https://www.parker.com/content/dam/parker/na/united-states/industries/aerospace/pdfs-aerospace/Aircraft+Composite+Material+Background.pdf\n",
      "Processing link: https://aerospaceer.com/composite-materials-in-aircraft/\n",
      "Processing link: https://pia.edu/blogs/composites-in-the-aircraft-industry/\n",
      "Processing link: https://www.researchgate.net/publication/387497678_COMPOSITE_MATERIALS_USED_IN_ARJ21_AIRCRAFT\n",
      "Processing link: https://www.cdiproducts.com/blog/revolutionizing-aerospace-the-impact-of-composites-in-aviation\n",
      "Processing link: https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/\n",
      "Processing link: https://www.coursesidekick.com/industrial-engineering/907651\n",
      "Processing link: https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a380-a-detailed-comparison/\n",
      "Processing link: https://aeropeep.com/what-aircraft-is-better-the-airbus-a350-900-or-the-boeing-787-9/\n",
      "Processing link: https://airinsight.com/comparing-the-airbus-a350-900-and-boeing-787-9/\n",
      "Processing link: https://www.aircraft-commerce.com/wp-content/uploads/aircraft-commerce-docs/Analysis+&+Fleet+Planning/2016/108_ANALYSIS.pdf\n",
      "Processing link: https://myaircraftcost.com/boeing-787-9-vs-boeing-787-10-a-comprehensive-comparison/\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://www.semanticscholar.org/paper/IMPACT-OF-COMPOSITE-MATERIALS-ON-AIRCRAFT-WEIGHT-IN-Rafi/1d16a9a07df543c85de4269726b03a1dac1587d9: Yes\n",
      "Page usefulness for https://aeropeep.com/what-aircraft-is-better-the-airbus-a350-900-or-the-boeing-787-9/: Yes\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-8-vs-airbus-a350-900-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://www.parker.com/content/dam/parker/na/united-states/industries/aerospace/pdfs-aerospace/Aircraft+Composite+Material+Background.pdf: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison-2/: No\n",
      "Page usefulness for https://www.webflite.com/compare/airliners/787-9/a350-900: Yes\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/: Yes\n",
      "Page usefulness for https://thisvsthat.io/a350-vs-b787: Yes\n",
      "Page usefulness for https://www.aerodinecomposites.com/news/how-aircraft-composites-are-making-aircraft-lighter-and-more-fuel-efficient/: No\n",
      "Page usefulness for https://www.coursesidekick.com/industrial-engineering/907651: No\n",
      "Page usefulness for https://www.coursehero.com/file/209840758/Aircraft-Design-and-Performance-Module-23docx/: Yes\n",
      "Page usefulness for https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/: Yes\n",
      "Page usefulness for https://aerospaceer.com/boeing-b787-vs-airbus-a350/: Yes\n",
      "Page usefulness for https://www.mhi.co.jp/technology/review/pdf/e594/e594170.pdf: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a380-a-detailed-comparison/: Yes\n",
      "Page usefulness for https://theflyingengineer.com/boeing-787-vs-airbus-a350-battle-of-the-skies/: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-a350-900-a-comprehensive-comparison/: No\n",
      "Page usefulness for https://www.cdiproducts.com/blog/revolutionizing-aerospace-the-impact-of-composites-in-aviation: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://www.aircraft-commerce.com/wp-content/uploads/aircraft-commerce-docs/Analysis+&+Fleet+Planning/2016/108_ANALYSIS.pdf: Yes\n",
      "Page usefulness for https://aerospaceer.com/composite-materials-in-aircraft/: Yes\n",
      "Page usefulness for https://airinsight.com/comparing-the-airbus-a350-900-and-boeing-787-9/: No\n",
      "Page usefulness for https://www.sensxpert.com/blog/impact-composites-aviation-industry/: Yes\n",
      "Page usefulness for https://www.wecanfigurethisout.org/ENERGY/Web_notes/Energy_Consumption/Transportation_Supporting_Files/Composites+in+the+Aircraft+Industry+-+Appropedia.pdf: Yes\n",
      "Page usefulness for https://pia.edu/blogs/composites-in-the-aircraft-industry/: No\n",
      "Page usefulness for https://www.researchgate.net/publication/387497678_COMPOSITE_MATERIALS_USED_IN_ARJ21_AIRCRAFT: No\n",
      "Page usefulness for https://myaircraftcost.com/boeing-787-9-vs-boeing-787-10-a-comprehensive-comparison/: Yes\n",
      "Page usefulness for https://simpleflying.com/boeing-787-vs-the-airbus-a350/: No\n",
      "Extracted context from https://aeropeep.com/what-aircraft-is-better-the-airbus-a350-900-or-the-boeing-787-9/ (first 200 chars): The A350 has a more advanced composite airframe, with 53% of the structure made from composites compared to 50% on the 787. This contributes to the A350's slightly lower operating empty weight and bet\n",
      "Extracted context from https://www.coursehero.com/file/209840758/Aircraft-Design-and-Performance-Module-23docx/ (first 200 chars): The Boeing 787-9 incorporates lightweight composite materials, a streamlined wing design, and advanced aerodynamics, resulting in reduced drag and enhanced fuel efficiency. The text provides limited d\n",
      "Extracted context from https://www.semanticscholar.org/paper/IMPACT-OF-COMPOSITE-MATERIALS-ON-AIRCRAFT-WEIGHT-IN-Rafi/1d16a9a07df543c85de4269726b03a1dac1587d9 (first 200 chars): Unfortunately, the provided webpage content appears to be truncated and does not contain sufficient information to comprehensively answer the user's specific query about the design differences between\n",
      "Extracted context from https://www.aircraft-commerce.com/wp-content/uploads/aircraft-commerce-docs/Analysis+&+Fleet+Planning/2016/108_ANALYSIS.pdf (first 200 chars): The 787 was originally pitched to offer fuel burns up to 20% lower than similar-sized previous generation aircraft. The 787-8 is about 3,500 to 5,500 nautical miles in range, slightly larger than the \n",
      "Extracted context from https://thisvsthat.io/a350-vs-b787 (first 200 chars): Aerodynamics and fuel efficiency are key differences between the Airbus A350 and Boeing 787. The A350 features advanced aerodynamics and efficient engines that contribute to its fuel efficiency. The B\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a380-a-detailed-comparison/ (first 200 chars): The Boeing 787-9 Dreamliner is designed with composite materials that reduce aircraft weight and improve fuel efficiency. While the provided content is limited, it offers a partial insight into Boeing\n",
      "Extracted context from https://www.webflite.com/compare/airliners/787-9/a350-900 (first 200 chars): Fuel capacity of Boeing 787-9: Not specified in this excerpt\n",
      "Fuel capacity of Airbus A350-900: 37,248 US gallons (141,000 liters)\n",
      "Range of Airbus A350-900: 8,300 nautical miles (15,372 kilometers)\n",
      "\n",
      "Th\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-dreamliner-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): Design Philosophy Differences:\n",
      "- Both Boeing 787 and Airbus A350 are advanced aircraft focused on efficiency and passenger comfort\n",
      "- Boeing 787 Dreamliner is designed to be 20% more fuel-efficient com\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-vs-airbus-a350-a-comprehensive-comparison/ (first 200 chars): The Boeing 787 is lighter and more fuel-efficient, with lower operating costs due to advanced materials and design. The A350's aerodynamic design also enhances fuel efficiency, particularly for long-h\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-vs-boeing-787-10-a-comprehensive-comparison/ (first 200 chars): Boeing 787-9: 2.6 liters/100 km fuel burn per seat\n",
      "Boeing 787-10: 2.8 liters/100 km fuel burn per seat\n",
      "\n",
      "The provided content offers limited details on design philosophies or comprehensive comparative \n",
      "Extracted context from https://myaircraftcost.com/boeing-787-9-dreamliner-vs-airbus-a350-900-a-comparison/ (first 200 chars): Airbus A350-900 features:\n",
      "- Higher purchase price but lower operational costs\n",
      "- Fuel efficiency improvement of approximately 25% compared to older aircraft\n",
      "- Spacious cabin with larger windows\n",
      "- Optim\n",
      "Extracted context from https://myaircraftcost.com/boeing-787-8-vs-airbus-a350-900-a-comprehensive-comparison/ (first 200 chars): Key wing design differences:\n",
      "- Boeing 787-8 has a distinctive raked wingtip design aimed at improving aerodynamic performance\n",
      "- Airbus A350-900 features a more traditional wing shape\n",
      "- Both aircraft a\n",
      "Extracted context from https://aerospaceer.com/boeing-b787-vs-airbus-a350/ (first 200 chars): Composite materials composition: Boeing 787 is made of 50% composite materials, which contributes to its lighter weight and improved fuel efficiency\n",
      "\n",
      "Fuel efficiency comparison: The A350 has slightly \n",
      "Extracted context from https://www.sensxpert.com/blog/impact-composites-aviation-industry/ (first 200 chars): In this truncated search result, the content provides limited context related to aircraft design and fuel efficiency. While the snippet mentions that composite materials like plastics contribute to we\n",
      "Extracted context from https://aerospaceer.com/composite-materials-in-aircraft/ (first 200 chars): Based on the webpage content provided, there is limited specific information directly addressing the detailed design philosophies of the Boeing 787 and Airbus A350. The only relevant extract is the ge\n",
      "Extracted context from https://www.wecanfigurethisout.org/ENERGY/Web_notes/Energy_Consumption/Transportation_Supporting_Files/Composites+in+the+Aircraft+Industry+-+Appropedia.pdf (first 200 chars): The provided webpage content offers limited information directly addressing the user's query. The key relevant extract is:\n",
      "\n",
      "Composite materials in aircraft design can reduce component weight by approx\n",
      "Extracted context from https://www.aviationfile.com/boeing-787-dreamliner-vs-airbus-a350-xwb-a-comprehensive-comparison-of-next-generation-wide-body-aircraft/ (first 200 chars): Based on the available content, here are the key extracted details relevant to the design philosophies and fuel efficiency differences:\n",
      "\n",
      "Both the Boeing 787 and Airbus A350 represent next-generation w\n",
      "LLM provided new search queries: ['Detailed engineering comparison Boeing 787 and Airbus A350 composite wing design aerodynamics', 'Quantitative fuel efficiency metrics Boeing 787 Dreamliner versus Airbus A350 XWB performance', 'Official technical specifications Boeing 787 and Airbus A350 composite material weight reduction strategies', 'Comparative analysis of Boeing 787 and Airbus A350 advanced materials engineering design philosophy']\n",
      "\n",
      "Preparing final report...\n",
      "\n",
      "==== DEEP RESEARCH REPORT ====\n",
      "\n",
      "Comparative Analysis: Design Philosophies and Fuel Efficiency of Boeing 787 and Airbus A350\n",
      "\n",
      "Executive Summary:\n",
      "This report examines the key design philosophies and fuel efficiency characteristics of the Boeing 787 Dreamliner and Airbus A350 XWB, two next-generation wide-body commercial aircraft that represent significant advancements in aerospace engineering.\n",
      "\n",
      "1. Composite Materials Design Philosophy\n",
      "\n",
      "Material Composition:\n",
      "- Boeing 787: 50% composites by weight (80% by volume)\n",
      "  - Material breakdown:\n",
      "    * 20% aluminum\n",
      "    * 15% titanium\n",
      "    * 10% steel\n",
      "    * 5% other materials\n",
      "\n",
      "- Airbus A350: 53% composites by weight\n",
      "  - Utilizes advanced aluminum-lithium alloys\n",
      "  - Composite materials in fuselage, wings, and tail sections\n",
      "\n",
      "2. Fuel Efficiency Characteristics\n",
      "\n",
      "Performance Metrics:\n",
      "- Boeing 787:\n",
      "  * 20% improved fuel efficiency compared to previous generation aircraft\n",
      "  * Fuel burn: 2.6-2.8 liters/100 km per seat\n",
      "  * Range: 3,500-5,500 nautical miles (787-8 variant)\n",
      "\n",
      "- Airbus A350:\n",
      "  * Approximately 25% fuel efficiency improvement\n",
      "  * Range: Up to 8,300 nautical miles\n",
      "  * Powered by Rolls-Royce Trent XWB engines\n",
      "\n",
      "3. Distinctive Design Approaches\n",
      "\n",
      "Boeing 787 Key Design Features:\n",
      "- Flexible wing design to reduce turbulence\n",
      "- Innovative single-barrel fuselage construction\n",
      "- Raked wingtip design\n",
      "- Focus on passenger comfort and lower maintenance costs\n",
      "\n",
      "Airbus A350 Key Design Features:\n",
      "- Curved, high-aspect-ratio wings\n",
      "- Embedded winglets\n",
      "- Wider fuselage for enhanced passenger space\n",
      "- Advanced aerodynamic profile to minimize drag\n",
      "\n",
      "4. Comparative Analysis\n",
      "\n",
      "Strengths:\n",
      "Boeing 787:\n",
      "- Lower maintenance costs\n",
      "- More flexible for different route configurations\n",
      "- Innovative lightweight design\n",
      "\n",
      "Airbus A350:\n",
      "- Longer range capabilities\n",
      "- Slightly higher passenger capacity\n",
      "- More advanced wing composite construction\n",
      "\n",
      "5. Technological Innovation Similarities\n",
      "\n",
      "Common Design Priorities:\n",
      "- Extensive use of composite materials\n",
      "- Focus on fuel efficiency\n",
      "- Enhanced passenger comfort\n",
      "- Reduced environmental impact\n",
      "- Advanced aerodynamic engineering\n",
      "\n",
      "Conclusion:\n",
      "Both the Boeing 787 and Airbus A350 represent breakthrough designs in commercial aviation, with marginally different approaches to achieving fuel efficiency and operational performance. The primary distinction lies in their specific design implementations rather than fundamental design philosophies.\n",
      "\n",
      "Recommendations:\n",
      "- Airlines should evaluate aircraft selection based on specific route requirements\n",
      "- Consider long-term operational costs beyond initial fuel efficiency metrics\n",
      "- Recognize that both aircraft offer significant technological advancements over previous generations\n",
      "\n",
      "Limitations of Analysis:\n",
      "This report is based on available published data and may not reflect the most recent technological updates or specific airline configurations.\n",
      "\n",
      "Prepared by: AI Research Assistant\n",
      "Date: [Current Date]\n",
      "\n",
      "Note: While comprehensive, this analysis synthesizes available information and should be complemented with manufacturer-specific technical documentation for absolute precision.\n"
     ]
    }
   ],
   "source": [
    "def main():\n",
    "    asyncio.run(async_main())\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyOe5BsaH0aplNCjknkFtnjg",
   "include_colab_link": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}