Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add: google custom search + website-scraper #471

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

iwilltry42
Copy link
Contributor

This PR

  1. makes the website-cleaner functions available as a package (not much code, but I think it's good to have consistency for any change done to the cleaning logic)
  2. adds a super basic website-scraper tool (which is not recursive (yet)) to get website content
  3. adds a basic google custom search tool leveraging the above.

Workflow:

  1. LLM calls the CSE search tool and gets a list of results back
  2. the LLM decides (based on title, excerpt and link), which links could be relevant
  3. the LLM (optionally) calls the website-scraper tool to get more information from links it deemed relevant
  4. result is returned as JSON {"answer": "<answer>", "sources": ["https://foo.com", "https://bar.de"]} which @ryanhopperlowe can use to generate citations

Issue obot-platform/obot#1935

@iwilltry42 iwilltry42 force-pushed the feat/google-search branch from 0c1ed3f to 7136665 Compare March 6, 2025 14:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant