Feat: General File Reader Tool

[tybalex]

This PR adds a general File-Reader Tool by reusing existing knowledge ingestion code.
This tool can parse the content of input workspace file(word, pdfs, slides) and convert to markdown format and print to stdout by default or write to a workspace file.

This tool also helps address issues like obot-platform/obot#1405, by:

1. the agent uses File Reader tool to convert .pdf file to markdown format
2. the agent then uses the Summarizer tool to summarize the markdown format content.

@cjellick @thedadams I wonder if gptscript could support a syntax such that File Reader Tool is always a prerequisite of Summarizer Tool.

[cjellick]

Hm. This isn't really a file reader tool. It's a "Convert to markdown" tool. I'm not sure that's what we want.

I had envisioned a tool that reads a file and sends the contents directly to the LLM. Do you not think that approach will work well?

What happens if the user just says "Read the court filings file" or "What's the highest grossing office in my spreadsheet?" Will this tool be called?

I'm also not sure that we want to create double the artifacts in the workspace.

[tybalex]

Hm. This isn't really a file reader tool. It's a "Convert to markdown" tool. I'm not sure that's what we want.

Technically this tool parses the text content from pdf/pptx/docx documents, so it is a reader. In terms of the format, maybe not necessarily markdown, parsing the content to plain text would be enough. I guess markdown format is more readable?

I had envisioned a tool that reads a file and sends the contents directly to the LLM. Do you not think that approach will work well?

We can support both, and make it sends content back to llm by default. In the case of when the file has too much text, then it should probably write to a file and use a summarizer tool to handle the text.

What happens if the user just says "Read the court filings file" or "What's the highest grossing office in my spreadsheet?" Will this tool be called?

For structured data like a spreadsheet(I mean excel/csv/json .. etc), we will need to handle it separately.

[tybalex]

@cjellick I made an update, now this tool will send the content directly to LLM by default. If user specifies an output file, then it will do so.

[cloudnautique]

@tybalex what happens if the amount of text exceeds the context window? Is it something we handle gracefully?

[tybalex]

@tybalex what happens if the amount of text exceeds the context window? Is it something we handle gracefully?

that's a really good question. It is a problem when it directly returns back to the LLM, I will need to think about it.
If it writes to a file then I guess it's ok.