Fetch extension fails for GUI but succeeds in CLI

[salman1993]

Describe the bug

the fetch extension fails in GUI (robots.txt error) but works in the CLI for the same prompt. tried reinstalling the extension in few different ways.

To Reproduce
Steps to reproduce the behavior:

1. Go to Goose app (make sure Fetch extension is enabled from extension browsing page) -> note that I am using internal Block version of the goose app
2. Run ths query - "read this doc - https://block.github.io/goose/docs/tutorials/custom-extensions/"
3. The same query fails in GUI with SSL cert error but works on CLI

Expected behavior
It should work on the GUI, same as it does on the CLI.

Screenshots

failure with GUI:

i also tried starting MCP server with uvx mcp-server-fetch --ignore-robots-txt & asked for more details:

Please provide following information:

• OS & Arch: mac OS (m2 chip)
• Interface: UI
• Version: 1.0.6-block
• Extensions enabled: Developer, Fetch
• Provider & Model: databricks - claude 3.5 sonnet v2

Additional context
I have also asked @zakiali to try this and he has seen the same error. Anecdotally, I have seen a lot of errors with fetch extension in GUI which tells me sth else is going on here.

Alternatively, its possible to scrape using Computer Controller but the results are much worse cause it doesn't convert to Markdown unlike the fetch extension

[yingjiehe-xyz]

Hi @lily-de, could you take a look?

[glenn-sq]

👋 experiencing similar behavior. fetch is failing in the Goose app because of robots.txt and it told me it couldn't skip trying to access it. However, fetch via the Goose CLI is working as expected.

[aljjen]

Describe the bug

the fetch extension fails in GUI (robots.txt error) but works in the CLI for the same prompt. tried reinstalling the extension in few different ways.

To Reproduce
Steps to reproduce the behavior:

1. Go to Goose app (make sure Fetch extension is enabled from extension browsing page) -> note that I am using internal Block version of the goose app
2. Run ths query - "read this doc - https://block.github.io/goose/docs/tutorials/custom-extensions/"
3. The same query fails in GUI with SSL cert error but works on CLI

Expected behavior
It should work on the GUI, same as it does on the CLI.

Screenshots

failure with GUI:

i also tried starting MCP server with uvx mcp-server-fetch --ignore-robots-txt & asked for more details:

Please provide following information:

• OS & Arch: mac OS (m2 chip)
• Interface: UI
• Version: 1.0.6-block
• Extensions enabled: Developer, Fetch
• Provider & Model: databricks - claude 3.5 sonnet v2

Additional context
I have also asked @zakiali to try this and he has seen the same error. Anecdotally, I have seen a lot of errors with fetch extension in GUI which tells me sth else is going on here.

Alternatively, its possible to scrape using Computer Controller but the results are much worse cause it doesn't convert to Markdown unlike the fetch extension

[salman1993]

I just tried this out with Goose.app version 1.0.4-block and it works with that version, so pretty sure this is a regression.

[salman1993]

just tried with 1.0.5-block.20250207 and now i am seeing the error: Failed to fetch robots.txt https://block.github.io/robots.txt due to a connection issue

this regression happened between 1.0.4 (~jan 31st) and 1.0.5 (feb 7)

[salman1993]

This is likely happening because when GOOSE_UV_REGISTRY is set, we set env var for SSL_CERT_FILE:

goose/ui/desktop/src/bin/uvx

Lines 72 to 93 in 57b0982

	if [ -n "${GOOSE_UV_REGISTRY:-}" ] && curl -s --head --fail "$GOOSE_UV_REGISTRY" > /dev/null; then
	log "Checking custom goose registry availability: $GOOSE_UV_REGISTRY"
	log "$GOOSE_UV_REGISTRY is accessible. Using it for UV registry."
	export UV_INDEX_URL="$GOOSE_UV_REGISTRY"
	if [ -n "${GOOSE_UV_CERT:-}" ] && curl -s --head --fail "$GOOSE_UV_CERT" > /dev/null; then
	log "Downloading certificate from: $GOOSE_UV_CERT"
	curl -sSL -o ~/.config/goose/mcp-hermit/cert.pem "$GOOSE_UV_CERT"
	if [ $? -eq 0 ]; then
	log "Certificate downloaded successfully."
	export SSL_CERT_FILE=~/.config/goose/mcp-hermit/cert.pem
	else
	log "Unable to download the certificate. Skipping certificate setup."
	fi
	else
	log "GOOSE_UV_CERT is either not set or not accessible. Skipping certificate setup."
	fi
	else
	log "GOOSE_UV_REGISTRY is either not set or not accessible. Falling back to default pip registry."
	export UV_INDEX_URL="https://pypi.org/simple"
	fi

SSL_CERT_FILE env var gets picked up by httpx, which then causes the SSL cert error:
encode/httpx#3510 (comment)

[salman1993]

replicating the error - the following script works when use_cert=False but fails with use_cert=True

import os
import httpx
import ssl

use_cert = True

URL = "https://block.github.io/goose/docs/tutorials/custom-extensions/"
ROBOT_URL = "https://block.github.io/goose/robots.txt"

CA_FILE = "/Users/smohammed/.config/goose/mcp-hermit/cert.pem"

if use_cert:
    # by default, httpx picks up SSL_CERT_FILE & runs this code block
    # https://github.com/encode/httpx/blob/e70d0b08c929e5140e38aa92a419de0c6d494148/httpx/_config.py#L34
    ctx = ssl.create_default_context(cafile=CA_FILE)
    client = httpx.Client(verify=ctx)
else:
    client = httpx.Client()

# https://github.com/modelcontextprotocol/servers/blob/d12118c8a88374d27c4d5130ad0783c3ffba64da/src/fetch/src/mcp_server_fetch/server.py#L121
try:
    response = client.get(URL, follow_redirects=True, timeout=10)
    print("Status: ", response.status_code)
except httpx.HTTPStatusError as e:
    print(f"HTTP error: {e}")
    raise
except httpx.TransportError as e:
    print(f"Transport error: {e}")
    raise

page_raw = response.text
print()
print(page_raw[:2_000] + " ...")

[wendytang]

#1376

[wendytang]

follow internal discussion here

[salman1993]

closing this for now since #1376 seems to have fixed this issue

[wendytang]

Resolved - use https://github.com/squareup/goose-releases/releases/tag/v1.0.9-block.202502251731-11177 or upcoming release 1.10.0