Skip to content

Latest commit

 

History

History
39 lines (33 loc) · 1.09 KB

README.md

File metadata and controls

39 lines (33 loc) · 1.09 KB

Mark Scraper

CLI to scrape webpages and return them in markdown format

mark-scraper http://example.com
# Example Domain
# ==============

# This domain is for use in illustrative examples in documents. You may use this
# domain in literature without prior coordination or asking for permission.

# [More information...](https://www.iana.org/domains/example)

Installation

pipx install mark-scraper
# or
uv tool install mark-scraper

TODO

Here is a list of features I would like to add to this project

  • Also save html
    • mark-scraper -h/--html example.com.html http://example.com > example.com.md
  • Take a screen shot of web page
    • mark-scraper -s/--screenshot screenshot.png http://example.com > example.com.md
  • Add Page Caching
    • Will return from cache if exists mark-scraper -c/--cache http://example.com > example.com.md
    • Bust cache mark-scraper -b/--bust-cache http://example.com > example.com.md
    • Clear cache rm -rf .mark-scraper/cache
  • Add log of pages scrapped

Development

uv sync
uv run pytest
uv run mark-scraper http://example.com