Scrape the AI legislation PDFs

[mkalish]

Description

Implement a scraping tool to iterate over the AI legislation table and download the most recent version of each legislation

AirTable: https://airtable.com/appJHkVlGCmNLPLXL/shrFvhCRSOE9dbbSz/tbl2tL08FmzPrXgT2

Acceptance Criteria

• A re-usable tool can be executed to scrape the AI legislation
• If the legislation is enacted, scrape that, otherwise scrape the most recent one
• Each scraped legislation should save in the following format {STATE}_{BILL-TITLE}_{BILL-STATUS}_{BILL-DATE}.html
  • Example: ALABAMA_HB168_ENACTED_2024_04_18.html
• Each scraped legislation should also include a relevant metadata file i.e ALABAMA_HB168_ENACTED_2024_04_18.md that should include
  • Bill title
  • State
  • Bill status
  • Legislative session
• All the scraped bills should go in a directory with that days date. Expected structure:
  • data/raw/20241231/ALABAMA_HB168_ENACTED_2024_04_18.html

Developer notes

• Might require creating an account with AirTable
• The bills are exposed as HTML and should be saved that way so we can keep the raw data and later tickets will parse it
• Most of the metadata can be found in the airtable