feature/workday_scraper #67
Conversation
Saw your comment on the #23 ticket. Is that an error inside docker? Is your docker instance not able to install chrome? Are you able to run it using python + venv? |
|
Just looked over the PR, I like it. Please include a test for the scraper. It can be something very simple and straightforward that tests if the scraper is properly retrieving info from the site. Thank you |
|
@nicobrenner Hey! I am able to run it locally through venv, and I also managed to build with docker just now but running it is causing some issues atm. I pushed a commit with a successful docker build |
There was a problem hiding this comment.
Thank you @noameron for this PR. Excellent work. One last request: please update the README.md file to include an update at the top. It should be 1 or 2 sentences about what is new. Extra brownie points if you also make a short video using it 🙏🏽🤗:
| @@ -10,6 +10,11 @@ WORKDIR /commandjobs | |||
| # Install any needed packages specified in requirements.txt | |||
| RUN pip3 install --no-cache-dir -r config/requirements.txt | |||
|
|
|||
| # Install required packages, including Chromium and ChromeDriver | |||
| @@ -14,7 +16,7 @@ | |||
| class WorkdayScraper: | |||
| def __init__(self, db_path='job_listings.db', update_func=None, done_event=None, result_queue=None): | |||
| self.db_path = db_path | |||
| self.driver = webdriver.Chrome(options=self.get_selenium_configs()) | |||
| self.driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=self.get_selenium_configs()) | |||
| "🕸 Scrape \"Work at a Startup jobs\"", | ||
| "🕸 Scrape \"Workday\"", |
|
@nicobrenner Hey! I just pushed a unit test for verifying the base elements of all job listings, I think next PR I should add a workflow for triggering the unit tests on each PR push. Other than that everything seems to work on my end when i checked |
|
Where should I update on the README regarding the new scarper for workday? |
Please add an entry at the top of the Optional: it would be great to have a short screencast of you demoing the new changes within Thank you |
Excellent. It's looking good. With the update to the README, the PR should be ready to merge 👍🏼 |
|
Just pushed the readme update! @nicobrenner |
Added support for generic Workday scraper, and updated requirements.
I am facing issues trying to build docker, due to chrome / chromeDriver.