From 0636451c0aea64accccecaaefaa38d9792f44350 Mon Sep 17 00:00:00 2001 From: Arthur Taylor Date: Sat, 16 Dec 2023 14:00:35 +0100 Subject: [PATCH] Update README.md --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 55af3ffe..9358c234 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ With ```Flathunter```, instead of visiting the same pages on the same sites eve ## Prerequisites * [Python 3.10+](https://www.python.org/) * [pipenv](https://pipenv.pypa.io/en/latest/) -* [Chromium](https://www.chromium.org/) / [Google Chrome](https://www.google.com/chrome/) (*optional to scan ads on immobilienscout24.de*) +* [Chromium](https://www.chromium.org/) / [Google Chrome](https://www.google.com/chrome/) (*optional to scan ads on immobilienscout24.de and Kleinanzeigen*) * [Docker]() (*optional*) * [GCloud CLI]() (*optional*) @@ -164,6 +164,10 @@ $ curl https://api.telegram.org/bot[BOT-TOKEN]/getUpdates to get list of messages the Bot has received. You will see your Chat ID in there. +#### Bot Detection + +Some sites (including Kleinanzeigen and ImmoScout24) implement bot detection to prevent scripts from scraping their sites. Flathunter includes support for running a headless Chrome browser to simulate human requests to the websites. **For crawling Kleinanzeigen and ImmoScout24, you will need to install Google Chrome** + #### Captchas Some sites (including ImmoScout24) implement a Captcha to avoid being crawled by evil web scrapers. Since our crawler is not an evil one, the people at [2Captcha](https://2captcha.com) and [Imagetyperz](https://imagetyperz.com/) provide services that help you solve them. You can head over to one of those services and buy some credit for captcha solving. You will need to install the API key for your captcha-solving account in the `config.yaml`. Check out `config.yaml.dist` to see how to configure `2Captcha` or `Imagetyperz` with Flathunter. **At this time, ImmoScout24 can not be crawled by Flathunter without using 2Captcha/Imagetyperz. Buying captcha solutions does not guarantee that you will get past the ImmoScout24 bot detection (see [#296](https://github.com/flathunters/flathunter/issues/296), [#302](https://github.com/flathunters/flathunter/issues/302))**.