4.1.4

• always add links to pool if robots shouldn't be respected

4.1.3

• refactor of internals

4.1.2

• make it possible to override $defaultClientOptions

4.1.1

• Bump minimum required version of spatie/robots-txt to 1.0.1.

4.1.0

Respect robots.txt

4.0.5

• improved extensibility by removing php native type hinting of url, queue and crawler pool Closures

4.0.4 - do not follow links that have attribute `rel` set to `nofollow`

do not follow links that have attribute  set to

4.0.3

• Support both Illuminate's and Tighten's Collection.

4.0.2

• fix bugs when installing into a Laravel app

4.0.1

• fix bugs when installed in Laravel projects