RFC: Skip proxies for requests allowed by robots.txt

[arunkannawadi]

Currently, once a proxy is set, scholarly uses it to get all pages, even those that are allowed by Google Scholar's robots.txt. This leads to Proxy API charging the user for such requests when no proxy would have worked just fine. Does it make sense for scholarly to skip using the proxies that are set in retrieving information from citations? pages and use proxies (if set) for scholar? pages only? Should we give an option for the users to stay behind a proxy even for such users? Looking for opinions, comments before I can work on this feature.

[ipeirotis]

I would recommend creating a tutorial instead, explaining that some scholarly requests are complying with robots.txt while others do not.

The ones that do not comply with robots.txt, really need a proxy.

That part will also explain to newcomers that just because the search_author works, it does not mean that the search_pubs will work.

[arunkannawadi]

Tutorial sounds good in general, but then should we leave it to the individual applications to turn on/off proxy depending on the type of query? I'm thinking of applications that build citation networks which need to query both authors and publications. It'd be cleaner if this was handled within the library instead in the application. Turning it off and on in the application has a small penalty in that, scholarly will spend a query each time proxy is turned on to check if it works.

[ipeirotis]

I believe that a proxy may be needed even for the vanilla queries, especially when people overload the scholar service. So, I would be hesitant to put code in scholarly that switches off the proxy automatically.

[arunkannawadi]

Agreed. So I will leave the default option as to use proxy for all queries and the user will have the option of explicitly overriding the use of proxy for queries that are not actively blocked. I will get a PR out next week or so, with updated docs on how to do the override, and we can discuss some design choices there.

On that note, it is scary that all uses of ScraperAPI have been silently routed through the user's real IP addresses so far without any severe warning to the user. Changes I'm making for v1.4.1 should fix them.

[arunkannawadi]

After some thought, I think I'd like to design the Navigator object to have two proxy managers instead of a single pm. Tne expensive one (ScraperAPI or Luminati) will be used for search publications (disallowed in robots.txt) and the other inexpensive/free one to search authors (allowed by robots.txt). The inexpensive one can just be freeproxy, which is good to fetch results from citations? database while still protecting the user's IP address. They can also be set to be the same, which would mimic the current behavior.

The use case that I'm really trying to optimize here is that of having a free ScraperAPI account that allows only 1000 successful requests a month, that I'd like to use only sparingly when required.

[ipeirotis]

I am ready to drop both Tor and FreeProxies. They do not really work in most cases anymore.

[arunkannawadi]

FreeProxy works just fine for scraping author profiles, especially if you handle the exceptions appropriately within scholarly. I can't comment on Tor since I haven't used it, but it might be an issue with scholarly handling Tor or a misconfiguration rather than an issue with the service itself. I wouldn't be opposed to temporarily removing it from documentation but still have the code in place until it is clear where the issue is.