Add support for sending User-Agent string to prerenderer and fix breaking headers issue

[jdotjdot]

Makes changes to argument signature of the BackendBase method get_response_for_url()

I've updated the README accordingly as well. I have not been able to get the tests running correctly on my computer to check them—all of the tests are failing for me, even the tests that are completely unrelated to any changes.

I reviewed pretty thoroughly though and don't think there should be any issues; I changed very little.

I wanted to do this because I use Prerender.io but all of the requests look like they're coming from python's requests module, and I'd love for Prerender.io to show the actual user agent so I can get better analytics on where crawler requests are coming from.

[skoczen]

+1 to this, thanks for submitting it! I'll pull it into the next release. I'm hoping to have a good couple of days at the end of the year to really get caught up on all my open-source projects.

Thanks for your patience!

[jdotjdot]

Any update on this? I'd love to get this merged so I can start using it!

[jdotjdot]

I added a fix for #23, but this PR has been sitting around long enough that the fix ended up just getting attached to this PR as well.

[skoczen]

Update - things are moving forward without me being in the way. Please see #32 for our next steps in this project, and thank you so much for your contributions, and for your patience. :)

[pcraciunoiu]

@jdotjdot if you're still up for it, and want to update this PR, I can help get it merged

[jdotjdot]

@jdotjdot if you're still up for it, and want to update this PR, I can help get it merged

Sadly, we moved off of django-seo-js and Prerender.io a long time ago, so I'm definitely not up to date enough with anything going on here in order to take this PR over the finish line. Anyone else is welcome to take my code and get it updated to merge.

[zekzekus]

@jdotjdot actually i was just forking the repository to make changes to add ability to send original user agent to seobackend (prerender.io specifically). Fortunately i found this early on. It seems the conflict is just on AUTHORS file and if any improvement suggested from you guys i would be happy to work on this. At work we are relying on this library heavily and we -as whole team- would be happy to contribute.

[zekzekus]

I created a new PR (#34) to work on this. At work we are also using some features from prerender.io which i would be happy to contribute to main repo is possible. My first priority is this topic at them moment.

[zekzekus]

I mentioned wrong person :) @pcraciunoiu please check my comments and PR

[pcraciunoiu]

Thanks @zekzekus , looks good!

[pcraciunoiu]

Moved to #34.