Add user-agent to openlibrary API calls

Why are these changes being introduced: * OpenLibrary API calls were failing as they were blocking traffic without User Agents being set Relevant ticket(s): * https://mitlibraries.atlassian.net/browse/TCO-109 How does this address that need: * Follow OpenLibrary documentation to add a User Agent for our requests, including an email address they can contact if issues arise with our calls Document any side effects to this change: * Created a new moira list for tacos maintainers (tacos-help) and added engx-lib as the owners * Updated Unpaywall code to use the same email (from ENV) as OpenLibrary to avoid setting the same value to two different variables * Added some VCR cassette scrubbing to keep some "not sensitive but probably best to not store in cassettes" data * Confirmed deleting cassettes will regenerate and scrub the data, but not require changing ENV in between runs as we see in some other apps (I'm not sure why to be honest).
MITLibraries · Jan 13, 2025 · f8335b3 · f8335b3
1 parent 161862b
commit f8335b3
Show file tree

Hide file tree

Showing 15 changed files with 105 additions and 81 deletions.
diff --git a/.env.test b/.env.test
@@ -1,3 +1,3 @@
 DETECTOR_VERSION=1
 LINKRESOLVER_BASEURL=https://mit.primo.exlibrisgroup.com/discovery/openurl?institution=01MIT_INST&rfr_id=info:sid/mit.tacos.api&vid=01MIT_INST:MIT
-UNPAYWALL_EMAIL=timdex@mit.edu
+TACOS_EMAIL=tacos-help@mit.edu
diff --git a/README.md b/README.md
@@ -65,7 +65,7 @@ changes, this is the signal which indicates that terms need to be re-evaluated.
 
 `ORIGINS`: comma-separated list of domains allowed to connect to (and thus query or contribute to) the application. Be sure to specify the port number if a connecting application is not using the standard ports (this applies mostly to local development). If not defined, no external connections will be permitted.
 
-`UNPAYWALL_EMAIL`: email address to include in API call as required in their [documentation](https://unpaywall.org/products/api). Your personal email is appropriate for development. Deployed and for tests, use the timdex moira list email.
+`TACOS_EMAIL`: email address to include in API calls or contact information. Currently used in API calls to [Unpaywall](https://unpaywall.org/products/api) and [OpenLibrary](https://openlibrary.org/developers/api). Your personal email is appropriate for development. Deployed and for tests, use the tacos-help moira list email.
 
 ### Optional
 

diff --git a/app.json b/app.json
@@ -11,7 +11,7 @@
     "LINKRESOLVER_BASEURL": {
       "required": false
     },
-    "UNPAYWALL_EMAIL": {
+    "TACOS_EMAIL": {
       "required": false
     }
   },

diff --git a/app/models/lookup_doi.rb b/app/models/lookup_doi.rb
@@ -30,7 +30,7 @@ def extract_metadata(external_data)
   end
 
   def url(doi)
-    "https://api.unpaywall.org/v2/#{doi}?email=#{ENV.fetch('UNPAYWALL_EMAIL')}"
+    "https://api.unpaywall.org/v2/#{doi}?email=#{ENV.fetch('TACOS_EMAIL')}"
   end
 
   def fetch(doi)

diff --git a/app/models/lookup_isbn.rb b/app/models/lookup_isbn.rb
@@ -36,7 +36,10 @@ def fetch_authors(isbn_json)
   end
 
   def parse_response(url)
-    resp = HTTP.headers(accept: 'application/json', 'Content-Type': 'application/json').follow.get(url)
+    email = ENV.fetch('TACOS_EMAIL')
+    resp = HTTP.headers(accept: 'application/json',
+                        'Content-Type': 'application/json',
+                        'User-Agent': "MITL TACOS (#{email})").follow.get(url)
 
     if resp.status == 200
       JSON.parse(resp.to_s)

diff --git a/test/test_helper.rb b/test/test_helper.rb
@@ -17,6 +17,28 @@
 VCR.configure do |config|
   config.cassette_library_dir = 'test/vcr_cassettes'
   config.hook_into :webmock
+
+  # Filter TACOS email. It's not sensitive, but keeping it out of code is still good practice to avoid spam
+  config.filter_sensitive_data('FAKE_TACOS_EMAIL') do
+    ENV.fetch('TACOS_EMAIL', nil).to_s
+  end
+
+  config.before_record do |interaction|
+    header = interaction.response&.headers&.[]('Report-To')
+    header&.each do |redacted_text|
+      interaction.filter!(redacted_text, '<REDACTED_REPORT_TO>')
+    end
+
+    header = interaction.response&.headers&.[]('Reporting-Endpoints')
+    header&.each do |redacted_text|
+      interaction.filter!(redacted_text, '<REDACTED_REPORTING_ENDPOINT>')
+    end
+
+    header = interaction.response&.headers&.[]('Nel')
+    header&.each do |redacted_text|
+      interaction.filter!(redacted_text, '<REDACTED_NEL>')
+    end
+  end
 end
 
 module ActionDispatch

diff --git a/test/vcr_cassettes/doi_10_1038/d41586-023-03497-2.yml b/test/vcr_cassettes/doi_10_1038/d41586-023-03497-2.yml
diff --git a/test/vcr_cassettes/doi_not_found.yml b/test/vcr_cassettes/doi_not_found.yml
diff --git a/test/vcr_cassettes/isbn_978-0-08-102133-0.yml b/test/vcr_cassettes/isbn_978-0-08-102133-0.yml
diff --git a/test/vcr_cassettes/isbn_not_found.yml b/test/vcr_cassettes/isbn_not_found.yml
diff --git a/test/vcr_cassettes/issn_1078-8956.yml b/test/vcr_cassettes/issn_1078-8956.yml
diff --git a/test/vcr_cassettes/issn_not_found.yml b/test/vcr_cassettes/issn_not_found.yml