Vendorise pdfs for integration test (#75)

This removes an external dependency from integration tests that was
causing a failure and blocking deploys. Previously, tests ran against
live pdf files on various government websites. So when those websites
changed the tests no longer worked.

This was blocking other work, so in order to unblock efficiently, I've
simply switched the tests to point at our own staging cdn. This is still
not ideal as now the tests depend on our cdn. But it is an improvement
and means we can progress other work

integration_tests/data/pipeline_out/ingest_unit_test_parser_input/TESTCCLW.executive.1332.1547.json

@@ -2,10 +2,10 @@
   "document_id": "TESTCCLW.executive.1332.1547",
   "document_name": "Presidential of the Republic of Indonesia Instruction Number 6 Year 2013 on Suspension of New Licenses and Improving Forest Governance of Primary Forest and Peatland",
   "document_description": "The first iteration of this instruction was issued in 2011 in order to implement commitments under the agreements in the Letter of Intent signed with the Kingdom of Norway in May 2011. The Instruction is intended to facilitate Indonesia's participation in internationally financed REDD activities and places a moratorium on clearance of primary peatland and forests within moratorium areas.\u00a0The initial moratorium was extended by Presidential Instruction 6/2013.\u00a0In 2019, President Joko Widodo signed Presidential Instruction 5/2019, making the moratorium on the clearance of primary forest and peatlands in moratorium areas permanent.\u00a0",
-  "document_source_url": "https://www.africau.edu/images/default/sample.pdf",
-  "document_cdn_object": "IDN/2013/presidential-of-the-republic-of-indonesia-instruction-number-6-year-2013-on-suspension-of-new-licenses-and-improving-forest-governance-of-primary-forest-and-peatland_7bb2a5b4c5f87553cf0e310db41255b0.pdf",
+  "document_source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_cdn_object": "IDN/2013/presidential-of-the-republic-of-indonesia-instruction-number-6-year-2013-on-suspension-of-new-licenses-and-improving-forest-governance-of-primary-forest-and-peatland_3c8998aa37b4897469334915dfcde33e.pdf",
   "document_content_type": "application/pdf",
-  "document_md5_sum": "7bb2a5b4c5f87553cf0e310db41255b0",
+  "document_md5_sum": "3c8998aa37b4897469334915dfcde33e",
   "document_slug": "indonesia_2013_presidential-instruction-6-year-2013-on-suspension-of-new-licenses-and-improving-forest-governance-of-primary-forest-and-peatland",
   "document_metadata": {
     "name": "Presidential of the Republic of Indonesia Instruction Number 6 Year 2013 on Suspension of New Licenses and Improving Forest Governance of Primary Forest and Peatland",
@@ -16,8 +16,8 @@
     "family_slug": "slug_TESTCCLW.family.1332.0",
     "publication_ts": "2013-01-01T00:00:00",
     "date": "01/01/2013",
-    "source_url": "https://www.africau.edu/images/default/sample.pdf",
-    "download_url": "http://ronlaw.gov.nr/nauru_lpms/files/subordinate_legislation/aab912643572c84083c256fa0857809b.pdf",
+    "source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+    "download_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
     "type": "Decree",
     "source": "CCLW",
     "category": "Policy",
@@ -47,4 +47,4 @@
     }
   },
   "pipeline_metadata": {}
-}
+}

integration_tests/data/pipeline_out/ingest_unit_test_parser_input/TESTCCLW.executive.1332.1548.json

@@ -2,10 +2,10 @@
   "document_id": "TESTCCLW.executive.1332.1548",
   "document_name": "Presidential of the Republic of Indonesia Instruction Number 6 Year 2013 on Suspension of New Licenses and Improving Forest Governance of Primary Forest and Peatland",
   "document_description": "The first iteration of this instruction was issued in 2011 in order to implement commitments under the agreements in the Letter of Intent signed with the Kingdom of Norway in May 2011. The Instruction is intended to facilitate Indonesia's participation in internationally financed REDD activities and places a moratorium on clearance of primary peatland and forests within moratorium areas.\u00a0The initial moratorium was extended by Presidential Instruction 6/2013.\u00a0In 2019, President Joko Widodo signed Presidential Instruction 5/2019, making the moratorium on the clearance of primary forest and peatlands in moratorium areas permanent.\u00a0",
-  "document_source_url": "https://www.africau.edu/images/default/sample.pdf",
-  "document_cdn_object": "IDN/2013/presidential-of-the-republic-of-indonesia-instruction-number-6-year-2013-on-suspension-of-new-licenses-and-improving-forest-governance-of-primary-forest-and-peatland_4b41a3475132bd861b30a878e30aa56a.pdf",
+  "document_source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_cdn_object": "IDN/2013/presidential-of-the-republic-of-indonesia-instruction-number-6-year-2013-on-suspension-of-new-licenses-and-improving-forest-governance-of-primary-forest-and-peatland_3c8998aa37b4897469334915dfcde33e.pdf",
   "document_content_type": "application/pdf",
-  "document_md5_sum": "4b41a3475132bd861b30a878e30aa56a",
+  "document_md5_sum": "3c8998aa37b4897469334915dfcde33e",
   "document_slug": "england_2013_presidential-of-the-republic-of-indonesia-instruction-number-6-year-2013-on-suspension-of-new-licenses-and-improving-forest-governance-of-primary-forest-and-peatland",
   "document_metadata": {
     "name": "Presidential of the Republic of Indonesia Instruction Number 6 Year 2013 on Suspension of New Licenses and Improving Forest Governance of Primary Forest and Peatland",
@@ -16,7 +16,7 @@
     "family_slug": "slug_TESTCCLW.family.1332.0",
     "publication_ts": "2013-01-01T00:00:00",
     "date": "01/01/2013",
-    "source_url": "https://www.africau.edu/images/default/sample.pdf",
+    "source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
     "download_url": null,
     "type": "Decree",
     "source": "CCLW",
@@ -47,4 +47,4 @@
     }
   },
   "pipeline_metadata": {}
-}
+}

integration_tests/data/pipeline_out/ingest_unit_test_parser_input/TESTCCLW.executive.1332.1549.json

@@ -2,10 +2,10 @@
   "document_id": "TESTCCLW.executive.1332.1549",
   "document_name": "Act on Promotion of Global Warming Countermeasures",
   "document_description": "This Law is one of the two key climate laws in Japan along with the Energy Conservation Law. The purpose of the Law is to reduce emissions of GHGs derived from anthropogenic activities. GHGs are carbon dioxide, methane, nitrous oxide, HFC, PFC and sulphur hexafluoride. The Council of Ministers for Global Environmental Conservation is established under the Law. The Council is chaired by the Prime Minister, and vice-chairmen are the Chief Cabinet Secretary, Minister of the Environment and Minister of Economy, Trade and Industry. Other members consist of all ministers other than vice-chairmen.\u00a0\u00a0Designated emitters, whose workplaces contain more than 1,500kL of oil equivalent of energy annually, are mandated to develop the Plan for Global Warming Countermeasure. While there is no reduction obligation under this law, annual emission of GHGs are reported to the Minister in charge. Emission reporting under this framework equals that of the reporting under the Energy Conservation Law.\u00a0\u00a0This Law stipulates that the State is responsible for implementing necessary measures to introduce Emission Trading Scheme (ETS) in Japan. It adds that examination and discussion of the design and the utilization of ETS starts upon the enactment of this Law.\u00a0\u00a0This Law also provides that the national and local governments are responsible for development and implementation of plans to reduce GHG emissions.\u00a0\u00a0The National Government adopted the Plan for Global Warming Countermeasures in May 2016, with the explicit aim to achieve the mid-term target set in Japan's INDC (2015) 26% GHG emissions reduction by 2030 (baseline 2013). In addition, the Plan also sets a long term goal of 80% GHG emissions reduction by 2050.\u00a0 Prefectural and municipal governments are also required to create local plans to reduce GHG emissions. The plans should include:\u00a0Duration of the planGoalsMeasures and actions intended for implementationPromotion of solar PV, wind and other renewable energiesMeasures and actions taken by business professionals and citizens to reduce GHG emissionPromotion of public transport use, conservation of green space and other GHG emission reduction measuresOn June 4th, 2021, the Diet approved the amending Act 54/2021 introducing a net zero target by 2050 into the law.",
-  "document_source_url": "https://www.africau.edu/images/default/sample.pdf",
-  "document_cdn_object": "JPN/1998/act-on-promotion-of-global-warming-countermeasures_4b41a3475132bd861b30a878e30aa56a.pdf",
+  "document_source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_cdn_object": "JPN/1998/act-on-promotion-of-global-warming-countermeasures_3c8998aa37b4897469334915dfcde33e.pdf",
   "document_content_type": "application/pdf",
-  "document_md5_sum": "4b41a3475132bd861b30a878e30aa56a",
+  "document_md5_sum": "3c8998aa37b4897469334915dfcde33e",
   "document_slug": "japan_1998_act-on-promotion-of-global-warming-countermeasures",
   "document_metadata": {
     "name": "Act on Promotion of Global Warming Countermeasures",
@@ -16,7 +16,7 @@
     "family_slug": "slug_TESTCCLW.family.1332.0",
     "publication_ts": "1998-01-01T00:00:00",
     "date": "01/01/1998",
-    "source_url": "https://www.africau.edu/images/default/sample.pdf",
+    "source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
     "download_url": null,
     "type": "Act",
     "source": "CCLW",
@@ -54,4 +54,4 @@
     }
   },
   "pipeline_metadata": {}
-}
+}

integration_tests/data/pipeline_out/ingest_unit_test_parser_input/TESTCCLW.executive.1332.1551.json

@@ -2,10 +2,10 @@
   "document_id": "TESTCCLW.executive.1332.1551",
   "document_name": "Revised Global Warming Countermeasures Promotion Law",
   "document_description": "This Law is one of the two key climate laws in Japan along with the Energy Conservation Law. The purpose of the Law is to reduce emissions of GHGs derived from anthropogenic activities. GHGs are carbon dioxide, methane, nitrous oxide, HFC, PFC and sulphur hexafluoride. The Council of Ministers for Global Environmental Conservation is established under the Law. The Council is chaired by the Prime Minister, and vice-chairmen are the Chief Cabinet Secretary, Minister of the Environment and Minister of Economy, Trade and Industry. Other members consist of all ministers other than vice-chairmen.\u00a0\u00a0Designated emitters, whose workplaces contain more than 1,500kL of oil equivalent of energy annually, are mandated to develop the Plan for Global Warming Countermeasure. While there is no reduction obligation under this law, annual emission of GHGs are reported to the Minister in charge. Emission reporting under this framework equals that of the reporting under the Energy Conservation Law.\u00a0\u00a0This Law stipulates that the State is responsible for implementing necessary measures to introduce Emission Trading Scheme (ETS) in Japan. It adds that examination and discussion of the design and the utilization of ETS starts upon the enactment of this Law.\u00a0\u00a0This Law also provides that the national and local governments are responsible for development and implementation of plans to reduce GHG emissions.\u00a0\u00a0The National Government adopted the Plan for Global Warming Countermeasures in May 2016, with the explicit aim to achieve the mid-term target set in Japan's INDC (2015) 26% GHG emissions reduction by 2030 (baseline 2013). In addition, the Plan also sets a long term goal of 80% GHG emissions reduction by 2050.\u00a0 Prefectural and municipal governments are also required to create local plans to reduce GHG emissions. The plans should include:\u00a0Duration of the planGoalsMeasures and actions intended for implementationPromotion of solar PV, wind and other renewable energiesMeasures and actions taken by business professionals and citizens to reduce GHG emissionPromotion of public transport use, conservation of green space and other GHG emission reduction measuresOn June 4th, 2021, the Diet approved the amending Act 54/2021 introducing a net zero target by 2050 into the law.",
-  "document_source_url": "https://www.env.go.jp/press/ontaihou/116348.pdf",
-  "document_cdn_object": "JPN/2021/revised-global-warming-countermeasures-promotion-law_5401db7b173822e3ba86e5230ae9fd90.pdf",
+  "document_source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_cdn_object": "JPN/2021/revised-global-warming-countermeasures-promotion-law_3c8998aa37b4897469334915dfcde33e.pdf",
   "document_content_type": "application/pdf",
-  "document_md5_sum": "5401db7b173822e3ba86e5230ae9fd90",
+  "document_md5_sum": "3c8998aa37b4897469334915dfcde33e",
   "document_slug": "japan_2021_revised-global-warming-countermeasures-promotion-law",
   "document_metadata": {
     "name": "Revised Global Warming Countermeasures Promotion Law",
@@ -16,7 +16,7 @@
     "family_slug": "slug_TESTCCLW.family.1332.0",
     "publication_ts": "2021-01-01T00:00:00",
     "date": "01/01/2021",
-    "source_url": "https://www.env.go.jp/press/ontaihou/116348.pdf",
+    "source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
     "download_url": null,
     "type": "Law",
     "source": "CCLW",
@@ -54,4 +54,4 @@
     }
   },
   "pipeline_metadata": {}
-}
+}

integration_tests/data/pipeline_out/ingest_unit_test_parser_input/TESTCCLW.executive.1332.1552.json

@@ -2,10 +2,10 @@
   "document_id": "TESTCCLW.executive.1332.1552",
   "document_name": "Basic Hydrogen Strategy",
   "document_description": "This document sets Japan's vision on how to achieve a hydrogen-based society by 2050 and provides an action plan for its realisation. It specifically seeks to decarbonise the energy, industry and transportation sectors.The Strategic Roadmap for Hydrogen and Fuel Cells defines 1) new targets on the specification of basic technologies and the breakdown of costs, 2) necessary measures for achieving these goals; and 3) that Japan will convene a working group consisting of experts to review the status of implementation in each area stipulated by the roadmap.",
-  "document_source_url": "https://www.meti.go.jp/english/press/2017/pdf/1226_003b.pdf",
-  "document_cdn_object": null,
-  "document_content_type": null,
-  "document_md5_sum": null,
+  "document_source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_cdn_object": "JPN/2017/basic-hydrogen-strategy_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_content_type": "application/pdf",
+  "document_md5_sum": "3c8998aa37b4897469334915dfcde33e",
   "document_slug": "japan_2017_basic-hydrogen-strategy",
   "document_metadata": {
     "name": "Basic Hydrogen Strategy",
@@ -16,7 +16,7 @@
     "family_slug": "slug_TESTCCLW.family.1332.0",
     "publication_ts": "2017-01-01T00:00:00",
     "date": "01/01/2017",
-    "source_url": "https://www.meti.go.jp/english/press/2017/pdf/1226_003b.pdf",
+    "source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
     "download_url": null,
     "type": "Strategy",
     "source": "CCLW",
@@ -45,4 +45,4 @@
     }
   },
   "pipeline_metadata": {}
-}
+}

integration_tests/data/pipeline_out/ingest_unit_test_parser_input/TESTCCLW.executive.1332.1553.json

@@ -2,10 +2,10 @@
   "document_id": "TESTCCLW.executive.1332.1553",
   "document_name": "Spanish Climate Change And Clean Energy Strategy Horizon 2007- 2012 -2020",
   "document_description": "The Spanish Climate Change and Clean Energy Strategy (EECCEL) horizon 2007-2012-2020 is part of the Spanish Sustainable Development Strategy (EEDS). The EECCEL includes different measures that contribute to sustainable development within the scope of climate change and clean energy.\u00a0\u00a0This Strategy is based on the reference framework of the 'Spanish Strategy for the fulfilment of the objectives under the Kyoto Protocol', and it takes into account the measures and Programmes adopted by the Autonomous Communities.\u00a0\u00a0The strategy has two chapters. The first one defines actions to fight against climate change and the second one, actions to achieve cleaner energy. Each chapter includes a description of the present situation, the objectives to be reached, the suggested measures and a selection of indicators for the corresponding follow-up.\u00a0\u00a0The operational objectives are:\u00a0- To ensure the reduction of GHG emissions in Spain, giving special importance to measures related to the energy sector. According to the national inventory, in 2005, emissions from energy process represented about 78.87% of total national emissions.\u00a0- To contribute to sustainable development and the fulfilment of climate change commitments by strengthening the use of flexible project-based mechanisms.\u00a0- To promote additional reduction measures in sectors concerned with diffuse pollution.\u00a0- To apply the National Climate Change Adaptation Plan (NCCAP) so as to integrate adaptation measures and strategies in sectoral policies.\u00a0- To increase public awareness with respect to clean energy and climate change.\u00a0- To promote research, development and innovation in matters of climate change and clean energy.\u00a0- To guarantee energy supply security by means of cleaner energies, mainly from renewable sources, achieving other environmental benefits (for example, air quality) and limiting the growth rate of external energy dependence.\u00a0- To boost energy- and resource efficiency for companies and for end users.\u00a0\u00a0The government has adopted a Plan of Urgent Measures (PMU), which together with the 2008-2012 Energy Saving and Efficiency Action Plan aims to consolidate the trend change of GHG emissions in Spain initiated in 2006.",
-  "document_source_url": "https://www.lse.ac.uk/GranthamInstitute/wp-content/uploads/laws/1674%20English.pdf",
-  "document_cdn_object": null,
-  "document_content_type": "text/html",
-  "document_md5_sum": null,
+  "document_source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_cdn_object": "ESP/2007/spanish-climate-change-and-clean-energy-strategy-horizon-2007-2012-2020_3c8998aa37b4897469334915dfcde33e.pdf",
+  "document_content_type": "application/pdf",
+  "document_md5_sum": "3c8998aa37b4897469334915dfcde33e",
   "document_slug": "spain_2007_spanish-climate-change-and-clean-energy-strategy-horizon-2007-2012-2020",
   "document_metadata": {
     "name": "Spanish Climate Change And Clean Energy Strategy Horizon 2007- 2012 -2020",
@@ -16,7 +16,7 @@
     "family_slug": "slug_TESTCCLW.family.1332.0",
     "publication_ts": "2007-01-01T00:00:00",
     "date": "01/01/2007",
-    "source_url": "https://www.lse.ac.uk/GranthamInstitute/wp-content/uploads/laws/1674%20English.pdf",
+    "source_url": "https://cdn.dev.climatepolicyradar.org/navigator/IND/2021/national-mission-on-sustainable-habitat_3c8998aa37b4897469334915dfcde33e.pdf",
     "download_url": null,
     "type": "Strategy",
     "source": "CCLW",
@@ -53,4 +53,4 @@
     }
   },
   "pipeline_metadata": {}
-}
+}