Failing unit tests (WOF dist shutdown)

[Joxit]

Describe the bug
Tests are failing due to WOF dist shutdown.

Steps to Reproduce

1. npm install
2. npm test

Expected behavior
Tests to work, should we mock the http call ? geocode.earth is supporting only inventory.json and not old .txt files 🤔

Environment (please complete the following information):

• OS: Debian ❤️
• node: 10

Pastebin/Screenshots

  ---
    operator: ok
    expected: true
    actual:   false
    at: expected.every (/code/whosonfirst/test/bundleList.js:73:11)
    stack: |-
           Error: ocean bundle(s) missing
             at Test.assert [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:228:54)
             at Test.bound [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at Test.assert (/code/whosonfirst/node_modules/tape/lib/test.js:347:10)
             at Test.bound [as assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at expected.every (/code/whosonfirst/test/bundleList.js:73:11)
             at Array.every (<anonymous>)
             at bundles.generateBundleList (/code/whosonfirst/test/bundleList.js:68:16)
             at Interface.rl.on.on (/code/whosonfirst/src/bundleList.js:120:5)
             at Interface.emit (events.js:203:15)
             at Interface.close (readline.js:397:8)
             
  ...
  ---
    operator: ok
    expected: true
    actual:   false
    at: bundles.generateBundleList (/code/whosonfirst/test/bundleList.js:100:9)
    stack: |-
           Error: venue bundle for regions are included
             at Test.assert [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:228:54)
             at Test.bound [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at Test.assert (/code/whosonfirst/node_modules/tape/lib/test.js:347:10)
             at Test.bound [as assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at bundles.generateBundleList (/code/whosonfirst/test/bundleList.js:100:9)
             at Interface.rl.on.on (/code/whosonfirst/src/bundleList.js:120:5)
             at Interface.emit (events.js:203:15)
             at Interface.close (readline.js:397:8)
             at ReadStream.onend (readline.js:173:10)
             at ReadStream.emit (events.js:203:15)
             
  ...
  ---
    operator: ok
    expected: true
    actual:   false
    at: expected.every (/code/whosonfirst/test/bundleList.js:131:11)
    stack: |-
           Error: ocean bundle(s) missing
             at Test.assert [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:228:54)
             at Test.bound [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at Test.assert (/code/whosonfirst/node_modules/tape/lib/test.js:347:10)
             at Test.bound [as assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at expected.every (/code/whosonfirst/test/bundleList.js:131:11)
             at Array.every (<anonymous>)
             at bundles.generateBundleList (/code/whosonfirst/test/bundleList.js:127:16)
             at Interface.rl.on.on (/code/whosonfirst/src/bundleList.js:120:5)
             at Interface.emit (events.js:203:15)
             at Interface.close (readline.js:397:8)
             
  ...
  ---
    operator: ok
    expected: true
    actual:   false
    at: expected.every (/code/whosonfirst/test/bundleList.js:176:11)
    stack: |-
           Error: ocean bundle(s) missing
             at Test.assert [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:228:54)
             at Test.bound [as _assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at Test.assert (/code/whosonfirst/node_modules/tape/lib/test.js:347:10)
             at Test.bound [as assert] (/code/whosonfirst/node_modules/tape/lib/test.js:80:32)
             at expected.every (/code/whosonfirst/test/bundleList.js:176:11)
             at Array.every (<anonymous>)
             at bundles.generateBundleList (/code/whosonfirst/test/bundleList.js:172:16)
             at Interface.rl.on.on (/code/whosonfirst/src/bundleList.js:120:5)
             at Interface.emit (events.js:203:15)
             at Interface.close (readline.js:397:8)
  ...

  

  161 tests
  221 passed
  4 failed  

  Failed Tests:   There were 4 failures

    x ocean bundle(s) missing
    x venue bundle for regions are included
    x ocean bundle(s) missing
    x ocean bundle(s) missing
  
npm ERR! Test failed.  See above for more details.

References

whosonfirst/src/bundleList.js

Lines 63 to 65 in fa17e35

	const wofDataHost = peliasConfig.get('imports.whosonfirst.dataHost') || 'https://dist.whosonfirst.org';
	const bundleIndexFile = path.join(metaDataPath, 'whosonfirst_bundle_index.txt');
	const bundleIndexUrl = `${wofDataHost}/bundles/index.txt`;

[orangejulius]

Yeah, it's a bit unfortunate that our unit tests now fail. (But credit to our old Mapzen teammate @trescube for predicting the need for this years ago in #226 :) )

Anyways, @missinglink and I have been discussing the changes with WOF and we've decided that since things are already broken we might as well make all the breaking changes we've wanted.

So, lets simply drop support for the bundles completely and remove all the bundle code. Going forward we'll probably remove them from the Geocode Earth downloads, that way both the data and importers can go 100% in on SQLite. This would imply finishing off #460 as well.

I will try to work on some of this in the next few days, but if anyone else wants to work on it as well, feel free! Let me know if I can help with any pointers or suggestions. @pixeldublu you've been doing a lot in your recent PR so I'll tag you here as well, just in case.

[Joxit]

Ha ha ha well done @trescube 🤣

Nice, you made my day better with your discussion ❤️ I can drop support for bundle in #487 this will simplify my work 🙂