Skip to content

Commit

Permalink
feat(parser): Filter out URLs before sending to pelias/model
Browse files Browse the repository at this point in the history
We have had numerous reports from Pelias users about concerning error
message during builds regarding the URL regex filter from
pelias/model#115.

While this filter is good, the resulting error message is alarming.
Looking today at the output of a planet build, it appears that many of
these errors come from the polylines file created by Valhalla out of the
OSM street network.

Looking at the contents of the polyline file and corresponding record on
OSM, it seems that Valhalla puts the contents of the `ref` tag in the
polyline file as an alternate name. The [ref tag](https://wiki.openstreetmap.org/wiki/Key:ref?uselang=en-US) will
often contain a URL.

This means that not only will the error happen frequently, but many
records that are actaully valid will be filtered out.

An example of this is the [Iowa Women of Achievement
bridge](ttps://www.openstreetmap.org/way/65066830) which is completely
valid in terms of name, geometry, and tagging but contains a URL in the
`ref` field.

The polylines importer currently selects a single name value from the
list of names in the polylines file by choosing the longest.

This PR adds an additional filter that first removes any URL-like values
from consideration, and should completely eliminate any of the otherwise
concerning errors while ensuring all valid records make it into
Elasticsearch.

Fixes pelias/whosonfirst#456
Fixes #216
Fixes pelias/docker#89
Connects pelias/model#116
  • Loading branch information
orangejulius committed Jul 3, 2019
1 parent 26caeec commit 435fe57
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 2 deletions.
7 changes: 5 additions & 2 deletions stream/parser.js
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,12 @@ function parser( precision ){
// each connected road can have one or more names
// we select one name to be the default.
function selectName( names ){
// return the longest name
// filter out URLs
// then return the longest name
// @todo: can we improve this logic?
return names.reduce( function( a, b ){
return names.filter( function ( name) {
return !name.match(/^http(s)?:\/\//);
}).reduce( function( a, b ){
return a.length > b.length ? a : b;
});
}
Expand Down
19 changes: 19 additions & 0 deletions test/stream/parser.js
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,25 @@ module.exports.tests.select_name = function(test, common) {
});
};

// URLs are filtered out
module.exports.tests.filter_url = function(test, common) {
test('parse: select name', function(t) {

var stream = parser(6);
var row = ['a','foo','foooooooo','https://this-website-should-not-be-the-name.com'].join('\0');
var expected = 'foooooooo';

function assert( actual, enc, next ){
t.deepEqual( actual.properties.name, expected, 'longest non-URL name selected' );
next();
}

stream.pipe( through.obj( assert, function(){ t.end(); } ));
stream.write(row);
stream.end();
});
};

module.exports.all = function (tape, common) {

function test(name, testFunction) {
Expand Down

0 comments on commit 435fe57

Please sign in to comment.