Update schema for Elasticsearch 5 and 6 #372
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change makes our Elasticsearch schema compatible with Elasticsearch 5 and 6. It shouldn't have any effect on performance or operation, but it will completely drop compatibility for Elasticsearch 2. The primary change is that Elasticsearch 5 introduces two types of text fields: `text` and `keyword`, whereas Elasticsearch 2 only had 1: `string`. Roughly, a `text` field is for true full text search and a `keyword` field is for simple values that are primarily used for filtering or aggregation (for example, our `source` and `layer` fields). The `string` datatype previously filled both of those roles depending on how it was configured. Fortunately, we had already roughly created a concept similar to the `keyword` datatype in our schema, but called it `literal`. This has been renamed to `keyword` to cut down on the number of terms needed One nice effect of this change is that it removes all deprecation warnings printed by Elasticsearch 5. Notably, as discovered in #337 (comment), these warnings were quite noisy and required special handling to work around Node.js header size restrictions. This special handling can now been removed. Fixes pelias/whosonfirst#457 Connects pelias/pelias#719 Connects pelias/pelias#461
orangejulius
force-pushed
the
elasticsearch-5-schema
branch
from
229d7f2 to
ae3c4ff
Compare
|
I've now tested this fairly extensively across several small builds, and can't identify any regressions. The schema is expected to still be equivalent for ES5+, so this is now ready for full planet testing. |
This was referenced Jul 10, 2019
orangejulius
added a commit
that referenced
this pull request
Nov 22, 2019
There were previously strict Node.js version compatibility issues with the schema. However, once we dropped support for Elasticsearch 2 in #372, this was no longer required.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change makes our Elasticsearch schema compatible with Elasticsearch 5 and 6. It shouldn't have any effect on performance or operation, but it will completely drop compatibility for Elasticsearch 2.
The primary change is that Elasticsearch 5 introduces two types of text fields:
textandkeyword, whereas Elasticsearch 2 only had 1:string.Roughly, a
textfield is for true full text search and akeywordfield is for simple values that are primarily used for filtering or aggregation (for example, oursourceandlayerfields). Thestringdatatype previously filled both of those roles depending on how it was configured.Fortunately, we had already roughly created a concept similar to the
keyworddatatype in our schema, but called itliteral. This has been renamed tokeywordto cut down on the number of terms neededOne nice effect of this change is that it removes all deprecation warnings printed by Elasticsearch 5. Notably, as discovered in #337 (comment), these warnings were quite noisy and required special handling to work around Node.js header size restrictions. This special handling can now been removed.
Fixes pelias/whosonfirst#457
Connects pelias/pelias#719
Connects pelias/pelias#461