95_analyzed_vs_not.asciidoc

Aggregations and Analysis

Some aggregations, such as the terms bucket, operate on string fields. And string fields may be either analyzed or not_analyzed, which begs the question: how does analysis affect aggregations?

The answer is "a lot", but it is best shown through an example. First, index some documents representing some different states in the US:

POST /agg_analysis/data/_bulk
{ "index": {}}
{ "state" : "New York" }
{ "index": {}}
{ "state" : "New Jersey" }
{ "index": {}}
{ "state" : "New Mexico" }
{ "index": {}}
{ "state" : "New York" }
{ "index": {}}
{ "state" : "New York" }

We want to build a list of unique states in our dataset, complete with counts. Simple…​let’s use a terms bucket:

GET /agg_analysis/data/_search?search_type=count
{
    "aggs" : {
        "states" : {
            "terms" : {
                "field" : "state"
            }
        }
    }
}

Which gives us the results…​.

{
...
   "aggregations": {
      "states": {
         "buckets": [
            {
               "key": "new",
               "doc_count": 5
            },
            {
               "key": "york",
               "doc_count": 3
            },
            {
               "key": "jersey",
               "doc_count": 1
            },
            {
               "key": "mexico",
               "doc_count": 1
            }
         ]
      }
   }
}

Oh dear, that’s not at all what we want! Instead of counting states, the aggregation is counting individual words. The underlying reason is simple: aggregations are built from the inverted index, and the inverted index is "post-analysis".

When we added those documents to Elasticsearch, the string "New York" was analyzed/tokenized into ["new", "york"]. These individual tokens were then used to populate fielddata, and ultimately we see counts for "new" instead of "New York"

This is obviously not the behavior that we wanted, but luckily it is easily corrected.

We need to define a multi-field for "state" and set it to not_analyzed. This will prevent "New York" from being analyzed, which means it will stay a single token in the aggregation. Let’s try the whole process over, but this time specify a "raw" multi-field:

DELETE /agg_analysis/
PUT /agg_analysis
{
  "mappings": {
    "data": {
      "properties": {
        "state" : {
          "type": "string",
          "fields": {
            "raw" : {
              "type": "string",
              "index": "not_analyzed"(1)
            }
          }
        }
      }
    }
  }
}

POST /agg_analysis/data/_bulk
{ "index": {}}
{ "state" : "New York" }
{ "index": {}}
{ "state" : "New Jersey" }
{ "index": {}}
{ "state" : "New Mexico" }
{ "index": {}}
{ "state" : "New York" }
{ "index": {}}
{ "state" : "New York" }

GET /agg_analysis/data/_search?search_type=count
{
  "aggs" : {
    "states" : {
        "terms" : {
            "field" : "state.raw" (2)
        }
    }
  }
}

1. This time we explicitly map the "state" field and include a not_analyzed sub-field.

2. The aggregation is run on "state.raw" instead of "state".

Now when we run our aggregation, we get results that make sense:

{
...
   "aggregations": {
      "states": {
         "buckets": [
            {
               "key": "New York",
               "doc_count": 3
            },
            {
               "key": "New Jersey",
               "doc_count": 1
            },
            {
               "key": "New Mexico",
               "doc_count": 1
            }
         ]
      }
   }
}

In practice, this kind of problem is very easy to spot. Your aggregations will simply return strange buckets and you’ll remember the analysis issue. It is a generalization, but there are not many instances where you want to use an analyzed field in an aggregation. When in doubt, add a multi-field so that you have the option for both.

High Cardinality Memory Implications

There is another reason to avoid aggregating analyzed fields — high cardinality fields consume a large amount of memory when loaded into fielddata. The analysis process often (although not always) generates a large number of tokens, many of which are unique. This increases the overall cardinality of the field and contributes to more memory pressure.

Some types of analysis are extremely unfriendly with regards to memory. Consider an ngram analysis process. The term "New York" might be ngram’ed into the following tokens:

• ne

• ew

• w{nbsp}

• {nbsp}y

• yo

• or

• rk

You can imagine how the ngramming process creates a huge number of unique tokens, especially when analyzing paragraphs of text. When these are loaded into memory you can easily exhaust your heap space.

So, before aggregating across fields, take a second to verify that the fields are not_analyzed. And if you want to aggregate analyzed fields, ensure the analysis process is not creating an obscene number of tokens.

Tip

At the end of the day, it doesn’t matter whether a field is analyzed or not_analyzed. The more unique values in a field — the higher the cardinality of the field —  the more memory that is required. This is especially true for string fields where every unique string must be held in memory — longer strings use more memory.