Unable to Perform Full-Text Search on Fields Indexed via dynamic_mapping in Quickwit

[systemslife]

I can perform full-text searches on fields indexed via field_mappings in Quickwit, but I am unable to search on fields indexed through dynamic_mapping. For example, with the schema below, I can search directly on fields like levelname or trace_id, but I cannot search on plan_name because it is indexed through dynamic_mapping.

Example query:

• query=ERROR: This search returns the document successfully.
• query=basic: This search does not return any results because plan_name is indexed using dynamic_mapping.

How can I perform a search on fields indexed through dynamic_mapping (like plan_name) in the same way as field_mappings?

Example Schema:

{
  "@timestamp": "2025-02-09T10:25:34.745128Z",
  "plan_name": "basic", 
  "levelname": "DEBUG", 
  "message": "[ERROR]",
  "time": "2025-02-09T10:25:34.744697Z",
  "trace_id": "b5ab2adc-e8c2-41de-bce6-069b734b66b1"
}

Indexing Configuration:

version: 0.7

index_id: app-logs

doc_mapping:
  mode: dynamic
  dynamic_mapping:
    indexed: true
    tokenizer: default
    record: position
    stored: true
  field_mappings:
    - name: "@timestamp"
      type: datetime
      input_formats:
        - "%Y-%m-%dT%H:%M:%S.%fZ"
      fast: true
    - name: levelname
      type: text
      tokenizer: raw
    - name: trace_id
      type: text
      tokenizer: raw
    - name: message
      type: text
      tokenizer: default
      record: position
      stored: true
  timestamp_field: "@timestamp"
  store_source: true

indexing_settings:
  commit_timeout_secs: 10

search_settings:
  default_search_fields: ["message"]

[fulmicoton]

It is not exactly because of that.

default_search_fields tells quickwit to search into message by default, that is if you do not target a specific field in your query.
If you search for plan_name:basic I think your doc will show up.

Two possibilities here:

• You wanted to add specifically plan_name as a default search. dynamic fields are not supported for default serach field in tantivy. You will have to add an entry for plan_name in the field mapping.
• You wanted to search by default into ALL dynamic fields. Not just a few. This is made possible by adding a concatenated field https://quickwit.io/docs/main-branch/configuration/index-config#concatenate with include_dynamic_fields: true and set it as the default search field.

I can't find concatenate field in the doc of 0.8. I cannot tell if this is because we just released them without documenting them or if it was really not there at that time.

[systemslife]

Thanks! The concatenate type solved my issue. However, as you mentioned, it seems that this feature was not added in version 0.8.

That said, I see that the concatenate improvement was merged in a pull request opened in 2024. Shouldn't it be available in versions released after that date?

After checking the tags one by one, I found that the qw-iridescent-20250128 Docker tag includes this concatenate improvement, so I am using it now. @fulmicoton

[fulmicoton]

@systemslife I am glad you found a workaround.

qw-iridescent-20250128

FYI, these unofficial tags are images we build for customers.