process_map

A processor that extracts and maps fields identified via dot path from the original payload into a new object, applies a list of processors to the newly constructed object, and finally maps the result back into the original payload.

process_map:
conditions: []
premap: {}
premap_optional: {}
processors: []
postmap: {}
postmap_optional: {}

This processor is useful for performing processors on subsections of a payload. For example, you could extract sections of a JSON object in order to construct a reduced request object for an [http](/docs/components/processors/http processor, then map the result back into a field within the original object.

The order of stages of this processor are as follows:

  • Conditions are tested (if specified) against each message, messages that do not pass will not be processed.
  • Messages that are flagged for processing are mapped according to the premap fields, creating a new object. If the premap stage fails (targets are not found) the message will not be processed.
  • Messages that are mapped are processed as a batch.
  • After all child processors are applied to the mapped messages they are mapped back into the original messages they originated from following the postmap fields. If the postmap stage fails the mapping is skipped and the message payload remains as it started.

If the premap is empty then the full payload is sent to the processors, if the postmap is empty then the processed result replaces the original contents entirely.

Batch Ordering

This processor supports batched messages, but the list of processors to apply must NOT change the ordering (or count) of the messages (do not use a group_by processor, for example).

Fields

conditions

array A list of conditions to test against messages. If any condition fails then the message will not be mapped and processed.

premap

object A map of source to destination paths used to create a new object from the original. An empty (or dot .) path indicates the root of the object. If a map source is not found then the message will not be processed, for optional sources use the field premap_optional.

# Examples
premap:
.: field.from.document
premap:
bar.baz: root.extra.baz
foo: root.body.foo

premap_optional

object A map of optional source to destination paths used to create a new object from the original.

processors

array A list of processors to apply to mapped payloads.

postmap

object A map of destination to source paths used to map results from processing back into the original payload. An empty (or dot .) path indicates the root of the object. If a source is not found then the mapping is abandoned, for optional sources use the postmap_optional field.

# Examples
postmap:
results.foo: .

postmap_optional

object A map of optional destination to source paths used to map results from processing back into the original payload.

parts

array An optional array of message indexes of a batch that the processor should apply to. If left empty all messages are processed. This field is only applicable when batching messages at the input level.

Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1.

Examples

Given a message payload of:

{
"doc": {
"id": "foo",
"title": "foo bar baz",
"description": "here's a thing",
"content": "this is a body"
}
}

We might wish to perform language detection on the doc.content field by sending it to a hypothetical HTTP service. We do not wish to overwrite the original document with the result, and instead want to place it within the path doc.language, and so this is a good use case for process_map:

pipeline:
processors:
- process_map:
premap:
content: doc.content
processors:
- http:
request:
url: http://localhost:1234
postmap:
doc.language: .

With the above config we would send our target HTTP service the payload {"content":"this is a body"}, and whatever the service returns will get mapped into our original document:

{
"doc": {
"id": "foo",
"title": "foo bar baz",
"description": "here's a thing",
"content": "this is a body",
"language": {
"code": "en",
"certainty": 0.2
}
}
}