broker

Allows you to route messages to multiple child outputs using a range of brokering patterns.

output:
broker:
pattern: fan_out
outputs: []
batching:
count: 1
byte_size: 0
period: ""

Processors can be listed to apply across individual outputs or all outputs:

output:
broker:
pattern: fan_out
outputs:
- foo:
foo_field_1: value1
- bar:
bar_field_1: value2
bar_field_2: value3
# Processors only applied to messages sent to bar.
processors:
- type: bar_processor
# Processors applied to messages sent to all brokered outputs.
processors:
- type: some_processor

Batching

It's possible to configure a batch policy with a broker using the batching fields, allowing you to create batches after your processing stages. Some inputs do not support broker based batching and specify this in their documentation.

Fields

copies

number The number of copies of each configured output to spawn.

pattern

string The brokering pattern to use.

Options are: fan_out, fan_out_sequential, round_robin, greedy, try.

outputs

array A list of child outputs to broker.

batching

object Allows you to configure a batching policy.

# Examples
batching:
byte_size: 5000
period: 1s
batching:
count: 10
period: 1s
batching:
condition:
text:
arg: END BATCH
operator: contains
period: 1m

batching.count

number A number of messages at which the batch should be flushed. If 0 disables count based batching.

batching.byte_size

number An amount of bytes at which the batch should be flushed. If 0 disables size based batching.

batching.period

string A period in which an incomplete batch should be flushed regardless of its size.

# Examples
period: 1s
period: 1m
period: 500ms

batching.condition

object A condition to test against each message entering the batch, if this condition resolves to true then the batch is flushed.

batching.processors

array A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.

# Examples
processors:
- archive:
format: lines
processors:
- archive:
format: json_array
processors:
- merge_json: {}

Patterns

The broker pattern determines the way in which messages are allocated and can be chosen from the following:

fan_out

With the fan out pattern all outputs will be sent every message that passes through Benthos in parallel.

If an output applies back pressure it will block all subsequent messages, and if an output fails to send a message it will be retried continuously until completion or service shut down.

fan_out_sequential

Similar to the fan out pattern except outputs are written to sequentially, meaning an output is only written to once the preceding output has confirmed receipt of the same message.

round_robin

With the round robin pattern each message will be assigned a single output following their order. If an output applies back pressure it will block all subsequent messages. If an output fails to send a message then the message will be re-attempted with the next input, and so on.

greedy

The greedy pattern results in higher output throughput at the cost of potentially disproportionate message allocations to those outputs. Each message is sent to a single output, which is determined by allowing outputs to claim messages as soon as they are able to process them. This results in certain faster outputs potentially processing more messages at the cost of slower outputs.

try

The try pattern attempts to send each message to only one output, starting from the first output on the list. If an output attempt fails then the broker attempts to send to the next output in the list and so on.

This pattern is useful for triggering events in the case where certain output targets have broken. For example, if you had an output type http_client but wished to reroute messages whenever the endpoint becomes unreachable you could use a try broker.