An output is a sink where we wish to send our consumed data after applying an optional array of processors. Only one output is configured at the root of a Benthos config. However, the output can be a broker which combines multiple outputs under a chosen brokering pattern, or a switch which is used to multiplex against different outputs.
An output config section looks like this:
Benthos outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Benthos will gracefully stop consuming until the issue is resolved.
When a Benthos output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka).
It's possible to instead have Benthos indefinitely retry an output until success with a
retry output. Some other outputs, such as the
broker, might also retry indefinitely depending on their configuration.
Dead Letter Queues
It's possible to create fallback outputs for when an output target fails using a
There are a few different ways of multiplexing in Benthos, here's a quick run through:
Some output fields support field interpolation, which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service.
For example, multiplexing against Kafka topics is a common pattern:
Refer to the field documentation for a given output to see if it support interpolation.
A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the