An output is a sink where we wish to send our consumed data after applying an optional array of processors. Only one output is configured at the root of a Benthos config. However, the output can be a broker which combines multiple outputs under a chosen brokering pattern, or a switch which is used to multiplex against different outputs.
An output config section looks like this:
Benthos outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Benthos will gracefully stop consuming until the issue is resolved.
When a Benthos output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka).
It's possible to instead have Benthos indefinitely retry an output until success with a
retry output. Some other outputs, such as the
broker, might also retry indefinitely depending on their configuration.
It's possible to create fallback outputs for when an output target fails using a
There are a few different ways of multiplexing in Benthos, here's a quick run through:
Some output fields support field interpolation, which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service.
For example, multiplexing against Kafka topics is a common pattern:
Refer to the field documentation for a given output to see if it support interpolation.
A more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the
Outputs that write to storage or message streaming services.amqp_0_9amqp_1aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqsazure_blob_storageazure_queue_storageazure_table_storagecacheelasticsearchgcp_pubsubhdfskafkamqttnatsnats_streamnsqredis_hashredis_listredis_pubsubredis_streamssql
Outputs that write directly to low level network protocols.http_clienthttp_servernanomsgsocketwebsocketzmq4
Outputs that write to Amazon Web Services products.aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqs
Outputs that write to Google Cloud Platform services.gcp_pubsub
Outputs that write to Microsoft Azure services.azure_blob_storageazure_queue_storageazure_table_storage