Outputs
An output is a sink where we wish to send our consumed data after applying an optional array of processors. Only one output is configured at the root of a Benthos config. However, the output can be a broker which combines multiple outputs under a chosen brokering pattern, or a switch which is used to multiplex against different outputs.
An output config section looks like this:
#
Back PressureBenthos outputs apply back pressure to components upstream. This means if your output target starts blocking traffic Benthos will gracefully stop consuming until the issue is resolved.
#
RetriesWhen a Benthos output fails to send a message the error is propagated back up to the input, where depending on the protocol it will either be pushed back to the source as a Noack (e.g. AMQP) or will be reattempted indefinitely with the commit withheld until success (e.g. Kafka).
It's possible to instead have Benthos indefinitely retry an output until success with a retry
output. Some other outputs, such as the broker
, might also retry indefinitely depending on their configuration.
#
Dead Letter QueuesIt's possible to create fallback outputs for when an output target fails using a try
output:
#
Multiplexing OutputsThere are a few different ways of multiplexing in Benthos, here's a quick run through:
#
Interpolation MultiplexingSome output fields support field interpolation, which is a super easy way to multiplex messages based on their contents in situations where you are multiplexing to the same service.
For example, multiplexing against Kafka topics is a common pattern:
Refer to the field documentation for a given output to see if it support interpolation.
#
Switch MultiplexingA more advanced form of multiplexing is to route messages to different output configurations based on a query. This is easy with the switch
output:
#
Categories- Services
- Network
- AWS
- GCP
- Azure
- Local
- Utility
Outputs that write to storage or message streaming services.
amqp_0_9amqp_1aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqsazure_blob_storageazure_queue_storageazure_table_storagecacheelasticsearchgcp_pubsubhdfskafkamqttnatsnats_streamnsqredis_hashredis_listredis_pubsubredis_streamssqlOutputs that write directly to low level network protocols.
http_clienthttp_servernanomsgsocketwebsocketzmq4Outputs that write to Amazon Web Services products.
aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqsOutputs that write to Google Cloud Platform services.
gcp_pubsubOutputs that write to Microsoft Azure services.
azure_blob_storageazure_queue_storageazure_table_storage