Benthos is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction based resiliency with back pressure, so when connecting to at-least-once sources and sinks it's able to guarantee at-least-once delivery without needing to persist messages during transit.
It's simple to deploy, comes with a wide range of connectors, and is totally data agnostic, making it easy to drop into your existing infrastructure. Benthos has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative.
Benthos is ready to commit to this relationship, are you?Get Started
Inputs that consume from storage or message streaming services.amqp_0_9amqp_1aws_kinesisaws_s3aws_sqsazure_blob_storageazure_queue_storagegcp_pubsubhdfskafkakinesismqttnatsnats_streamnsqredis_listredis_pubsubredis_streams
Inputs that consume directly from low level network protocols.http_clienthttp_servernanomsgsocketsocket_serverwebsocketzmq4
Inputs that consume from Google Cloud Platform services.gcp_pubsub
Inputs that provide utility by generating data or combining/wrapping other inputs.brokerdynamicgenerateinprocread_untilresourcesequencesubprocess
Processors that specialize in translating messages from one format to another.archiveavrobloblangcompressdecompressgrokparse_logprotobufunarchivexml
Higher level processors that compose other processors and modify their behavior.branchcatchfor_eachgroup_bygroup_by_valueinsert_partparallelswitchtrywhileworkflow
Outputs that write to storage or message streaming services.amqp_0_9amqp_1aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqsazure_blob_storageazure_queue_storageazure_table_storagecacheelasticsearchgcp_pubsubhdfskafkamqttnatsnats_streamnsqredis_hashredis_listredis_pubsubredis_streamssql
Outputs that write directly to low level network protocols.http_clienthttp_servernanomsgsocketwebsocketzmq4
Outputs that write to Amazon Web Services products.aws_dynamodbaws_kinesisaws_kinesis_firehoseaws_s3aws_snsaws_sqs
Outputs that write to Google Cloud Platform services.gcp_pubsub
Outputs that write to Microsoft Azure services.azure_blob_storageazure_queue_storageazure_table_storage