Benthos
The stream processor for mundane tasks
It's boringly easy to use
Written in Go, deployed as a static binary. Configured with a single YAML file, allowing you to declare connectors and a list of processing stages.
# Installcurl -Lsf https://sh.benthos.dev | bash# Runbenthos -c ./config.yaml
- Mapping
- Multiplexing
- Enrichments
input:kafka_balanced:addresses: [ TODO ]topics: [ foo, bar ]consumer_group: foogrouppipeline:processors:- jmespath:query: '{ message: @, meta: { link_count: length(links) } }'output:s3:bucket: TODOpath: "${!metadata:kafka_topic}/${!json_field:message.id}.json"
Takes Care of the Dull Stuff
Most stream processing tasks are actually just boring transformations, service queries or multiplexing. Benthos specializes in these tasks, letting you focus on the more exciting features of your architecture.
At Meltwater it's enriching over 450 million documents per day with a network of more than 20 NLP services. It sounds very interesting but rest assured, it's totally drab.
Well Connected
Benthos is able to glue a wide range of sources and sinks together and hook into a variety of databases, caches and your own services, enabling you to seamlessly deploy it without changing your existing infrastructure.
Reliable and Scalable
Benthos runs fast, has a low memory footprint and processes messages using a transaction model, making it able to guarantee at-least-once delivery even in the event of crashes or unexpected server faults.
It's completely stateless, allowing for easy deployment and liberal scaling. It also exposes metrics and tracing events to targets of your choice.
Extendable
Sometimes the components that come with Benthos aren't enough. Luckily, Benthos has been designed to be easily plugged with whatever components you need.
You can either write plugins directly in Go (recommended) or you can configure Benthos to run your plugin as a subprocess.
Sponsored by the following heroes
