A Benthos stream consists of four components; an input, an optional buffer, processor pipelines and an output. Under normal use a Benthos instance is a single stream, and these components are configured within the service config file.
Alternatively, Benthos can be run in
--streams mode, where a single running
Benthos instance is able to run multiple entirely isolated streams. Adding
streams in this mode can be done in two ways:
Static configuration files allows you to maintain a directory of static stream configuration files that will be traversed by Benthos.
An HTTP REST API allows you to dynamically create, read the status of, update, and delete streams at runtime.
These two methods can be used in combination, i.e. it's possible to update and delete streams that were created with static files.
resource section of a Benthos config defines named resources (
rate_limits, etc) that can be referenced throughout a stream configuration.
When running in streams mode these resources are also shared across streams.
An individual stream config should not define any resources. Instead, they should be defined in a service-wide config:
benthos --streams -c ./config.yaml
This means it is not currently possible to modify resources dynamically. If this is blocking you from using streams mode then consider raising a ticket.
Metrics from all streams are aggregated and exposed via the method specified in
the config of the Benthos instance running in
--streams mode, with
their metrics prefixed by their respective stream name.
For example, a Benthos instance running in streams mode with the configured
benthos running a stream named
foo would have metrics from
registered with the prefix
This can cause problems if your streams are short lived and uniquely named as
the number of metrics registered will continue to climb indefinitely. In order
to avoid this you should either limit the streams that register their metrics
# Only register metrics for the stream `foo`. Others will be ignored.metrics:whitelist:paths:- foochild:type: prometheusprefix: benthos
rename rules to set the same prefix for groups of streams:
# Rename all stream metric prefixes of the form `foo_<uuid_v4>` to just `foo`.metrics:rename:by_regexp:- pattern: "foo_[0-9\\-a-zA-Z]+\\.(.*)"value: "foo.$1"child:type: prometheusprefix: benthos