Migrating to Version 3
Benthos version 3 comes with some breaking service and configuration changes as well as a few breaking API changes.
Service
Memory Map File Buffer Removed
The long deprecated mmap_file
buffer has been removed. If you were still relying on this buffer implementation then please raise an issue.
Old Metrics Paths Removed
The following undocumented metrics paths have been removed:
input.parts.count
input.read.success
input.read.error
input.send.success
input.send.error
input.ack.success
input.ack.error
output.running
output.parts.count
output.send.success
output.parts.send.success
output.send.error
All of these paths have remaining equivalents.
Metric Path Changes
The http_client
output client metrics have been renamed from output.*.output.http_client
to output.*.client
.
Configuration
Metrics Prefix
The configuration field prefix
within metrics
has been moved from the root
of the config object to individual types. E.g. when using statsd
the field
metrics.prefix
should be replaced with metrics.statsd.prefix
.
JSON Paths
Many components within Benthos use an unspecified "JSON dot path" syntax for querying and setting fields within JSON documents. The format of these paths has been formalised to make them clearer and more generally useful, but this potentially breaks your paths when they query against hierarchies that contain arrays.
The formal specification for v3 can be found in this document.
The following components are affected:
awk
processor (all of thejson_*
functions)json
processor (path
field)process_field
processor (path
field)process_map
processor (premap
,premap_optional
,postmap
andpostmap_optional
fields)check_field
condition (path
field)json_field
function interpolations3
input (sqs_body_path
,sqs_bucket_path
andsqs_envelope_path
fields)dynamodb
output (json_map_columns
field values)
Migration Guide
In order to replicate the exact same behaviour as currently exists your paths should be updated to include the character *
wherever an array exists. For example, the default value of sqs_body_path
for the s3
input has been updated from Records.s3.object.key
to Records.*.s3.object.key
.
Process DAG Stage Names
The process_dag
processor now only permits workflow stage names matching the following regular expression: [a-zA-Z0-9_-]+
. The reasoning for this restriction is to potentially expand the features of process_dag
in the future with custom root fields (e.g. $on_error
).
Go API
Modules
Benthos now fully adheres to Go Modules, import paths must therefore now contain the major version (v3) like so:
import "github.com/Jeffail/benthos/v3/lib/processor"
It should be pretty quick to update your imports, either using a tool or just:
grep "Jeffail/benthos" . -Rl | grep -e "\.go$" | xargs -I{} sed -i 's/Jeffail\/benthos/Jeffail\/benthos\/v3/g' {}
Other
- Constructors for buffer components now require a
types.Manager
, giving them parity with other components:buffer.New(conf Config, mgr types.Manager, log log.Modular, stats metrics.Type) (Type, error)