I need help, attention and affirmation, and therefore it's time for a development update. Around five months ago I posted a roadmap for Benthos v4 that included some utterly unattainable goals that only a super human could achieve.
Now that most of those features are ready to test, namely a new plugins API and config templating, I'm looking for people to try them out and give feedback. Please read on if that sounds like fun to you, or also if it doesn't sound fun but you intend to do it anyway.
The new config templates functionality allows you to define parameterised templates for Benthos configuration snippets. These templates can then be imported with a cli flag and used in Benthos configs like native Benthos components.
This is going to be super useful in situations where you have commonly used configuration patterns with small differences that prevent you from using resources.
The current state of templates is that they'll be included in the next release as an experimental feature, meaning any aspect of this functionality is subject to change outside of major version releases. This includes the config spec of templates, how they work, and so on.
Defining a template looks roughly like this:
And you're able to import templates with the
And using it in a config looks like any other component:
Benthos has had Go plugins for a while now and they're fairly well received. However, they can sometimes be confusing as they expose Benthos internals that aren't necessary to understand as plugin authors.
It was also an issue for me as a maintainer that the current plugin APIs hook directly into Benthos packages that have no business being public. This makes it extra difficult to improve the service without introducing breaking changes.
The new APIs are simpler, more powerful (in the ways that matter), add milk after the water, and most importantly are air-gapped from Benthos internals so that they can evolve independently. Here's a sneaky glance of what a processor plugin looks like:
You can play around with these APIs right now by pulling the latest commit with:
And you can find more examples along with the API docs at pkg.go.dev.
The package will remain in an experimental state under
public/x/service for a month or so, and once it's "ready" (I'm personally happy with it) then it'll be moved to
public/service and will be considered stable.
The goal is to allow everyone to migrate to the new APIs whilst still supporting the old ones, and then when Benthos V4 is tagged the old ones will vanish and we're no longer blocked on them.
Similar to the templates there is a Github discussion open for feedback. Be honest, be brutal.
I've been babbling on for months so if this stuff is news to you then you're clearly out of the loop. Worry not, for you can remedy the situation by joining one or more of our glorious community spaces.