First Look at the V4 Roadmap

Benthos has been at major version 3 for over a year now, and I consider it to be a pretty cool achievement that given all the dope features added we've managed to keep both the Benthos config spec and APIs fully backwards compatible.

However, eventually it would be nice to cut a new major release and prune all of the dead weight that has accumulated during this time. Since major version releases don't come often I wanted to be sure that we've considered and planned any other potential breaking changes that could be bundled along with it.

Up until now Benthos has never had a roadmap or really any plan beyond just building what we want to use or want to build, this is known in the industry as attention-span-driven development. Alas, if we're going to get mileage out of version 4 then some planning is necessary, and I figured we might as well put together our very first roadmap.

A few months ago I asked for feedback, I already had my own wish list of things to change in the next major release but I wanted to give you all an opportunity to factor in your own use cases. I've attempted to capture all of the feedback and create issues for the stuff that's achievable, then I marked the issues that require breaking changes and added them to my roadmap plans. I think it's currently in a state that works for me and is something deliverable, therefore I think it's now worth sharing and allowing you all to help shape it further.

Benthos is blessed with a decent and growing number of contributors. However, it's still clear that if I personally were to burn out then the project would pretty much grind to a temporary halt, and therefore my sanity is a higher priority than committing to a rigid plan. Here's a few things to clarify about this roadmap before you get too excited:

  1. This isn't final, it's going to mutate over time in order to flex around "everything else" going on.
  2. This isn't everything. The only items included in this roadmap are items that I consider required to have ready for v4. Any features that can definitely be implemented without breaking changes are not included and can be worked on at any time, including right now.
  3. There is no timeline or estimate for this work (by design). If you are blocked on any of the items on this roadmap and aren't able to contribute then please still make sure I'm aware and I'll factor that in, but do not expect promises or commitments (unless you're paying for them).

With that made clear and everyone sufficiently bored let's get into the planned work as it currently stands. I've created an issue for every item here where you can read more details beyond my elevator pitch.

Improved plugin APIs#

Click here to access the issue.

This is by far the biggest item of work I want to establish before v4. The plugin APIs are currently heavily tied into the same component interfaces that are used internally. This means that it's not possible for me to modify the signatures of internal components without breaking the plugin APIs. This has historically put us in awkward positions where in order to make a change that's backwards compatible with both our configuration spec and the plugin APIs we have to implement nasty tricks.

If we're instead able to isolate the plugin APIs with an air gap then it will allow us to iterate on the internal components without impacting the APIs used for plugins.

The plan is to fully implement an isolated (and nicer) plugin API, give everyone a lot of time to try it out, provide feedback, and migrate, all within good time before v4 so that I don't pull the rug out from under current plugin users.

Streams Mode API for Resources#

Click here to access the issue.

This one's pretty simple, we want to expand the streams mode APIs to allow the mutation of resources. This is blocked behind a breaking change (to the plugin APIs) as it would require sweeping changes to how resources are accessed.

Input Scheduling Capabilities#

Click here to access the issue.

Sometimes it's nice to slow things down, this issue would allow us to configure inputs that are triggered in scheduled bursts rather than realtime streams in order to have them behave similar to batch processors. Implementing this will require a minor review of the input initialization flow, which could potentially lead to breaking changes to the internal API.

Configuration Templating#

Click here to access the issue.

This would allow you to create reusable, parameterized, configuration templates and have them natively supported within Benthos. This issue is pretty dope but also a significant amount of work, it could easily result in breaking changes being required and so I'd like to have this at least planned out and understood before v4.

Improved Logging#

Click here to access the issue.

As Benthos has evolved it has gained a few oddities in how logging works. This issue adjusts logging to lean more into structured logging fields and update the configuration defaults to be more sensible. This will mostly impact internal components that create logs, and therefore depends on having the isolated plugin APIs.

Improved Metrics#

Click here to access the issue.

Similar to the logging issue, metrics in Benthos are a bit wonky due to the collision between targets that do and don't support labels/tagging. Since Prometheus and other tag based metrics types seem to be winning out nowadays I think we can flip the defaults to favour tags over long metric names.

Configuration File Reloading#

Click here to access the issue.

Pretty much self explanatory. I believe this can be implemented without any breaking changes, but it would be good to have it understood (or finished) before v4 just in case.

Tracking these Features#

There's a project on Github containing all of these issues, but the way that I've configured it is unique as issues aren't necessarily tracked by their progress. Issues in the "Blocked" column are unable to progress without a breaking change and therefore are blocked on v4. Issues in the "Unblocked" column are features that can be worked on, and will either become done if they were able to be completed without breaking changes, or will be put back into "Blocked" once they reach a point where breaking changes are needed.

Once the "Unblocked" column has been emptied, and all of our v4 issues are either blocked or done, that will indicate that we are ready to commit to a new major version release, at which point a v4 branch will be created and that work can be started.

I'm hoping that this will make it easier for me to minimize disruption. Ideally, I want the process of implementing Benthos v4 to be a simple case of deleting old deprecated stuff, and then removing flags/feature toggles in order to make new breaking features the default, having already been implemented and tested. There should be no green field work as part of the new v4 branch.

What's Next#

Make sure you get your thoughts and opinions added to the issues you're interested in. I'm also going to try and open up mini forums over our Discord server to get feedback on the plans. If any of these issues are something you'd personally like then please add a thumbs up emoji to it, as that helps me prioritize them.

If you're interested in getting involved then make sure you've joined one or more of our glorious community spaces.