The v4 roadmap was outlined more than a year ago, in that time all the major features planned have been completed and released in backwards compatible ways into v3, leaving only the breaking changes left (the fun stuff).
We're now also at a point where the breaking changes that we want to include in v4 have been completed on a branch, and a migration guide is ready to use which describes how to prepare for v4. Therefore, it seems to me, that we're ready to actually prepare the release.
Why
Benthos follows semantic versioning, which means a major version release is not necessarily an indication of significant improvements to the product, but an indication that things have changed and therefore upgrading should be done with care (and likely some effort). As such, all the major improvements that could have been part of this release (config templates, new plugin APIs, etc) are already part of version 3 and being actively used.
The vast majority of these improvements have been an evolution or consolidation of prior features, which have since become deprecated (and hidden from the documentation). Doing things this way has enabled you to slowly learn about and adopt the nice new goodies whilst also running the old stuff. However, the reality is that the old deprecated features are a maintenance burden. Not only do we need to ensure that a growing number of potentially unused features remain bug free, but these features often frustrate or entirely prevent us from making improvements elsewhere.
As such, after keeping v3 going for more than two years, it feels like it's about the right time for spring cleaning. The absolute top priority of this release is to remove deprecated components, which for people keeping up with the newest features would in theory mean no breaking changes at all. However, since major releases are a rare event (this could possibly be the last one) it also makes sense to use this as an opportunity to change things about Benthos that we feel aren't intuitive.
The biggest of these intentionally breaking changes is the way in which metrics work. Benthos exposes a huge amount of metrics but the names can often be confusing and difficult to understand, and some components expose inconsistent or undocumented metrics. For v4 they have been massively simplified and made consistent, leaning on labels/tags for presenting (optionally) granular series, and allowing full customisation for all metric destination types with a generic mapping field.
The full list of changes we've decided to add are outlined in the the migration guide, which is an attempt at finding a good balance between keeping to a minimal impact for the majority of users, and tackling the biggest culprits of confusion in Benthos land. It's extremely important to stress that at this stage nothing is absolute, we can pull these changes back if they seem to heavy, and likewise we can push further if there's hunger for more. Ultimately, we need feedback in the coming weeks in order to get this right.
Timescales
None of this is set in stone but it currently looks as though we will have a release candidate available for v4 in the coming weeks. You can expect edge tagged docker images to include v4 changes soon, followed by an image tagged 4.0.0-rc1 and so on, at the same time there will be artifacts you can obtain from the github releases page: https://github.com/Jeffail/benthos/releases.
At this stage we'll be looking for general testing as well as feedback on the migration process. If you find yourself in a situation where a feature you rely upon in v3 is gone and you can't find an alternative in v4 then we want to hear about it as soon as possible. There will be a temporary website up with v4 documentation but the URL will change each time the docs are updated, so we'll post them in several of the community spaces, make sure you're in one if you're interested in testing.
We'll likely be playing with release candidates for a few weeks, in which time any official releases we put out will be v3 patches for any bug fixes we've added in the meantime. Once we're happy with release candidates the official v4 tag will be pushed, built and the documentation site will be bumped to represent v4. When this happens the domain v3.benthos.dev will be set up to contain the v3 documentation.
Preparing
If you're interested in minimizing the effort to migrate to v4 right now then the first thing worth doing is moving away from any deprecated components in your v3 configs. You can detect them by pulling the latest version of v3 and running the linter with the --deprecated flag:
benthos lint --deprecated ./configs/*.yaml
This will report any deprecated components and fields that you still have in your configs.
Next, if you're using plugins or have a custom build of Benthos then you need to ensure you're no longer using any of the packages within ./lib, all of those packages are being removed in v4 in favour of the new ./public packages. The plugin APIs within ./lib all have a newer (and better) alternative, but if you were using some of the misc packages as helper libraries then copy them from the v3 branch into your codebase.
The metrics changes are likely to affect almost every deployment, it is recommended that you make the best use of the new labels and rework your dashboards. However, if in your case the dashboards are difficult to change then it is likely easier to use a mapping in your new v4 config to reproduce the metrics series your dashboards currently rely upon. In order to prepare for this it's worth making an inventory of each series you use, the labels they have, and preparing the mapping you'll need. The documentation for the new metrics system can be found here: https://github.com/Jeffail/benthos/blob/main/website/docs/components/metrics/about.md
Finally, it's worth getting familiar with the changes outlined in the migration guide. There are breaking changes in some components, mostly default values being changed in order to make them more intuitive, but they should all be understood before upgrading in order to avoid unintended consequences.
Please Help
If you're interested in helping out then we'd love to get you involved and the best way is to read the migration guide, try out the release candidates, and give us feedback in any of the community spaces.
The major concern that we're looking for feedback on is whether the breaking changes are manageable for you, if you have identified things that will significantly hinder your migration then it's important we hear about it as soon as possible. This allows us to triage whether it's worth building tooling to help with the transition, or to potentially roll-back those changes if necessary.