Bloblang
Bloblang, or blobl for short, is a language designed for mapping data of a wide variety of forms. It's a safe, fast, and powerful way to perform document mapping within Benthos. It also has a Go API for writing your own functions and methods as plugins.
Bloblang is available as a processor and it's also possible to use blobl queries in function interpolations.
You can also execute Bloblang mappings on the command-line with the blobl
subcommand:
This document outlines the core features of the Bloblang language, but if you're totally new to Bloblang then it's worth following the walkthrough first.
#
AssignmentA Bloblang mapping expresses how to create a new document by extracting data from an existing input document. Assignments consist of a dot path argument on the left-hand side describing a field to be created within the new document, and a right-hand side query describing what the content of the new field should be.
The keyword root
on the left-hand side refers to the root of the new document, the keyword this
on the right-hand side refers to the current context of the query, which is the read-only input document when querying from the root of a mapping:
Since the document being created starts off empty it is sometimes useful to begin a mapping by copying the entire contents of the input document, which can be expressed by assigning this
to root
.
If the new document root
is never assigned to or otherwise mutated then the original document remains unchanged.
#
Non-structured DataBloblang is able to map data that is unstructured, whether it's a log line or a binary blob, by referencing it with the content
function, which returns the raw bytes of the input document:
And your newly mapped document can also be unstructured, simply assign a value type to the root
of your document:
And the resulting message payload will be the raw value you've assigned.
#
DeletingIt's possible to selectively delete fields from an object by assigning the function deleted()
to the field path:
#
VariablesAnother type of assignment is a let
statement, which creates a variable that can be referenced elsewhere within a mapping. Variables are discarded at the end of the mapping and are mostly useful for query reuse. Variables are referenced within queries with $
:
#
MetadataBenthos messages contain metadata that is separate from the main payload, in Bloblang you can modify the metadata of the resulting message with the meta
assignment keyword, and you can query the metadata of the input message with the meta
function:
The meta
function returns the read-only metadata of the input message, so it will not reflect changes you've made within the same mapping. This is why it's possible to begin a mapping by removing all old metadata meta = deleted()
and still be able to query the original metadata.
If you wish to set a metadata value and then refer back to it later then first set it as a variable.
#
Special Characters in PathsQuotes can be used to describe sections of a field path that contain whitespace, dots or other special characters:
#
CoalesceThe pipe operator (|
) used within brackets allows you to coalesce multiple candidates for a path segment. The first field that exists and has a non-null value will be selected:
Opening brackets on a field begins a query where the context of this
changes to value of the path it is opened upon, therefore in the above example this
within the brackets refers to the contents of this.thing
.
#
LiteralsBloblang supports number, boolean, string, null, array and object literals:
The values within literal arrays and objects can be dynamic query expressions, as well as the keys of object literals.
#
CommentsYou might've already spotted, comments are started with a hash (#
) and end with a line break:
#
Boolean Logic and ArithmeticBloblang supports a range of boolean operators !
, >
, >=
, ==
, <
, <=
, &&
, ||
and arithmetic operators +
, -
, *
, /
, %
:
#
Conditional MappingUse if
expressions to perform maps conditionally:
And add an else
for alternative maps:
#
Pattern MatchingA match
expression allows you to perform conditional mappings on a value, each case should be either a boolean expression, a literal value to compare against the target value, or an underscore (_
) which captures values that have not matched a prior case:
Within a match block the context of this
changes to the pattern matched expression, therefore this
within the match expression above refers to this.doc
.
Match cases can specify a literal value for simple comparison:
The match expression can also be left unset which means the context remains unchanged, and the catch-all case can also be omitted:
If no case matches then the mapping is skipped entirely, hence we would end up with the original document in this case.
#
FunctionsFunctions can be placed anywhere and allow you to extract information from your environment, generate values, or access data from the underlying message being mapped:
You can find a full list of functions in this doc.
#
MethodsMethods provide most of the power in Bloblang as they allow you to augment query values and can be added to any expression:
You can find a full list of methods in this doc.
#
MapsDefining named maps allows you to reuse common mappings on values with the apply
method:
Within a map the keyword root
refers to a newly created document that will replace the target of the map, and this
refers to the original value of the target. The argument of apply
is a string, which allows you to dynamically resolve the mapping to apply.
#
Import MapsIt's possible to import maps defined in a file with an import
statement:
Imports from a Bloblang mapping within a Benthos config are relative to the process running the config. Imports from an imported file are relative to the file that is importing it.
#
FilteringBy assigning the root of a mapped document to the deleted()
function you can delete a message entirely:
#
Error HandlingFunctions and methods can fail under certain circumstances, such as when they receive types they aren't able to act upon. These failures, when not caught, will cause the entire mapping to fail. However, the method catch
can be used in order to return a value when a failure occurs instead:
Since catch
is a method it can also be attached to bracketed map expressions:
And one of the more powerful features of Bloblang is that a single catch
method at the end of a chain of methods can recover errors from any method in the chain:
However, the catch
method only acts on errors, sometimes it's also useful to set a fall back value when a query returns null
in which case the method or
can be used the same way:
#
Unit TestingIt's possible to execute unit tests for your Bloblang mappings using the standard Benthos unit test capabilities outlined in this document.