Skip to main content

subprocess

Executes a command as a subprocess and, for each message, will pipe its contents to the stdin stream of the process followed by a newline.

# Common config fields, showing default values
label: ""
subprocess:
name: cat
args: []

The subprocess must then either return a line over stdout or stderr. If a response is returned over stdout then its contents will replace the message. If a response is instead returned from stderr it will be logged and the message will continue unchanged and will be marked as failed.

Rather than separating data by a newline it's possible to specify alternative codec_send and codec_recv values, which allow binary messages to be encoded for logical separation.

The execution environment of the subprocess is the same as the Benthos instance, including environment variables and the current working directory.

The field max_buffer defines the maximum response size able to be read from the subprocess. This value should be set significantly above the real expected maximum response size.

Subprocess requirements#

It is required that subprocesses flush their stdout and stderr pipes for each line. Benthos will attempt to keep the process alive for as long as the pipeline is running. If the process exits early it will be restarted.

Messages containing line breaks#

If a message contains line breaks each line of the message is piped to the subprocess and flushed, and a response is expected from the subprocess before another line is fed in.

Fields#

name#

The command to execute as a subprocess.

Type: string
Default: "cat"

# Examples
name: cat
name: sed
name: awk

args#

A list of arguments to provide the command.

Type: array
Default: []

max_buffer#

The maximum expected response size.

Type: number
Default: 65536

codec_send#

Determines how messages written to the subprocess are encoded, which allows them to be logically separated.

Type: string
Default: "lines"
Requires version 3.37.0 or newer
Options: lines, length_prefixed_uint32_be, netstring.

codec_recv#

Determines how messages read from the subprocess are decoded, which allows them to be logically separated.

Type: string
Default: "lines"
Requires version 3.37.0 or newer
Options: lines, length_prefixed_uint32_be, netstring.

parts#

An optional array of message indexes of a batch that the processor should apply to. If left empty all messages are processed. This field is only applicable when batching messages at the input level.

Indexes can be negative, and if so the part will be selected from the end counting backwards starting from -1.

Type: array
Default: []