blob_storage

BETA: This component is experimental and therefore subject to change outside of major version releases.

Sends message parts as objects to an Azure Blob Storage Account container. Each object is uploaded with the filename specified with the container field.

# Common config fields, showing default values
output:
blob_storage:
storage_account: ""
storage_access_key: ""
container: ""
path: ${!count("files")}-${!timestamp_unix_nano()}.txt
max_in_flight: 1

In order to have a different path for each object you should use function interpolations described here, which are calculated per message of a batch.

Performance

This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages with the field max_in_flight.

Fields

storage_account

The storage account to upload messages to.

Type: string
Default: ""

storage_access_key

The storage account access key.

Type: string
Default: ""

container

The container for uploading the messages to. This field supports interpolation functions.

Type: string
Default: ""

# Examples
container: messages-${!timestamp("2006")}

path

The path of each message to upload. This field supports interpolation functions.

Type: string
Default: "${!count(\"files\")}-${!timestamp_unix_nano()}.txt"

# Examples
path: ${!count("files")}-${!timestamp_unix_nano()}.json
path: ${!meta("kafka_key")}.json
path: ${!json("doc.namespace")}/${!json("doc.id")}.json

blob_type

Block and Append blobs are comprised of blocks, and each blob can support up to 50,000 blocks. This field supports interpolation functions.

Type: string
Default: "BLOCK"
Options: BLOCK, APPEND.

max_in_flight

The maximum number of messages to have in flight at a given time. Increase this to improve throughput.

Type: number
Default: 1

timeout

The maximum period to wait on an upload before abandoning it and reattempting.

Type: string
Default: "5s"