Skip to main content

azure_queue_storage

BETA

This component is mostly stable but breaking changes could still be made outside of major version releases if a fundamental problem with the component is found.

Sends messages to an Azure Storage Queue.

Introduced in version 3.36.0.

# Common config fields, showing default values
output:
  label: ""
  azure_queue_storage:
    storage_account: ""
    storage_access_key: ""
    storage_connection_string: ""
    queue_name: ""
    max_in_flight: 1
    batching:
      count: 0
      byte_size: 0
      period: ""
      check: ""

Only one authentication method is required, storage_connection_string or storage_account and storage_access_key. If both are set then the storage_connection_string is given priority.

In order to set the queue_name you can use function interpolations described here, which are calculated per message of a batch.

Performance#

This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages with the field max_in_flight.

This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more in this doc.

Fields#

storage_account#

The storage account to upload messages to. This field is ignored if storage_connection_string is set.

Type: string
Default: ""

storage_access_key#

The storage account access key. This field is ignored if storage_connection_string is set.

Type: string
Default: ""

storage_connection_string#

A storage account connection string. This field is required if storage_account and storage_access_key are not set.

Type: string
Default: ""

queue_name#

The name of the target Queue Storage queue. This field supports interpolation functions.

Type: string
Default: ""

ttl#

The TTL of each individual message as a duration string. Defaults to 0, meaning no retention period is set This field supports interpolation functions.

Type: string
Default: "" 

# Examples


ttl: 60s


ttl: 5m


ttl: 36h

max_in_flight#

The maximum number of messages to have in flight at a given time. Increase this to improve throughput.

Type: int
Default: 1
Requires version 3.45.0 or newer

batching#

Allows you to configure a batching policy.

Type: object 

# Examples


batching:
  byte_size: 5000
  count: 0
  period: 1s


batching:
  count: 10
  period: 1s


batching:
  check: this.contains("END BATCH")
  count: 0
  period: 1m

batching.count#

A number of messages at which the batch should be flushed. If 0 disables count based batching.

Type: int
Default: 0

batching.byte_size#

An amount of bytes at which the batch should be flushed. If 0 disables size based batching.

Type: int
Default: 0

batching.period#

A period in which an incomplete batch should be flushed regardless of its size.

Type: string
Default: "" 

# Examples


period: 1s


period: 1m


period: 500ms

batching.check#

A Bloblang query that should return a boolean value indicating whether a message should end a batch.

Type: string
Default: "" 

# Examples


check: this.type == "end_of_transaction"

batching.processors#

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.

Type: array
Default: [] 

# Examples


processors:
  - archive:
      format: lines


processors:
  - archive:
      format: json_array


processors:
  - merge_json: {}