http_client

Sends messages to an HTTP server.

# Common config fields, showing default values
output:
http_client:
url: http://localhost:4195/post
verb: POST
headers:
Content-Type: application/octet-stream
rate_limit: ""
timeout: 5s
max_in_flight: 1
batching:
count: 1
byte_size: 0
period: ""

When the number of retries expires the output will reject the message, the behaviour after this will depend on the pipeline but usually this simply means the send is attempted again until successful whilst applying back pressure.

The URL and header values of this type can be dynamically set using function interpolations described here.

The body of the HTTP request is the raw contents of the message payload. If the message has multiple parts (is a batch) the request will be sent according to RFC1341. This behaviour can be overridden by archiving your batches.

Propagating Responses

It's possible to propagate the response from each HTTP request back to the input source by setting propagate_response to true. Only inputs that support synchronous responses are able to make use of these propagated responses.

Performance

This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages with the field max_in_flight.

This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more in this doc.

Fields

url

The URL to connect to. This field supports interpolation functions.

Type: string
Default: "http://localhost:4195/post"

verb

A verb to connect with

Type: string
Default: "POST"

# Examples
verb: POST
verb: GET
verb: DELETE

headers

A map of headers to add to the request. This field supports interpolation functions.

Type: object
Default: {"Content-Type":"application/octet-stream"}

# Examples
headers:
Content-Type: application/octet-stream

oauth

Allows you to specify open authentication.

Type: object
Default: {"access_token":"","access_token_secret":"","consumer_key":"","consumer_secret":"","enabled":false,"request_url":""}

# Examples
oauth:
access_token: baz
access_token_secret: bev
consumer_key: foo
consumer_secret: bar
enabled: true
request_url: http://thisisjustanexample.com/dontactuallyusethis

basic_auth

Allows you to specify basic authentication.

Type: object
Default: {"enabled":false,"password":"","username":""}

# Examples
basic_auth:
enabled: true
password: bar
username: foo

tls

Custom TLS settings can be used to override system defaults.

Type: object
Default: {"client_certs":[],"enabled":false,"root_cas_file":"","skip_cert_verify":false}

tls.enabled

Whether custom TLS settings are enabled.

Type: bool
Default: false

tls.skip_cert_verify

Whether to skip server side certificate verification.

Type: bool
Default: false

tls.root_cas_file

The path of a root certificate authority file to use.

Type: string
Default: ""

tls.client_certs

A list of client certificates to use.

Type: array
Default: []

# Examples
client_certs:
- cert: foo
key: bar
client_certs:
- cert_file: ./example.pem
key_file: ./example.key

copy_response_headers

Sets whether to copy the headers from the response to the resulting payload.

Type: bool
Default: false

rate_limit

An optional rate limit to throttle requests by.

Type: string
Default: ""

timeout

A static timeout to apply to requests.

Type: string
Default: "5s"

retry_period

The base period to wait between failed requests.

Type: string
Default: "1s"

max_retry_backoff

The maximum period to wait between failed requests.

Type: string
Default: "300s"

retries

The maximum number of retry attempts to make.

Type: number
Default: 3

backoff_on

A list of status codes whereby retries should be attempted but the period between them should be increased gradually.

Type: array
Default: [429]

drop_on

A list of status codes whereby the attempt should be considered failed but retries should not be attempted.

Type: array
Default: []

successful_on

A list of status codes whereby the attempt should be considered successful (allows you to configure non-2XX codes).

Type: array
Default: []

propagate_response

Whether responses from the server should be propagated back to the input.

Type: bool
Default: false

max_in_flight

The maximum number of messages to have in flight at a given time. Increase this to improve throughput.

Type: number
Default: 1

batching

Allows you to configure a batching policy.

Type: object
Default: {"byte_size":0,"condition":{"static":false,"type":"static"},"count":1,"period":"","processors":[]}

# Examples
batching:
byte_size: 5000
period: 1s
batching:
count: 10
period: 1s
batching:
condition:
text:
arg: END BATCH
operator: contains
period: 1m

batching.count

A number of messages at which the batch should be flushed. If 0 disables count based batching.

Type: number
Default: 1

batching.byte_size

An amount of bytes at which the batch should be flushed. If 0 disables size based batching.

Type: number
Default: 0

batching.period

A period in which an incomplete batch should be flushed regardless of its size.

Type: string
Default: ""

# Examples
period: 1s
period: 1m
period: 500ms

batching.condition

A condition to test against each message entering the batch, if this condition resolves to true then the batch is flushed.

Type: object
Default: {"static":false,"type":"static"}

batching.processors

A list of processors to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.

Type: array
Default: []

# Examples
processors:
- archive:
format: lines
processors:
- archive:
format: json_array
processors:
- merge_json: {}