hdfs

Sends message parts as files to a HDFS directory.

output:
hdfs:
hosts:
- localhost:9000
user: benthos_hdfs
directory: ""
path: ${!count:files}-${!timestamp_unix_nano}.txt
max_in_flight: 1

Each file is written with the path specified with the 'path' field, in order to have a different path for each object you should use function interpolations described here. When sending batched messages the interpolations are performed per message part.

Performance

This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages with the field max_in_flight.

Fields

hosts

string A list of hosts to connect to.

# Examples
hosts: localhost:9000

user

string A user identifier.

directory

string A directory to store message files within. If the directory does not exist it will be created.

path

string The path to upload messages as, interpolation functions should be used in order to generate unique file paths.

This field supports interpolation functions.

# Examples
path: ${!count:files}-${!timestamp_unix_nano}.txt

max_in_flight

number The maximum number of messages to have in flight at a given time. Increase this to improve throughput.