gcp_bigquery_select
This component is experimental and therefore subject to change or removal outside of major version releases.
Executes a SELECT
query against BigQuery and creates a message for each row received.
Introduced in version 3.63.0.
# Config fields, showing default values
input:
label: ""
gcp_bigquery_select:
project: ""
table: ""
columns: []
where: ""
job_labels: {}
args_mapping: ""
prefix: ""
suffix: ""
Once the rows from the query are exhausted, this input shuts down, allowing the pipeline to gracefully terminate (or the next input in a sequence to execute).
Examples​
- Word counts
Here we query the public corpus of Shakespeare's works to generate a stream of the top 10 words that are 3 or more characters long:
input:
gcp_bigquery_select:
project: sample-project
table: bigquery-public-data.samples.shakespeare
columns:
- word
- sum(word_count) as total_count
where: length(word) >= ?
suffix: |
GROUP BY word
ORDER BY total_count DESC
LIMIT 10
args_mapping: |
root = [ 3 ]
Fields​
project
​
GCP project where the query job will execute.
Type: string
table
​
Fully-qualified BigQuery table name to query.
Type: string
# Examples
table: bigquery-public-data.samples.shakespeare
columns
​
A list of columns to query.
Type: array
where
​
An optional where clause to add. Placeholder arguments are populated with the args_mapping
field. Placeholders should always be question marks (?
).
Type: string
# Examples
where: type = ? and created_at > ?
where: user_id = ?
job_labels
​
A list of labels to add to the query job.
Type: object
Default: {}
args_mapping
​
An optional Bloblang mapping which should evaluate to an array of values matching in size to the number of placeholder arguments in the field where
.
Type: string
# Examples
args_mapping: root = [ "article", now().ts_format("2006-01-02") ]
prefix
​
An optional prefix to prepend to the select query (before SELECT).
Type: string
suffix
​
An optional suffix to append to the select query.
Type: string