Source
yaml
id: postgres-to-bigquery
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.singer.taps.PipelinewisePostgres
docker:
image: python:3.8
host: host.docker.internal
port: 5432
dbName: postgres
username: postgres
password: "{{ secret('DB_PASSWORD') }}"
streamsConfigurations:
- replicationMethod: FULL_TABLE
selected: true
- id: load
type: io.kestra.plugin.singer.targets.AdswerveBigQuery
addMetadataColumns: true
datasetId: singer
docker:
image: python:3.8
from: "{{ outputs.extract.raw }}"
location: US
projectId: yourProjectName
serviceAccount: "{{ secret('GCP_CREDS') }}"
About this blueprint
Ingest Postgres BigQuery
This flow will extract raw data from a Postgres database and load it into BigQuery using the Singer protocol. The credentials to both Postgres and BigQuery can be provided as secrets.