Blueprints

Load data from Postgres to BigQuery using Singer

Source

yaml
id: postgres-to-bigquery
namespace: company.team

tasks:
  - id: extract
    type: io.kestra.plugin.singer.taps.PipelinewisePostgres
    docker:
      image: python:3.8
    host: host.docker.internal
    port: 5432
    dbName: postgres
    username: postgres
    password: "{{ secret('DB_PASSWORD') }}"
    streamsConfigurations:
      - replicationMethod: FULL_TABLE
        selected: true

  - id: load
    type: io.kestra.plugin.singer.targets.AdswerveBigQuery
    addMetadataColumns: true
    datasetId: singer
    docker:
      image: python:3.8
    from: "{{ outputs.extract.raw }}"
    location: US
    projectId: yourProjectName
    serviceAccount: "{{ secret('GCP_CREDS') }}"

About this blueprint

Ingest Postgres BigQuery

This flow will extract raw data from a Postgres database and load it into BigQuery using the Singer protocol. The credentials to both Postgres and BigQuery can be provided as secrets.

Pipelinewise Postgres

Adswerve Big Query

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra