Blueprints

Use Postgres CopyIn to ingest CSV data into Postgres table

Source

yaml
id: postgres-copyin
namespace: company.team

tasks:
  - id: download_products
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv

  - id: create_products_table
    type: io.kestra.plugin.jdbc.postgresql.Query
    url: "jdbc:postgresql://{{ secret('POSTGRES_HOST') }}:5432/postgres"
    username: "{{ secret('POSTGRES_USERNAME') }}"
    password: "{{ secret('POSTGRES_PASSWORD') }}"
    sql: |
      CREATE TABLE IF NOT EXISTS products(
        product_id varchar(5),
        product_name varchar(100),
        product_category varchar(50),
        brand varchar(50)
      );

  - id: copyin_products
    type: io.kestra.plugin.jdbc.postgresql.CopyIn
    url: "jdbc:postgresql://{{ secret('POSTGRES_HOST') }}:5432/postgres"
    username: "{{ secret('POSTGRES_USERNAME') }}"
    password: "{{ secret('POSTGRES_PASSWORD') }}"
    format: CSV
    from: "{{ outputs.download_products.uri }}"
    table: products
    header: true
    delimiter: ","

About this blueprint

Postgres

This flow demonstrates using Postgres CopyIn task to ingest data from CSV file into Postgres table. The flow has the following tasks: 1. download_products: This task downloads the CSV file containing products data using HTTP download. 2. create_products_table: This task creates the products table in Postgres if it does not exists. 3. copyin_products: This task copies the data from CSV file dowloaded in the first task into the Postgres table using the Postgres CopyIn task.

Download

Query

Copy In

More Related Blueprints

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra