Blueprints

Sync Hubspot CRM data to BigQuery on a schedule

About this blueprint

API BigQuery Ingest

This flow will sync data from Hubspot CRM to BigQuery on a schedule. The sync task from the CloudQuery plugin uses the hubspot source and the bigquery destination.

Note how we use the sa.json credentials file to authenticate with GCP and the HUBSPOT_APP_TOKEN environment variable to authenticate with Hubspot CRM.

To avoid rate limiting issues, you can set the max_requests_per_second parameter in the hubspot source configuration. In this example, we set it to 5 requests per second.

The schedule trigger runs the flow every day at 6:00 AM.

Additionally, you can generate an API key to use premium plugins. You can add the API key as an environment variable:

yaml
  - id: hn_to_duckdb
    type: io.kestra.plugin.cloudquery.Sync
    env:
      CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
yaml
id: hubspot_to_bigquery
namespace: blueprints

tasks:
  - id: sync
    type: io.kestra.plugin.cloudquery.Sync
    inputFiles:
      sa.json: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
    env:
      HUBSPOT_APP_TOKEN: "{{ secret('HUBSPOT_API_TOKEN') }}"
      GOOGLE_APPLICATION_CREDENTIALS: sa.json
    configs:
      - kind: destination
        spec:
          name: bigquery
          path: cloudquery/bigquery
          registry: cloudquery
          version: "v3.3.16"
          write_mode: "append"
          spec:
            project_id: kestra-prd
            dataset_id: hubspot
      - kind: source
        spec:
          name: hubspot
          path: cloudquery/hubspot
          registry: cloudquery
          version: "v3.0.18"
          destinations: ["bigquery"]
          tables: ["*"]
          spec:
            max_requests_per_second: 5

triggers:
  - id: schedule
    type: io.kestra.core.models.triggers.types.Schedule
    cron: 0 6 * * *

Sync

Schedule

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra