About this blueprint
API BigQuery Ingest
This flow will sync data from Hubspot CRM to BigQuery on a schedule. The sync task from the CloudQuery plugin uses the hubspot source and the bigquery destination.
Note how we use the sa.json credentials file to authenticate with GCP and the HUBSPOT_APP_TOKEN environment variable to authenticate with Hubspot CRM.
To avoid rate limiting issues, you can set the max_requests_per_second parameter in the hubspot source configuration. In this example, we set it to 5 requests per second.
The schedule trigger runs the flow every day at 6:00 AM.
Additionally, you can generate an API key to use premium plugins. You can add the API key as an environment variable:
yaml
- id: hn_to_duckdb
type: io.kestra.plugin.cloudquery.Sync
env:
CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
yaml
id: hubspot_to_bigquery
namespace: blueprints
tasks:
- id: sync
type: io.kestra.plugin.cloudquery.Sync
inputFiles:
sa.json: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
env:
GOOGLE_APPLICATION_CREDENTIALS: sa.json
HUBSPOT_APP_TOKEN: "{{ secret('HUBSPOT_API_TOKEN') }}"
CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
configs:
- kind: destination
spec:
name: bigquery
path: cloudquery/bigquery
registry: cloudquery
version: "v3.3.16"
write_mode: "append"
spec:
project_id: kestra-prd
dataset_id: hubspot
- kind: source
spec:
name: hubspot
path: cloudquery/hubspot
registry: cloudquery
version: "v3.0.18"
destinations: ["bigquery"]
tables: ["*"]
spec:
max_requests_per_second: 5
triggers:
- id: schedule
type: io.kestra.core.models.triggers.types.Schedule
cron: 0 6 * * *