Synchronize GitHub Issues to Analytical DuckDB Table

How it works

1. Schedule - the hourly-sync-schedule trigger fires at the top of every hour (0 * * * *).
2. Fetch issues - fetch-github-issues calls the GitHub REST API and retrieves the 10 most recent open issues of kestra-io/kestra.
3. Status check - evaluate-sync-status (an If task) continues only when the API answers with HTTP 200.
4. Persist audit record - index-database-batch runs a DuckDB query that stores the sync timestamp, the HTTP status code, and the pipeline status in the github_sync_audit table.
5. Log confirmation - log-sync-success confirms the sync. If the status is not 200, log-sync-warning is logged instead and the database step is skipped.

What you get

• Automated ingestion: a recurring GitHub API call without any external cron service.
• Safe execution checks: the database step runs only when the upstream API returns 200 OK.
• Embedded storage: an in-process DuckDB table for auditing and lightweight analytics, with no database server to manage.
• Clear logs: a success or warning message for every execution.

Who it's for

• Engineering teams tracking issue volume and repository activity.
• Data engineers building lightweight embedded analytical pipelines.
• DevOps engineers who want to orchestrate API-to-database workflows.

Why orchestrate this with Kestra

• Declarative scheduling: the cron trigger manages the cadence without a third-party scheduler.
• Zero-infrastructure analytics: the DuckDB plugin runs embedded inside the task runner.
• Observability: every run has task-level logs, durations, and outputs in the Kestra UI, so failures are easy to trace.

Prerequisites

• A running Kestra instance with outbound HTTPS access to https://api.github.com.

Secrets

No secrets are required: the flow reads public issues from the unauthenticated GitHub API, which allows 60 requests per hour per IP address. For private repositories or higher rate limits, create a secret and send it in an Authorization header:

• GITHUB_TOKEN: a GitHub personal access token with read access to the repository.

Quick start

1. Import this blueprint into your Kestra instance.
2. Change the uri of fetch-github-issues to the repository you want to track.
3. Click Execute to run it once and check the Gantt and Logs tabs.
4. Leave the flow enabled and the hourly-sync-schedule trigger will take over from the next full hour.

How to extend

• Keep history between runs: the default jdbc:duckdb: URL is an in-memory database that lives for a single execution. Point url to a DuckDB file or a MotherDuck database to retain data across runs.
• Load the issues themselves: store the response body and read it in DuckDB with read_json_auto() for full issue analytics.
• Alert on problems: add a Slack or email notification in the else branch or in an errors block.
• Export results: push aggregated metrics to object storage such as S3 or Google Cloud Storage.

Links

• Kestra HTTP Request task
• Kestra DuckDB Query task
• GitHub REST API - List repository issues