Idempotent API Poller with Dynamic KV Watermark

Poll external REST APIs incrementally without duplicating records across runs and retries. This blueprint demonstrates high-watermark cursor ingestion against the GitHub Commits REST API using Kestra's native HTTP and Key-Value (KV) store tasks.

How idempotency works here

Idempotency ensures that executing the flow multiple times against the same upstream state yields the exact same state without double-counting records:

• Safe first run: If no high-watermark exists in the KV store, the flow safely falls back to a relative lookback window (now() minus initial_lookback) instead of failing.
• Boundary offset (+1 second): The GitHub Commits API since query parameter is inclusive (committer.date >= since). Advancing the high-watermark to maximum_committer_date + 1 second guarantees the boundary commit is never fetched again on subsequent polls.
• Post-commit state persistence: The new watermark is persisted to the KV store only after records are fetched and validated, never before. Any downstream data loading tasks must be placed before advance_watermark so the cursor advances only when records have committed. If an API request or downstream task fails, the watermark remains unchanged so the next run retries the exact same window.
• Immediate rerun safety: Triggering the workflow twice in rapid succession results in zero new records fetched on the second run, leaving the KV store intact.

How it works

1. read_watermark (io.kestra.plugin.core.kv.Get) retrieves the current high-watermark key (watermark_{owner}_{repository}) from the namespace KV store. The errorOnMissing: false attribute allows clean cold-start execution.
2. compute_since (io.kestra.plugin.core.output.OutputValues) computes the query timestamp: the existing KV watermark if present, or now() - initial_lookback hours in ISO-8601 UTC format.
3. fetch_commits (io.kestra.plugin.core.http.Request) sends a GET request to https://api.github.com/repos/{owner}/{repo}/commits with the since query filter, GitHub headers, timeout, and automatic 3-attempt retry.
4. process_records (io.kestra.plugin.core.flow.If) evaluates the response array:
   • Records found: If 100 or more records were returned, check_page_saturation halts the execution with an io.kestra.plugin.core.execution.Fail error so the watermark is not advanced over unread older commits. Otherwise, log_commit_summary logs the count and min/max timestamps, log_sample_commits outputs the latest 10 commits (SHA, date, message headline), and advance_watermark (io.kestra.plugin.core.kv.Set) saves max_committer_date + 1s back to the KV store. Any downstream ingestion tasks belong immediately before advance_watermark.
   • No records: log_no_new_records logs that no new records arrived since the watermark, and the KV store is left completely untouched.

Pagination and data integrity

The GitHub Commits API limits single-page responses to at most 100 commits (per_page=100) and returns commits in reverse-chronological order (newest first).

• If records_count < 100, the result contains all commits within the polling window, guaranteeing that no commits are pending on subsequent pages.
• If records_count == 100, the single-page limit has been reached. Because Git history traverses newest-to-oldest, older commits remain on subsequent pages. Advancing the watermark to the newest timestamp would permanently skip those older commits. To protect data integrity, the flow explicitly halts with an io.kestra.plugin.core.execution.Fail error and leaves the watermark untouched.
• Once saturated, the flow will fail on every subsequent run until recovered. Operators can resolve this by:
  1. Setting a schedule interval frequent enough that commit velocity per window stays below 100.
  2. Manually updating the KV watermark key watermark_{owner}_{repository} in the Kestra UI to a newer timestamp, acknowledging that the unread commits in that backlog will not be ingested.
  3. Extending the blueprint with a multi-page fetch loop.

Concurrency control

The flow enforces concurrency: { limit: 1, behavior: QUEUE }. This serializes overlapping runs (such as a scheduled execution running while a manual run is active), preventing race conditions where two executions read the same watermark and produce duplicate downstream events.

Limitations

• Branch coverage: Only the default repository branch (or the branch resolved by GitHub's API) is polled. Commits on secondary branches are not tracked until merged into the tracked branch.
• Committer date ordering: The GitHub Commits API since parameter filters on committer.date. If older commits are merged via fast-forward or rebase while preserving historical committer dates older than the current watermark, they will not be returned by since.
• Single-page boundary: The poller fetches up to 100 commits per cycle. If more than 100 commits occur within one cycle, the flow halts to avoid data skipping and requires manual recovery or interval adjustment as described above.

What you get

• Incremental REST API poller operating with zero scripts or external dependencies.
• Resilient, crash-safe watermark state tracked in Kestra's built-in KV store.
• Guard against data loss on saturated API pages.
• Full execution observability with commit summaries and structured sample logs.
• Protection against boundary overlap and race conditions.

Who it's for

• Data engineers building incremental ELT/ETL ingestion pipelines from REST APIs.
• Platform teams implementing audit watchers or change tracking without full CDC infrastructure.
• Anyone looking for a robust, scriptless state-tracking pattern in Kestra.

Why orchestrate this with Kestra

Kestra's integrated KV store eliminates the need for external state databases (like Redis or Postgres) simply to store pagination cursors. Combining native HTTP retries, expression-based date manipulation, concurrency queueing, and conditional branching allows production-grade ingestion using purely declarative YAML.

Prerequisites

• A running Kestra instance with the Core plugin (KV store and HTTP tasks are included out of the box).
• Outbound internet access to api.github.com.

Secrets

• No secrets or credentials are required. The GitHub Commits API allows up to 60 unauthenticated requests per hour per IP.
• Optional: For repositories requiring authentication or higher rate limits (5,000 requests/hour), add an Authorization header to fetch_commits:

  headers:
    Authorization: "Bearer {{ secret('GITHUB_TOKEN') }}"
  

Quick start

1. Import this flow into your Kestra instance and save it.
2. Click Execute with the default inputs (owner: kestra-io, repository: kestra, initial_lookback: 6).
3. View the execution logs: log_commit_summary reports the commits found within the last 6 hours, and advance_watermark saves the new high-watermark to the KV store. If the first run finds no commits, no watermark is created; increase initial_lookback and run again.
4. Click Execute again immediately: the flow queries with the newly saved watermark, reports No new commits found..., and leaves the watermark untouched.
5. Enable the poll_schedule trigger to run automated polling every 15 minutes.

How to reset

To clear the state and re-ingest from scratch:

• In the Kestra UI, navigate to KV Store under the company.team namespace (or Namespaces > company.team > KV Store), locate the key watermark_kestra-io_kestra, and click Delete.
• Next execution will automatically fall back to the initial_lookback window.

Links

• GitHub REST API - List commits
• Kestra KV Store documentation
• Kestra HTTP Request task