Schedule icon
Request icon
OutputValues icon
Log icon
If icon
Fail icon
SlackIncomingWebhook icon
Pause icon
Loop icon
TriggerRun icon

Backfill missing Dagster asset partitions safely

Reconcile Dagster asset partitions in Kestra, backfill approved gaps one at a time and verify coverage through GraphQL.

Categories
Data

Safety

An upstream outage can leave a partitioned Dagster asset looking healthy while downstream models silently read incomplete history. This flow makes the target set explicit, keeps protected keys out of the write plan, asks for approval and verifies the asset after every remediation pass.

Guard What it stops
Exact asset_key GraphQL selector A similarly named asset being treated as the target
protected_partition_keys filter A known exception being launched accidentally
dry_run and auto_remediate gates A schedule making changes by default
Pause approval An unattended backfill starting during an incident
GraphQL re-read after approval Launching a stale plan after a partition recovered or disappeared
Loop with concurrencyLimit: 1 Concurrent Dagster runs competing for the same upstream resources
Final materialization query and Fail Reporting success while a gap remains

How it works

  1. read_status (io.kestra.plugin.core.http.Request) asks Dagster GraphQL for materialization events of the exact asset_key.
  2. build_plan (io.kestra.plugin.core.output.OutputValues) compares those partitions with partition_keys, records protected keys and lists the gaps.
  3. approval (io.kestra.plugin.core.flow.Pause) is reached only when auto_remediate is true and dry_run is false. The schedule is shipped disabled and always supplies safe inputs.
  4. reread_after_approval (io.kestra.plugin.core.http.Request) refreshes the plan. A recovered missing key is skipped. A previously materialized key that disappeared fails the run before any write.
  5. launch_one_partition (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) invokes io.kestra.plugin.dagster.TriggerRun with one partition in runConfig and waits for that run to finish.
  6. verify_status (io.kestra.plugin.core.http.Request) reads the asset again. fail_if_gap_remains (io.kestra.plugin.core.execution.Fail) makes an ineffective or partial backfill a failed execution.
  7. alert_flow_failure sends a Slack alert from the errors block whenever the flow cannot prove recovery.

What you get

  • A dry-run inventory of exact missing partitions before any Dagster run starts.
  • Approval-gated sequential backfills with a fresh eligibility read.
  • A failure when a run refuses to materialize its partition or only partially repairs the set.
  • A repeatable reconciliation: a second run sees zero gaps and launches nothing.

Who it's for

  • Platform teams operating Dagster assets whose downstream warehouse models require complete history.
  • Data engineers responding to a failed daily ingestion or asset materialization window.
  • On-call teams that need an auditable approval and a proof of recovery rather than a blind retry.

Why orchestrate this with Kestra

Dagster records materializations and can launch jobs, but the incident response still needs a bounded target list, an approval decision, serial execution and a post-change control. Kestra keeps those reads, the TriggerRun writes, the pause, alerts and the final failure decision in one execution with durable history.

Tested end to end

End-to-end tests use Dagster 1.11.16 with its webserver and daemon plus Kestra 2.0.5. The partition_backfill_job emits a real AssetMaterialization event for each configured partition. The evidence packet records healthy, dry-run, declined approval, remediation, repeat, ineffective write and approval-race executions with independent GraphQL assertions.

Case Execution Result
Healthy control cHQO0R66C6uKyupm8vxAc SUCCESS, 0 gaps across 4 requested partitions, no run launched
Degraded dry run 6eLfqDyWLfIMuCKFUMrKSv SUCCESS, 1 gap listed, no write
Protected key 54mk9Z9JBTrOFCP9BDaclT SUCCESS, 1 protected key spared
Declined approval 3GW05RmfjfTiWXT4ljrDNP SUCCESS, 1 gap remained and no run launched
Remediation 12ZndKeNj95uc0qziG3uOL SUCCESS, 2 gaps to 0, 2 sequential Dagster runs
Repeat 1jqi9QnHDHHSXelJU6TlS8 SUCCESS, 0 gaps and no duplicate run
Ineffective write 1mPLvqiGoIxQsuBETMJbMu FAILED, noop job left 2027-02-16 missing
Approval race 4z0fQjlJGLvF4XS0cn6mCe SUCCESS, external 2027-02-17 fill skipped and 2027-02-18 repaired
Lookalike control 26yYfmpHNRiaIpx3CgWq7o SUCCESS, exact target gap stayed visible despite daily_orders_backup materialization

Prerequisites

  • Kestra 2.0.5 with the Dagster plugin and a worker that can reach the Dagster GraphQL endpoint.
  • A Dagster job that accepts one partition key in its run configuration and emits a materialization for that key.
  • A service identity allowed to launch that job and read asset materializations.

Secrets

  • DAGSTER_GRAPHQL_URL: Dagster GraphQL endpoint used as the default for dagster_base_url.
  • SLACK_WEBHOOK_URL: Slack Incoming Webhook used by findings and flow-failure alerts.

Inputs

Input Type Purpose
dagster_base_url STRING Dagster GraphQL endpoint
job_name, repository, location STRING Dagster launch selector
asset_key STRING Exact asset to inspect
partition_keys ARRAY Bounded expected coverage window
protected_partition_keys ARRAY Keys reported but never launched
partition_mode STRING Optional job-specific run configuration
auto_remediate BOOL Arm approval and writes
dry_run BOOL Keep true to make no change

Quick start

  1. Configure the Dagster URL, launch selector, asset key and expected partition keys.
  2. Run with auto_remediate: false and dry_run: true; confirm the gap list matches Dagster's UI or GraphQL response.
  3. Set auto_remediate: true and dry_run: false, resume the approval task with approved: true and review the sequential TriggerRun tasks.
  4. Confirm the final read reports zero unprotected gaps before enabling the disabled schedule.

How to extend

  • Replace the bounded partition_keys input with a list generated by the upstream calendar while retaining a maximum window size.
  • Add a second independent GraphQL assertion for downstream model freshness before declaring the incident closed.
  • Add a per-asset policy map when several assets need different protected keys or launch selectors.

Limits: the flow reads at most 100 recent materialization events from Dagster and expects the requested window to fit that bound. It launches one job per partition and does not roll back a partition that was already materialized. A job that reports success without emitting the requested materialization is intentionally treated as an ineffective write and fails the execution.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.