Guard Apify scrapes with cost caps and an automated quality gate

Web scraping fails in two expensive ways: a runaway crawl that burns budget, and a silent degradation where a target site changes its DOM and the actor returns empty or half-null data that quietly poisons a warehouse. This blueprint wraps any Apify actor in automated cost and quality guardrails so neither failure ships unnoticed. It runs the actor under hard spend and run-time caps, checks the returned item count against a healthy threshold, promotes good data downstream, and pages the on-call through PagerDuty when a scrape degrades. No human sits in the loop.

How it works

1. A io.kestra.plugin.core.trigger.Schedule trigger fires the flow every six hours on cron 0 */6 * * *.
2. run_actor (io.kestra.plugin.apify.actor.Run) runs the actor named in the actor_id input under hard caps: maxItems, maxTotalChargeUsd, memory, and requestTimeout. Cost is bounded at the source.
3. get_dataset (io.kestra.plugin.apify.dataset.Get) fetches results from the run's defaultDatasetId and exposes the actor's reported usageTotalUsd.
4. quality_gate (io.kestra.plugin.core.flow.If) compares the returned item count against the min_items input. On a healthy count, promote (io.kestra.plugin.core.storage.Write) writes the dataset to internal storage as JSON and log_healthy logs the item count and cost.
5. Below the threshold, alert_degraded (io.kestra.plugin.pagerduty.PagerDutyAlert) pages the on-call with the actor, item count, and a likely cause (DOM change or block).

What you get

• Bounded scrape spend through actor-level cost and run caps.
• An automated deterministic quality gate, no human approval pause.
• Healthy datasets promoted to internal storage for downstream processing.
• Immediate PagerDuty alerts when a scrape degrades, before bad data spreads.

Who it's for

• Data engineers running production scrapes into warehouses or CRMs.
• Platform and on-call teams who need early warning on broken selectors.
• Teams managing Apify spend across many actors.

Why orchestrate this with Kestra

Apify's own scheduler can run an actor on a timer, but it cannot branch on the result, promote good data, and page a separate incident tool when quality drops. Kestra adds an event-driven Schedule trigger, conditional If branching on the item count, task retries, full execution lineage across the Apify, storage, and PagerDuty steps, and a declarative YAML definition you version in Git. The guardrail logic lives in your orchestration layer, not buried in an actor.

Prerequisites

• A Kestra instance.
• An Apify account with an actor to run.
• A PagerDuty integration with a routing key for alerts.

Secrets

• APIFY_API_TOKEN: Apify API token used by both the actor run and dataset fetch.
• PAGERDUTY_ROUTING_KEY: PagerDuty Events API routing key for the degraded-scrape alert.

Quick start

1. Add the APIFY_API_TOKEN and PAGERDUTY_ROUTING_KEY secrets to your Kestra instance.
2. Set the actor_id, start_url, and min_items inputs for your scrape.
3. Execute the flow manually to confirm the actor runs and the gate branches as expected.
4. Leave the six-hour schedule enabled for continuous guarded runs.

How to extend

• Tune min_items per actor to match each scrape's healthy baseline.
• Add a null-ratio or schema check to the If condition for stricter quality control.
• Swap the PagerDuty alert for Slack or email, or send both.
• Replace the storage Write with a load step into BigQuery, Snowflake, or S3.

Links

• Apify plugin documentation
• Apify orchestration with Kestra