Apify runs the actor and drops the result in a dataset. Then what? Kestra waits for the run to finish, pulls the items, checks them, and loads them into your warehouse, vector store, or CRM. Cost caps and an automated quality gate stop a runaway crawl or a broken selector before it costs you money or ships bad data.
Apify runs the actor and lands the result in a dataset; Kestra turns that into a finished pipeline. io.kestra.plugin.apify.actor.Run waits for the run, surfaces the dataset ID and cost, and dataset.Get reads the items straight into the next task. From there one flow loads them into BigQuery or Snowflake, transforms them with dbt, embeds them into a vector store for RAG, or pushes them to a CRM, with cost caps and a quality gate so a runaway crawl or a broken selector never reaches production.
Apify runs the actor, the proxies, and the Store. Kestra runs the steps around the scrape: what waits for it, what validates it, what it feeds, and where the audit trail lives.
When an Apify run finishes, the data is sitting in a dataset and the next move is yours: fetch it, validate it, route it onward. Kestra's actor.Run blocks until the run finishes, then exposes defaultDatasetId, status, and usageTotalUsd as outputs. dataset.Get reads the items into the next task, so there is no receiver to host and no polling script to maintain.
Apify actors bill per compute unit or per result, so a misconfigured crawl is a real bill. The Run task sets maxItems, maxTotalChargeUsd, memory, and requestTimeout as native run caps, and the run returns usageTotalUsd. An automated If can branch on actual cost or item count and halt the pipeline before you pay to process garbage, with no human in the loop.
Apify stops at the dataset. Kestra chains the scrape to a dbt transform and a BigQuery, Snowflake, or Postgres load, to embeddings for a vector store, or to a HubSpot upsert. Step retries stay local: a load retry never re-runs the scrape, and the actor run ID flows forward into every downstream task for traceability.
Scrapers degrade silently when a site changes its markup: empty datasets, half-null rows, a selector that stopped matching, and the run still reports success. Kestra runs an automated check on the item count and null ratio with io.kestra.plugin.core.flow.If and halts before the data reaches a warehouse or a CRM, paging the owner instead of poisoning a table downstream.
Scraping fifty competitor URLs or fifty search terms in a single actor run gives you one blob and one failure mode. Kestra's ForEach runs many Run calls concurrently with a concurrencyLimit, and allowFailure isolates a blocked target so a single 403 or CAPTCHA wall does not sink the batch. Each target's items land tagged with their source.
Growth, research, and sales teams should not file a ticket to run a one-off scrape. Kestra Apps wraps a scrape flow in a typed form: enter a URL or a search term, run the actor, get the dataset. Every run lands in execution history with the requester's identity, so self-service does not mean unaudited.
Patterns data and AI teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
A daily actor scrapes places, listings, or prices, and Kestra lands the rows in BigQuery, Snowflake, or Postgres. Run caps the cost, dataset.Get pulls the items, a reshape step types the columns, and a load retry never re-runs the scrape.
maxItems and maxTotalChargeUsd bound the bill before the crawl starts.
A Postgres or warehouse retry never re-triggers the Apify actor.
Point the same flow at BigQuery, Snowflake, or DuckDB by changing the load task.
Apify's Website Content Crawler returns clean, boilerplate-free text, the ideal input for retrieval. Kestra crawls on a schedule, embeds it into a vector store with IngestDocument, and drops the previous index so answers never go stale. An AIAgent step then answers grounded questions from the fresh crawl.
The crawler strips navs and footers so embeddings index content, not boilerplate.
drop: true rebuilds the index each run so retrieval reflects the latest crawl.
Start on the built-in KV store, then switch to Pinecone, Qdrant, or PGVector for scale.
Scrapes break quietly when a site ships new markup. Kestra runs the actor under hard cost caps, then an automated If checks the item count. A healthy count promotes the data; a count below the threshold pages the on-call via Slack or PagerDuty, so a broken selector never poisons a table.
If branches on item count or null ratio; nobody has to watch a dashboard.
maxTotalChargeUsd caps spend even when a crawl misbehaves.
An empty-but-successful run still alerts, because empty data is the real risk.
Competitive monitoring means scraping dozens of sites or search terms on one schedule. Kestra's ForEach runs the actor across every target with a bounded concurrencyLimit, and allowFailure keeps one blocked target from sinking the batch. Only changed targets trigger an alert.
concurrencyLimit respects Apify and target rate limits while still running in parallel.
allowFailure per target means a 403 on one site does not fail the run.
An If compares against the last run and notifies only when something moved.
The fastest payback for scraping is lead generation. Kestra runs a Google Maps or contact-details actor, dedups and validates the rows, then upserts them into HubSpot with io.kestra.plugin.hubspot.contacts.Create. Slack posts the count of new contacts so sales sees the pipeline fill in real time.
A transform step drops duplicates so HubSpot does not fill with repeats.
Search-then-update keeps existing contacts clean on re-runs.
Slack posts the count of new and updated contacts after every run.
| Capability | ||||
|---|---|---|---|---|
| Wait for an actor, then fetch its dataset | Run blocks, dataset.Get reads | Sensor or custom operator | Custom op or sensor | HTTP node plus manual polling |
| Cost caps enforced on the run | maxItems + maxTotalChargeUsd | Custom code | Custom code | No native cost cap |
| Automated quality gate before downstream | Native If on item count | BranchPythonOperator | Asset checks, Python | IF node, limited |
| Chain to dbt, BigQuery, Snowflake, CRM | Native plugins | Operator per tool, Python glue | Integrations, mostly Python | Prebuilt nodes, shallow |
| Parallel fan-out with per-target isolation | ForEach + allowFailure | Dynamic task mapping, Python | Dynamic partitions, Python | Loop node, no isolation |
| Self-service scrape forms | Kestra Apps | No native form layer | No native form layer | Forms in paid tiers |
| Declarative YAML, IaC-friendly (Terraform) | Pure YAML + Terraform provider | Python DAGs | Python assets | Visual JSON, limited IaC |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Run an Apify actor and update Notion
A ready-to-run flow: scrape with an actor, fetch the dataset, and write the results to a Notion page.
Build a RAG pipeline and answer questions
Ingest documents into a vector store and answer grounded questions, the AI pattern Apify crawls feed.
Kestra quickstart
Start Kestra locally with Docker and run your first flow in under a minute.
Wait for the actor, enforce cost and quality guardrails, and chain the data into your warehouse, vector store, or CRM in one flow. Open source, self-hosted, event-driven.
