Firecrawl Crawl

Firecrawl Crawl

Certified

Crawl a website with Firecrawl

Starts a Firecrawl crawl from a URL, waits for it to finish and stores every crawled page as one row of an ION file in the internal storage. Firecrawl keeps crawl results for a limited time, so they are downloaded as soon as the crawl completes. If the task is killed, times out or fails while waiting, the crawl is cancelled on the Firecrawl side so it stops consuming credits.

yaml
type: io.kestra.plugin.firecrawl.Crawl

Crawl the documentation pages of a site to markdown.

yaml
id: firecrawl_crawl_docs
namespace: company.team

tasks:
  - id: crawl_docs
    type: io.kestra.plugin.firecrawl.Crawl
    apiKey: "{{ secret('FIRECRAWL_API_KEY') }}"
    url: https://example.com/docs
    limit: 50
    includePaths:
      - /docs/.*

  - id: log_count
    type: io.kestra.plugin.core.log.Log
    message: "Crawled {{ outputs.crawl_docs.pagesCount }} pages into {{ outputs.crawl_docs.uri }}"

Crawl a site with a slower polling frequency and a maximum duration.

yaml
id: firecrawl_crawl_with_timeout
namespace: company.team

tasks:
  - id: crawl_site
    type: io.kestra.plugin.firecrawl.Crawl
    apiKey: "{{ secret('FIRECRAWL_API_KEY') }}"
    url: https://example.com
    limit: 200
    maxDiscoveryDepth: 3
    onlyMainContent: true
    pollFrequency: PT10S
    maxDuration: PT30M
Properties

Firecrawl API key

API key sent as a bearer token on every request; store it as a Kestra secret.

Start URL

Absolute URL the crawl starts from.

Allow subdomains

When true, the crawl follows links to subdomains of the start URL.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Firecrawl API base URL

Base URL of the Firecrawl API without the version suffix. Defaults to https://api.firecrawl.dev; set it to target a self-hosted Firecrawl instance.

Crawl the entire domain

When true, the crawl can follow links to sibling and parent paths instead of only deeper paths.

SubTypestring

Paths to exclude

Regular expressions matched against URL paths; matching pages are skipped.

SubTypestring

HTML tags to exclude

CSS selectors or tags that must be removed from the scraped content.

SubTypestring
Default["markdown"]

Output formats

Formats requested for each page, for example markdown, html, rawHtml, links or summary. Defaults to markdown.

SubTypestring

Paths to include

Regular expressions matched against URL paths; only matching pages are crawled.

SubTypestring

HTML tags to include

CSS selectors or tags that must be kept in the scraped content.

Default100

Page limit

Maximum number of pages to crawl, at least 1. Defaults to 100.

Maximum discovery depth

Maximum number of link hops from the start URL, at least 0.

DefaultPT1H

Maximum wait duration

Maximum total time to wait for the crawl to finish, must be positive. When it is exceeded the crawl is cancelled on the Firecrawl side and the task fails. Defaults to 60 minutes.

Only main content

When true, Firecrawl strips headers, navigation and footers and keeps only the main content of each page.

DefaultPT5S

Poll frequency

Interval between two status checks of the crawl, at least 1 second. Each status check returns the pages crawled so far, so keep it moderate for large crawls. Defaults to 5 seconds.

Per-page request timeout

Maximum time Firecrawl spends scraping a single page. This is distinct from the task-level timeout.

Wait before scraping

Time to wait for dynamic content to load before each page is scraped.

Firecrawl crawl ID

Credits used by the crawl

Number of crawled pages

Final crawl status

Formaturi

Crawled pages URI

ION file in the internal storage containing one row per crawled page.

Unitcredit

Firecrawl credits used by the crawl, when reported by the API

Unitpage

Number of pages crawled and stored

More Plugins in this Category

Ashby
Ashby

Ashby plugin for Kestra

Data

1 tasks

1 blueprints

SAP HANA
SAP HANA

Tasks that query SAP HANA using the official SAP HANA JDBC driver (ngdbc.jar).

Data

3 tasks

3 blueprints