Firecrawl Scrape
CertifiedScrape a web page with Firecrawl
Firecrawl Scrape
Scrape a web page with Firecrawl
Scrapes a single URL through the Firecrawl API and returns clean, LLM-ready content such as markdown. The full document is stored as an ION file in the internal storage; the markdown is also returned inline when it is small enough.
type: io.kestra.plugin.firecrawl.ScrapeExamples
Scrape a page to markdown and log it.
id: firecrawl_scrape
namespace: company.team
tasks:
- id: scrape
type: io.kestra.plugin.firecrawl.Scrape
apiKey: "{{ secret('FIRECRAWL_API_KEY') }}"
url: https://example.com
- id: log_markdown
type: io.kestra.plugin.core.log.Log
message: "{{ outputs.scrape.markdown }}"
Scrape only the main content of a page as markdown and links.
id: firecrawl_scrape_main_content
namespace: company.team
tasks:
- id: scrape
type: io.kestra.plugin.firecrawl.Scrape
apiKey: "{{ secret('FIRECRAWL_API_KEY') }}"
url: https://example.com/blog
formats:
- markdown
- links
onlyMainContent: true
waitFor: PT2S
Properties
apiKey *string
Firecrawl API key
API key sent as a bearer token on every request; store it as a Kestra secret.
url *string
URL to scrape
Absolute URL of the page to scrape.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150baseUrl string
Firecrawl API base URL
Base URL of the Firecrawl API without the version suffix. Defaults to https://api.firecrawl.dev; set it to target a self-hosted Firecrawl instance.
excludeTags array
HTML tags to exclude
CSS selectors or tags that must be removed from the scraped content.
formats array
["markdown"]Output formats
Formats requested for each page, for example markdown, html, rawHtml, links or summary. Defaults to markdown.
includeTags array
HTML tags to include
CSS selectors or tags that must be kept in the scraped content.
onlyMainContent booleanstring
Only main content
When true, Firecrawl strips headers, navigation and footers and keeps only the main content of each page.
requestTimeout string
Per-page request timeout
Maximum time Firecrawl spends scraping a single page. This is distinct from the task-level timeout.
waitFor string
Wait before scraping
Time to wait for dynamic content to load before each page is scraped.
Outputs
markdown string
Markdown content
Markdown of the page, only returned when markdown was requested and the content is under 100,000 characters; otherwise read it from uri.
statusCode integer
HTTP status code of the scraped page
title string
Page title
uri string
uriScraped document URI
ION file in the internal storage containing one row with every requested format and the page metadata.
