Apify Get
CertifiedFetch Apify dataset items
Apify Get
Fetch Apify dataset items
Polls the Apify dataset items API until data is available or the 300s timeout elapses, using exponential backoff. Empty responses are retried because the actor may still be writing items.
type: io.kestra.plugin.apify.dataset.GetExamples
Get dataset with a given id.
id: apify_get_dataset_flow_required_properties
namespace: company.team
tasks:
- id: list_runs
type: io.kestra.plugin.apify.dataset.Get
apiToken: "{{ secret('APIFY_API_TOKEN') }}"
datasetId: mecGriFjtDHRNtYOZ
Get dataset with a given id and specific options.
id: apify_get_dataset_flow
namespace: company.team
tasks:
- id: list_runs
type: io.kestra.plugin.apify.dataset.Get
apiToken: "{{ secret('APIFY_API_TOKEN') }}"
datasetId: RNtYOZmecGriFjtDH
clean: false
offset: 1
limit: 10
fields:
- userId
- "#id"
- "#createdAt"
- postMeta
omit:
- "#id"
flatten: true
sort: ASC
skipEmpty: false
Properties
apiToken *string
Apify API token
Personal Apify API token used for all requests; required.
datasetId *string
Dataset ID
Apify dataset identifier to read items from.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150clean booleanstring
trueClean items
Skip empty records and fields prefixed with # when true; default true.
fields array
Fields
Comma-separated fields to keep in each item; others are dropped.
flatten booleanstring
falseFlatten fields
Fields to flatten into dotted keys (foo.bar) instead of nested objects.
limit integerstring
1000Limit
Maximum items to return; defaults to 1000.
offset integerstring
0Offset
Number of leading items to skip; default 0.
omit array
Omit
Comma-separated fields to remove from each item.
options
HTTP client options
Optional HttpConfiguration applied to every Apify call. SDK-backed tasks honour timeout.readIdleTimeout only, the raw HTTP tasks honour the full configuration.
io.kestra.core.http.client.configurations.HttpConfiguration
falseIf true, allow a failed response code (response code >= 400)
List of response code allowed for this request
The authentication to use.
io.kestra.core.http.client.configurations.BasicAuthConfiguration
The password for HTTP basic authentication.
The username for HTTP basic authentication.
io.kestra.core.http.client.configurations.BearerAuthConfiguration
The token for bearer token authentication.
io.kestra.core.http.client.configurations.DigestAuthConfiguration
The password for HTTP Digest authentication.
The username for HTTP Digest authentication.
UTF-8The default charset for the request.
java.nio.charset.Charset
trueWhether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).
Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.
trueWhether redirects should be followed automatically.
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODYThe enabled log.
The proxy configuration.
io.kestra.core.http.client.configurations.ProxyConfiguration
The address of the proxy server.
The password for proxy authentication.
The port of the proxy server.
DIRECTDIRECTHTTPSOCKSThe type of proxy to use.
The username for proxy authentication.
The SSL request options
io.kestra.core.http.client.configurations.SslOptions
Whether to disable checking of the remote SSL certificate.
Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.
The timeout configuration.
io.kestra.core.http.client.configurations.TimeoutConfiguration
The time allowed to establish a connection to the server before failing.
PT5MThe time allowed for a read connection to remain idle before closing it.
simplified booleanstring
falseSimplified output
Enable Apify simplified output mode; default false.
skipEmpty booleanstring
trueSkip empty items
Drop empty records when true; default true.
skipFailedPages booleanstring
falseSkip failed pages
Skip items containing errorInfo when true; default false.
skipHidden booleanstring
falseSkip hidden
Skip fields prefixed with # when true; default false.
sort string
ASCASCDESCSort direction
Set to DESC to return newest items first; default ASC.
unwind array
Unwind fields
Fields to unwind in order; array elements become separate records, objects merge into parents. Unwound items ignore the sortDirection flag.
view string
View
Dataset view name to filter and project items per the Apify schema.
Outputs
dataset array
Items fetched from the Apify dataset
