Apify Get

Apify Get

Certified

Fetch Apify dataset items

Polls the Apify dataset items API until data is available or the 300s timeout elapses, using exponential backoff. Empty responses are retried because the actor may still be writing items.

yaml
type: io.kestra.plugin.apify.dataset.Get

Get dataset with a given id.

yaml
id: apify_get_dataset_flow_required_properties
namespace: company.team

tasks:
  - id: list_runs
    type: io.kestra.plugin.apify.dataset.Get
    apiToken: "{{ secret('APIFY_API_TOKEN') }}"
    datasetId: mecGriFjtDHRNtYOZ

Get dataset with a given id and specific options.

yaml
id: apify_get_dataset_flow
namespace: company.team

tasks:
  - id: list_runs
    type: io.kestra.plugin.apify.dataset.Get
    apiToken: "{{ secret('APIFY_API_TOKEN') }}"
    datasetId: RNtYOZmecGriFjtDH
    clean: false
    offset: 1
    limit: 10
    fields: userId, #id, #createdAt, postMeta
    omit: #id
    flatten: postMeta
    sort: ASC
    skipEmpty: false
Properties

Apify API token

Personal Apify API token used for all requests; required.

Dataset ID

Apify dataset identifier to read items from.

Defaulttrue

Clean items

Skip empty records and fields prefixed with # when true; default true.

SubTypestring

Fields

Comma-separated fields to keep in each item; others are dropped.

Defaultfalse

Flatten fields

Fields to flatten into dotted keys (foo.bar) instead of nested objects.

Default1000

Limit

Maximum items to return; defaults to 1000.

Default0

Offset

Number of leading items to skip; default 0.

SubTypestring

Omit

Comma-separated fields to remove from each item.

HTTP client options

Optional HttpConfiguration applied to every Apify call, including timeouts, retries, and proxy settings.

Definitions
allowFailedbooleanstring
Defaultfalse

If true, allow a failed response code (response code >= 400)

allowedResponseCodesarray
SubTypeinteger

List of response code allowed for this request

auth

The authentication to use.

type*Requiredobject
passwordstring

The password for HTTP basic authentication.

usernamestring

The username for HTTP basic authentication.

type*Requiredobject
tokenstring

The token for bearer token authentication.

type*Requiredobject
passwordstring

The password for HTTP Digest authentication.

usernamestring

The username for HTTP Digest authentication.

basicAuthPasswordDeprecatedstring

The password for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.

basicAuthUserDeprecatedstring

The username for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.

connectTimeoutDeprecatedstring
Formatduration

The time allowed to establish a connection to the server before failing.

connectionPoolIdleTimeoutDeprecatedstring
Formatduration

The time an idle connection can remain in the client's connection pool before being closed.

defaultCharsetstring
DefaultUTF-8

The default charset for the request.

enabledTcpExtendedKeepAlivebooleanstring
Defaulttrue

Whether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).

Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.

followRedirectsbooleanstring
Defaulttrue

Whether redirects should be followed automatically.

logLevelDeprecatedstring
Possible Values
ALLTRACEDEBUGINFOWARNERROROFFNOT_SPECIFIED

The log level for the HTTP client.

logsarray
SubTypestring
Possible Values
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODY

The enabled log.

maxContentLengthDeprecatedinteger

The maximum content length of the response.

proxy

The proxy configuration.

addressstring

The address of the proxy server.

passwordstring

The password for proxy authentication.

portintegerstring

The port of the proxy server.

typestring
DefaultDIRECT
Possible Values
DIRECTHTTPSOCKS

The type of proxy to use.

usernamestring

The username for proxy authentication.

proxyAddressDeprecatedstring

The address of the proxy server.

proxyPasswordDeprecatedstring

The password for proxy authentication.

proxyPortDeprecatedinteger

The port of the proxy server.

proxyTypeDeprecatedstring
Possible Values
DIRECTHTTPSOCKS

The type of proxy to use.

proxyUsernameDeprecatedstring

The username for proxy authentication.

readIdleTimeoutDeprecatedstring
Formatduration

The time allowed for a read connection to remain idle before closing it.

readTimeoutDeprecatedstring
Formatduration

The maximum time allowed for reading data from the server before failing.

ssl

The SSL request options

insecureTrustAllCertificatesbooleanstring

Whether to disable checking of the remote SSL certificate.

Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.

timeout

The timeout configuration.

connectTimeoutstring

The time allowed to establish a connection to the server before failing.

readIdleTimeoutstring
DefaultPT5M

The time allowed for a read connection to remain idle before closing it.

Reference (ref) of the pluginDefaults to apply to this task.

Defaultfalse

Simplified output

Enable Apify simplified output mode; default false.

Defaulttrue

Skip empty items

Drop empty records when true; default true.

Defaultfalse

Skip failed pages

Skip items containing errorInfo when true; default false.

Defaultfalse

Skip hidden

Skip fields prefixed with # when true; default false.

DefaultASC
Possible Values
ASCDESC

Sort direction

Set to DESC to return newest items first; default ASC.

SubTypestring

Unwind fields

Fields to unwind in order; array elements become separate records, objects merge into parents. Unwound items ignore the sortDirection flag.

View

Dataset view name to filter and project items per the Apify schema.

Items fetched from the Apify dataset

More Plugins in this Category

Clickhouse
Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

5 tasks

2 blueprints

Weaviate
Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

4 tasks

2 blueprints