PluginsTasks

Sync​Sync

yaml
type: "io.kestra.plugin.cloudquery.Sync"

Execute a CloudQuery sync.

Examples

Start a CloudQuery sync based on a YAML configuration. You need an API key to download plugins. You can add the API key as an environment variable called CLOUDQUERY_API_KEY.

yaml
id: cloudquery_sync
namespace: company.team

tasks:
  - id: hn_to_duckdb
    type: io.kestra.plugin.cloudquery.Sync
    env:
      CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
    incremental: false
    configs:
      - kind: source
        spec:
          name: hackernews
          path: cloudquery/hackernews
          version: v3.0.13
          tables: ["*"]
          destinations: ["duckdb"]
          spec:
            item_concurrency: 100
            start_time: "{{ now() | dateAdd(-1, 'DAYS') }}"
      - kind: destination
        spec:
          name: duckdb
          path: cloudquery/duckdb
          version: v4.2.10
          write_mode: overwrite-delete-stale
          spec:
            connection_string: hn.db

Start a CloudQuery sync based on a file(s) input.

yaml
id: cloudquery_sync
namespace: company.team

tasks:
  - id: hn_to_duckdb
    type: io.kestra.plugin.cloudquery.Sync
    incremental: false
    env:
        AWS_ACCESS_KEY_ID: "{{ secret('AWS_ACCESS_KEY_ID') }}"
        AWS_SECRET_ACCESS_KEY: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
        AWS_DEFAULT_REGION: "{{ secret('AWS_DEFAULT_REGION') }}"
        CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
        PG_CONNECTION_STRING: "postgresql://postgres:{{ secret('DB_PASSWORD') }}@host.docker.internal:5432/demo?sslmode=disable"
    configs:
      - sources.yml
      - destination.yml

Properties

configs

  • Type: array
  • SubType:
  • Dynamic: ❌
  • Required: ✔️

CloudQuery configurations.

A list of CloudQuery configurations or files containing CloudQuery configurations.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: ghcr.io/cloudquery/cloudquery:latest

The task runner container image, only used if the task runner is container-based.

docker

Deprecated, use 'taskRunner' instead

env

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Additional environment variables for the CloudQuery process.

incremental

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

Whether to use Kestra's internal KV Store backend to save incremental index.

Kestra can automatically add a backend option to your sources and store the incremental indexes in the KV Store. Use this boolean to activate this option.

inputFiles

  • Type:
    • object
    • string
  • Dynamic: ✔️
  • Required: ❌

The files to create on the local filesystem. It can be a map or a JSON object.

namespaceFiles

Inject namespace files.

Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the include or exclude properties to limit which namespace files will be injected.

outputFiles

  • Type: array
  • SubType: string
  • Dynamic: ❌
  • Required: ❌

The files from the local filesystem to send to Kestra's internal storage.

Must be a list of glob) expressions relative to the current working directory, some examples: my-dir/, my-dir/*/ or my-dir/my-file.txt.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker", "entryPoint": [] }

The task runner to use.

Task runners are provided by plugins, each have their own properties. If you change from the default one, be careful to also configure the entrypoint to an empty list if needed.

Outputs

exitCode

  • Type: integer
  • Required: ✔️
  • Default: 0

outputFiles

  • Type: object
  • SubType: string
  • Required: ❌

taskRunner

vars

  • Type: object
  • Required: ❌

Definitions

io.kestra.core.models.tasks.NamespaceFiles

  • enabled
    • Type:
      • boolean
      • string
    • Dynamic: ✔️
    • Required: ❌
  • exclude
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • include
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Cpu

  • cpus
    • Type: integer
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1

io.kestra.plugin.scripts.runner.docker.Memory

  • kernelMemory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memoryReservation
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwap
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwappiness
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • oomKillDisable
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.exec.scripts.models.DockerOptions

  • image
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
    • Min length: 1
  • config
    • Type:
      • string
      • object
    • Dynamic: ✔️
    • Required: ❌
  • cpu
    • Type: Cpu
    • Dynamic: ❌
    • Required: ❌
  • credentials
  • deviceRequests
  • entryPoint
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • extraHosts
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • host
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: Memory
    • Dynamic: ❌
    • Required: ❌
  • networkMode
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • privileged
    • Type: boolean
    • Dynamic: ✔️
    • Required: ❌
  • pullPolicy
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: ALWAYS
    • Possible Values:
      • IF_NOT_PRESENT
      • ALWAYS
      • NEVER
  • shmSize
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • user
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • volumes
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Credentials

  • auth
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • identityToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registry
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registryToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunnerDetailResult

io.kestra.plugin.scripts.runner.docker.DeviceRequest

  • capabilities
    • Type: array
    • SubType: array
    • Dynamic: ❌
    • Required: ❌
  • count
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
  • deviceIds
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • driver
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • options
    • Type: object
    • SubType: string
    • Dynamic: ❌
    • Required: ❌