PluginsCloudQueryTasks

CloudQueryCLI​Cloud​Query​C​L​I

yaml
type: "io.kestra.plugin.cloudquery.CloudQueryCLI"

Execute CloudQuery commands from a CLI.

Examples

Run a CloudQuery sync from CLI. You need an API key to download plugins. You can add the API key as an environment variable called CLOUDQUERY_API_KEY.

yaml
id: cloudquery_sync_cli
namespace: company.team

tasks:
  - id: hn_to_duckdb
    type: io.kestra.plugin.cloudquery.CloudQueryCLI
    env:
      CLOUDQUERY_API_KEY: "{{ secret('CLOUDQUERY_API_KEY') }}"
    inputFiles:
      config.yml: |
        kind: source
        spec:
          name: hackernews
          path: cloudquery/hackernews
          version: v3.0.13
          tables: ["*"]
          backend_options:
            table_name: cq_cursor
            connection: "@@plugins.duckdb.connection"
          destinations:
            - "duckdb"
          spec:
            item_concurrency: 100
            start_time: "{{ now() | dateAdd(-1, 'DAYS') }}"
          ---
          kind: destination
          spec:
            name: duckdb
            path: cloudquery/duckdb
            version: v4.2.10
            write_mode: overwrite-delete-stale
            spec:
              connection_string: hn.db
    commands:
      - cloudquery sync config.yml --log-console

Properties

commands

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ✔️
  • Min items: 1

List of CloudQuery commands to run.

containerImage

  • Type: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: ghcr.io/cloudquery/cloudquery:latest

The task runner container image, only used if the task runner is container-based.

docker

Deprecated, use 'taskRunner' instead

env

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

Additional environment variables for the CloudQuery process.

inputFiles

  • Type:
    • object
    • string
  • Dynamic: ✔️
  • Required: ❌

The files to create on the local filesystem. It can be a map or a JSON object.

namespaceFiles

Inject namespace files.

Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the include or exclude properties to limit which namespace files will be injected.

outputFiles

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

The files from the local filesystem to send to Kestra's internal storage.

Must be a list of glob) expressions relative to the current working directory, some examples: my-dir/, my-dir/*/ or my-dir/my-file.txt.

taskRunner

  • Type: TaskRunner
  • Dynamic: ❌
  • Required: ❌
  • Default: { "type": "io.kestra.plugin.scripts.runner.docker.Docker", "entryPoint": [] }

The task runner to use.

Task runners are provided by plugins, each have their own properties. If you change from the default one, be careful to also configure the entrypoint to an empty list if needed.

Outputs

exitCode

  • Type: integer
  • Required: ✔️
  • Default: 0

outputFiles

  • Type: object
  • SubType: string
  • Required: ❌

vars

  • Type: object
  • Required: ❌

Definitions

io.kestra.core.models.tasks.NamespaceFiles

  • enabled
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌
    • Default: true
  • exclude
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌
  • include
    • Type: array
    • SubType: string
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Cpu

  • cpus
    • Type: integer
    • Dynamic: ❌
    • Required: ❌

io.kestra.core.models.tasks.runners.TaskRunner

  • type
    • Type: string
    • Dynamic: ❌
    • Required: ✔️
    • Validation RegExp: \p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*(\.\p{javaJavaIdentifierStart}\p{javaJavaIdentifierPart}*)*
    • Min length: 1

io.kestra.plugin.scripts.runner.docker.Memory

  • kernelMemory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memoryReservation
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwap
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memorySwappiness
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • oomKillDisable
    • Type: boolean
    • Dynamic: ❌
    • Required: ❌

io.kestra.plugin.scripts.exec.scripts.models.DockerOptions

  • image
    • Type: string
    • Dynamic: ✔️
    • Required: ✔️
    • Min length: 1
  • config
    • Type:
      • string
      • object
    • Dynamic: ✔️
    • Required: ❌
  • cpu
    • Type: Cpu
    • Dynamic: ❌
    • Required: ❌
  • credentials
  • deviceRequests
  • entryPoint
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • extraHosts
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • host
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • memory
    • Type: Memory
    • Dynamic: ❌
    • Required: ❌
  • networkMode
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • pullPolicy
    • Type: string
    • Dynamic: ❌
    • Required: ❌
    • Default: ALWAYS
    • Possible Values:
      • IF_NOT_PRESENT
      • ALWAYS
      • NEVER
  • shmSize
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • user
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • volumes
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.Credentials

  • auth
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • identityToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • password
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registry
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • registryToken
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • username
    • Type: string
    • Dynamic: ✔️
    • Required: ❌

io.kestra.plugin.scripts.runner.docker.DeviceRequest

  • capabilities
    • Type: array
    • SubType: array
    • Dynamic: ❌
    • Required: ❌
  • count
    • Type: integer
    • Dynamic: ❌
    • Required: ❌
  • deviceIds
    • Type: array
    • SubType: string
    • Dynamic: ✔️
    • Required: ❌
  • driver
    • Type: string
    • Dynamic: ✔️
    • Required: ❌
  • options
    • Type: object
    • SubType: string
    • Dynamic: ❌
    • Required: ❌