Vertex AI serves the Gemini models. Kestra runs everything around the call: schedule it, retry it, rate-limit a batch, branch on the result, and chain it with your warehouse and storage, so model inference is a governed step with one execution history, not a notebook script.
A raw Vertex AI call has no scheduler, no retry on a 429, no batch fan-out, and no cost guardrail. Kestra adds all of it: it batch-enriches BigQuery rows under a concurrency limit, keeps a RAG index fresh from GCS documents with Gemini embeddings, classifies Pub/Sub events inline, and caps spend with an automated If on the token-count metric. The model call becomes one observable, retryable step.
Vertex AI runs the models, the regions, and the IAM. Kestra runs the steps around the call: what triggers it, how a batch is rate-limited, what it is chained with, and how cost is capped.
A raw SDK call has no cron, no backoff on a 429, and no per-step retry. Kestra triggers io.kestra.plugin.gcp.vertexai.ChatCompletion on a Schedule or an event and wraps it with retry and timeout, so transient quota errors recover automatically instead of failing the run.
Enriching thousands of rows means looping model calls without tripping quotas. Kestra reads rows from BigQuery, fans them out with ForEach whose concurrencyLimit caps the request rate, calls Vertex AI per row, and writes the result back, all as one scheduled, retryable batch.
A retrieval index goes stale unless something rebuilds it. Kestra ingests documents on a Schedule, chunks them, embeds them with a Gemini model via io.kestra.plugin.ai.rag.IngestDocument, and writes the vectors to a store, so a RAG app always retrieves grounded, current context.
LLM spend runs away quietly inside a loop. Vertex AI tasks emit total.token.count as a metric, and Kestra branches on it with an automated If to stop a batch when a threshold is crossed. The guardrail is code, not a person watching a dashboard.
Some pipelines need a model decision per event. Kestra classifies a Pub/Sub message with Vertex AI, then routes on the label with an automated If, escalating high-priority events and logging the rest, in milliseconds, with no human gate on routine routing.
Patterns AI and data engineering teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
Rows need an LLM-generated label or summary. Kestra selects the unenriched rows from BigQuery, fans them out with a ForEach whose concurrency caps the request rate, calls a Gemini model per row, and writes the result back. Email confirms the batch.
concurrencyLimit caps in-flight calls so the batch stays under quota.
Each enriched value updates its row, so the table fills in incrementally.
A Schedule runs the batch nightly with retries, not a notebook.
A retrieval index must track the latest source content. Kestra pulls documents from GCS, chunks them, and embeds them with a Gemini model into a vector store on a schedule, so a RAG application always retrieves current, grounded context. Slack confirms each reindex.
Rebuilding nightly means retrieval reflects the latest documents.
Use the KV store for a demo, Qdrant or PGVector to scale.
Vertex AI serves the same Gemini models used for the embeddings.
Events need a model decision the moment they arrive. The Pub/Sub RealtimeTrigger starts a run per event, Vertex AI assigns a label, and an automated If escalates the high-priority ones to an audit table and Discord. No human gate on routine routing.
Each message is classified and routed in its own governed run.
An If on the label routes, never a human Pause.
A tight prompt forces a single-token label the branch can trust.
A large enrichment batch can burn budget unnoticed. Kestra reads the batch from BigQuery, calls Vertex AI under a concurrency limit, and an automated If on the emitted total.token.count metric halts the run when spend crosses a threshold. Email reports the cap.
Vertex AI tasks emit total.token.count for the guardrail to read.
An If halts the batch when cost crosses the threshold.
Spend is bounded by the flow, not discovered on the invoice.
Model training belongs in the pipeline, not a separate console. Kestra uploads the training data to GCS, runs CustomJob to train on Vertex AI while streaming logs and waiting for completion, then loads the results into BigQuery. Slack confirms.
CustomJob streams logs and waits, so downstream steps run after it.
GCS staging and a BigQuery load bracket the training step.
A failed job retries and lands in execution history like any task.
| Capability | Vertex AI Pipelines | |||
|---|---|---|---|---|
| Schedule and retry model calls | Schedule + per-task retry | Pipeline runs, Python SDK | Cron + Python | Cron + Python |
| Batch enrichment with rate control | ForEach concurrencyLimit | Custom component | Dynamic tasks, Python | Dynamic tasks, Python |
| RAG ingestion to a vector store | Native ai.rag tasks | Custom component | Python operators | Python operators |
| Cost guardrail on token usage | If on token-count metric | Custom code | Python logic | Python logic |
| Chain with warehouse, GCS, dbt | Native plugins | ML-focused, custom | Operators, Python | Operators, Python |
| Declarative YAML, no Python | Pure YAML | Python SDK | Python DAGs | Python DAGs |
| Self-hosted or air-gapped option | Open source, self-hosted | GCP-managed | GCP-managed | Self or managed |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Batch-enrich warehouse rows, keep a RAG index fresh, classify events inline, and cap cost with automated guardrails in one flow. Open source, self-hosted, event-driven.
