Extract Structured Expense Data from Receipt Images with Gemini Multimodal

Turn a folder of receipt photos into structured expense records without a dedicated OCR pipeline. This blueprint watches an S3 inbox for new receipt images, asks Google Gemini's multimodal model to read the image and describe the vendor, date, currency, and total, then reshapes that description into strict JSON with a second Gemini call. It solves the gap between a model that can read an image and a downstream system that needs typed fields it can insert into a database.

How it works

1. The new_receipt trigger (io.kestra.plugin.aws.s3.Trigger) polls the expense-receipts bucket's inbox/ prefix every 5 minutes and starts one execution per batch of new images, downloading matches to internal storage.
2. describe_receipt (io.kestra.plugin.gemini.MultimodalCompletion) sends the downloaded image alongside a text prompt to gemini-3.5-flash-lite, asking for the vendor, purchase date, currency, and total as labeled plain text.
3. structure_fields (io.kestra.plugin.gemini.StructuredOutputCompletion) takes that free-text description and reshapes it into a fixed JSON schema (vendor, purchaseDate, currency, total), so downstream systems get typed fields instead of prose.
4. notify_finance (io.kestra.plugin.googleworkspace.chat.GoogleChatIncomingWebhook) posts the structured result to a Google Chat space for the finance team to review.

What you get

• Image-in, typed-JSON-out extraction without a separate OCR service.
• A two-step Gemini pattern (multimodal read, then structured reshape) that keeps each call focused and easy to debug.
• Automatic polling so new receipts are picked up without manual uploads to a processing tool.
• A Google Chat notification the finance team can act on immediately.

Who it's for

• Finance and operations teams processing expense receipts or vendor invoices at volume.
• Data engineers building document extraction pipelines without standing up dedicated OCR infrastructure.
• Teams already using Gemini's API key access who want to avoid a second Vertex AI or Document AI integration.

Why orchestrate this with Kestra

Gemini can read one image and describe it, but it has no concept of a receipts folder, no polling loop, and no way to hand its answer to a second model call or a chat notification. Kestra adds the Trigger that watches storage, the execution history around each Gemini call, and the chaining between the multimodal read and the structured reshape, so a failed step does not force a re-read of the image. Everything lives in one declarative YAML flow you can version.

Prerequisites

• A Google Gemini API key with access to a multimodal-capable model.
• An S3 bucket receiving receipt images under an inbox/ prefix.
• A Google Chat incoming webhook for the finance space.

Secrets

• GEMINI_API_KEY: API key for both Gemini calls.
• AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY: credentials for the S3 bucket.
• GOOGLE_CHAT_WEBHOOK: incoming webhook URL for the finance Chat space.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Point the new_receipt trigger at your own bucket and prefix.
3. Upload a test receipt image and enable the trigger.
4. Confirm the structured JSON and the Google Chat message.

How to extend

• Add a database write task after structure_fields to insert the record directly into an expense ledger.
• Swap the JSON schema for invoice line items instead of a single total.
• Add an If task to flag totals above a threshold for manual review.
• Replace Google Chat with Slack or Microsoft Teams for the completion notification.

Links

• plugin-gemini docs