Fetch JSON from an API in Python and load it into MongoDB

Pull a JSON payload from any REST API and land it in MongoDB without standing up custom glue code. This flow runs a Python script inside an isolated Docker container to call the GitHub API, captures the response as a JSON file, and bulk loads that document into a MongoDB collection. It is a ready made template for API to document store ingestion, web scraping pipelines, and keeping a NoSQL collection in sync with an upstream HTTP source.

How it works

• The generate_json task is an io.kestra.plugin.scripts.python.Script running on the io.kestra.plugin.scripts.runner.docker.Docker task runner. It installs the requests and kestra dependencies, calls https://api.github.com, and writes the parsed response to output.json declared in outputFiles. It also emits the raw data and HTTP status through Kestra.outputs().
• The load_to_mongodb task is an io.kestra.plugin.mongodb.Load. It connects to MongoDB over the connection.uri, targets the local database and the github collection, and ingests the file referenced by {{ outputs.generate_json.outputFiles['output.json'] }}.

What you get

• An end to end API to MongoDB ingestion pipeline in a single YAML file.
• Reproducible runs thanks to a pinned Python environment in Docker.
• The HTTP response persisted both as a Kestra internal storage file and inside MongoDB.
• Clear task outputs (data, status) you can branch on or reuse downstream.

Who it's for

• Data engineers building ingestion pipelines from REST APIs into a document store.
• Backend and platform teams syncing third party data into MongoDB.
• Analysts and developers prototyping NoSQL data collection without bespoke scripts.

Why orchestrate this with Kestra

A raw Python cron job gives you no retries, no lineage, and no visibility when an API call fails. With Kestra you get event and schedule triggers, automatic retries, run history, and outputs passed cleanly between tasks, all declared in version controlled YAML. MongoDB itself has no scheduler or workflow engine, so Kestra fills the gap by deciding when ingestion runs, handling transient API failures, and wiring the Python extraction step to the MongoDB load step.

Prerequisites

• A running Kestra instance with Docker available to the worker.
• A reachable MongoDB instance (the flow points at mongodb://host.docker.internal:27017/).

Secrets

This flow uses an open API and a local MongoDB URI, so no secret() values are referenced. If you point it at an authenticated API or a secured MongoDB, store credentials as secrets (for example {{ secret('MONGODB_URI') }}) and reference them in the connection.uri and script.

Quick start

1. Add this flow to your Kestra instance.
2. Confirm MongoDB is reachable at the configured connection.uri.
3. Execute the flow.
4. Inspect the github collection in the local database to confirm the document was loaded.

How to extend

• Swap https://api.github.com for your own API endpoint and adjust the parsing logic.
• Add a Schedule or webhook trigger to ingest on a cadence or on demand.
• Change the target database and collection, or template them from inputs.
• Add a transformation task between extraction and load to reshape the payload.

Links

• Plugin docs: MongoDB
• Plugin docs: Python scripts