
MongoDB Bulk
CertifiedRun MongoDB bulkWrite from NDJSON
MongoDB Bulk
Run MongoDB bulkWrite from NDJSON
Reads newline-delimited bulk operations from internal storage and executes MongoDB bulkWrite.
Supports insertOne, updateOne, updateMany, replaceOne, deleteOne and deleteMany in the db.collection.bulkWrite() format (https://www.mongodb.com/docs/manual/reference/method/db.collection.bulkWrite/).
insertOne also accepts the legacy flat form where the body is the document itself; a body whose only field is an object named document is read as the bulkWrite form.
Blank and whitespace-only lines are skipped.
Delete, update and replace operations support collation and hint.
Inherits chunking from AbstractLoad (default 1000 per bulk request).
The file must be UTF-8: a line that isn't valid UTF-8 fails the task with its line number.
When an operation fails, the task fails with its record number (counting emitted operations, not file lines); when a line can't be read, with its line number.
Either way, its outputs count the operations applied before it, which are kept.
A retry runs every line again, so it is only safe for operations that change nothing more when repeated, such as replaceOne, or updateOne with $set, using upsert: true as long as the filter still matches the document after the modification (such as filtering directly on _id): an insertOne is inserted again, or fails with a duplicate key error if it sets _id, $inc or $push updates are applied twice, and a deleteOne whose filter matches several documents deletes another one.
type: io.kestra.plugin.mongodb.BulkExamples
id: mongodb_bulk
namespace: company.team
tasks:
- id: make_actions
type: io.kestra.plugin.core.storage.Write
content: |
{ "insertOne" : {"document": {"firstName": "John", "lastName": "Doe", "city": "Paris"}}}
{ "insertOne" : {"document": {"firstName": "Ravi", "lastName": "Singh", "city": "Mumbai"}}}
{ "deleteMany": {"filter": {"city": "Bengaluru"}}}
- id: bulk
type: io.kestra.plugin.mongodb.Bulk
connection:
uri: "mongodb://YOUR_HOST:27017"
database: "my_database"
collection: "my_collection"
from: "{{ outputs.make_actions.uri }}"
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-mongodb --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version.
Properties
collection *string
MongoDB collection
connection *
MongoDB connection properties
io.kestra.plugin.mongodb.MongoDbConnection
Connection string to MongoDB server
URL format like mongodb://mongodb0.example.com: 27017
database *string
MongoDB database
from *string
Source file
Path in internal storage (kestra://...) containing BSON or JSON documents to stream into MongoDB.
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150chunk integerstring
1000Bulk chunk size
Number of write models sent per bulkWrite call; defaults to 1000. Must be at least 1.
Outputs
deletedCount integer
0Deleted documents
Documents removed by bulk delete models. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.
insertedCount integer
0Inserted documents
Count of documents created by bulk writes. 0 for unacknowledged writes (w=0), which have no counts to report.
matchedCount integer
0Matched documents
Documents matched by update or replacement models. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.
modifiedCount integer
0Modified documents
Documents updated by bulk operations. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.
size integer
Rows processed
Total records read from the source file. When a write fails, the records applied before it.
upsertedCount integer
0Upserted documents
Documents created by update or replacement models with upsert: true that matched no document. Only Bulk sends these models, so it's always 0 for Load. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.
wasAcknowledged boolean
Whether the writes were acknowledged
False with an unacknowledged write concern (w=0): the counts are then 0.
Metrics
records counter
countNumber of documents processed in the bulk operation; when a write fails, those applied before it
requests.count counter
countNumber of bulk requests sent to MongoDB