MongoDB Bulk

MongoDB Bulk

Certified

Run MongoDB bulkWrite from NDJSON

Reads newline-delimited bulk operations from internal storage and executes MongoDB bulkWrite. Supports insertOne, updateOne, updateMany, replaceOne, deleteOne and deleteMany in the db.collection.bulkWrite() format (https://www.mongodb.com/docs/manual/reference/method/db.collection.bulkWrite/). insertOne also accepts the legacy flat form where the body is the document itself; a body whose only field is an object named document is read as the bulkWrite form. Blank and whitespace-only lines are skipped. Delete, update and replace operations support collation and hint. Inherits chunking from AbstractLoad (default 1000 per bulk request). The file must be UTF-8: a line that isn't valid UTF-8 fails the task with its line number. When an operation fails, the task fails with its record number (counting emitted operations, not file lines); when a line can't be read, with its line number. Either way, its outputs count the operations applied before it, which are kept. A retry runs every line again, so it is only safe for operations that change nothing more when repeated, such as replaceOne, or updateOne with $set, using upsert: true as long as the filter still matches the document after the modification (such as filtering directly on _id): an insertOne is inserted again, or fails with a duplicate key error if it sets _id, $inc or $push updates are applied twice, and a deleteOne whose filter matches several documents deletes another one.

yaml
type: io.kestra.plugin.mongodb.Bulk
yaml
id: mongodb_bulk
namespace: company.team

tasks:
  - id: make_actions
    type: io.kestra.plugin.core.storage.Write
    content: |
      { "insertOne" : {"document": {"firstName": "John", "lastName": "Doe", "city": "Paris"}}}
      { "insertOne" : {"document": {"firstName": "Ravi", "lastName": "Singh", "city": "Mumbai"}}}
      { "deleteMany": {"filter": {"city": "Bengaluru"}}}

  - id: bulk
    type: io.kestra.plugin.mongodb.Bulk
    connection:
      uri: "mongodb://YOUR_HOST:27017"
    database: "my_database"
    collection: "my_collection"
    from: "{{ outputs.make_actions.uri }}"
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-mongodb --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version.

Properties

MongoDB collection

MongoDB connection properties

Definitions
uri*string

Connection string to MongoDB server

URL format like mongodb://mongodb0.example.com: 27017

MongoDB database

Source file

Path in internal storage (kestra://...) containing BSON or JSON documents to stream into MongoDB.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Default1000

Bulk chunk size

Number of write models sent per bulkWrite call; defaults to 1000. Must be at least 1.

Default0

Deleted documents

Documents removed by bulk delete models. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.

Default0

Inserted documents

Count of documents created by bulk writes. 0 for unacknowledged writes (w=0), which have no counts to report.

Default0

Matched documents

Documents matched by update or replacement models. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.

Default0

Modified documents

Documents updated by bulk operations. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.

Rows processed

Total records read from the source file. When a write fails, the records applied before it.

Default0

Upserted documents

Documents created by update or replacement models with upsert: true that matched no document. Only Bulk sends these models, so it's always 0 for Load. After a task fails on a driver rejection, this count doesn't include the batch the driver sent just before the rejected record. 0 for unacknowledged writes (w=0), which have no counts to report.

Whether the writes were acknowledged

False with an unacknowledged write concern (w=0): the counts are then 0.

Unitcount

Number of documents processed in the bulk operation; when a write fails, those applied before it

Unitcount

Number of bulk requests sent to MongoDB