MongoDB Load
CertifiedBulk insert documents from internal storage
MongoDB Load
Bulk insert documents from internal storage
Reads a Kestra internal storage file of JSON/BSON records and inserts them with MongoDB bulkWrite. Inherits chunking (default 1000). Optionally derives _id from a field and removes that field. Objects whose first key is a MongoDB Extended JSON type wrapper (such as $date, $oid or $numberDecimal) are decoded to that BSON type; a malformed wrapper fails the task with the record and field it was found in. When a write fails, the task fails with the number of the failing record, and its outputs count the records applied before it, which stay in the collection. A retry reads the whole file again: without idKey, the records already written are inserted a second time; with idKey, records with a present idKey fail on retry with a duplicate key error (records where idKey is missing or null receive a generated ObjectId and are inserted again).
type: io.kestra.plugin.mongodb.LoadExamples
id: mongodb_load
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: load
type: io.kestra.plugin.mongodb.Load
connection:
uri: "mongodb://root:example@localhost:27017/?authSource=admin"
database: "my_database"
collection: "my_collection"
from: "{{ inputs.file }}"
Properties
collection *string
MongoDB collection
connection *
MongoDB connection properties
io.kestra.plugin.mongodb.MongoDbConnection
Connection string to MongoDB server
URL format like mongodb://mongodb0.example.com: 27017
database *string
MongoDB database
from *string
Source file
Path in internal storage (kestra://...) containing BSON or JSON documents to stream into MongoDB.
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150chunk integerstring
1000Bulk chunk size
Number of write models sent per bulkWrite call; defaults to 1000. Must be at least 1.
idKey string
Field used as _id
If set, the value of this field is stored as _id (converted to an ObjectId if a 24-character hex string or an $oid wrapper, or stored directly as a custom primary key). If the field is missing or null, a new ObjectId is generated.
removeIdKey booleanstring
trueRemove idKey field
When true (default), drops the source field after copying it to _id. Has no effect when idKey is _id.
Outputs
deletedCount integer
0Deleted documents
Documents removed by bulk delete models. 0 for unacknowledged writes (w=0), which have no counts to report.
insertedCount integer
0Inserted documents
Count of documents created by bulk writes. 0 for unacknowledged writes (w=0), which have no counts to report.
matchedCount integer
0Matched documents
Documents matched by update or replacement models. 0 for unacknowledged writes (w=0), which have no counts to report.
modifiedCount integer
0Modified documents
Documents updated by bulk operations. 0 for unacknowledged writes (w=0), which have no counts to report.
size integer
Rows processed
Total records read from the source file. When a write fails, the records applied before it.
upsertedCount integer
0Upserted documents
Documents created by update or replacement models with upsert: true that matched no document. Only Bulk sends these models, so it's always 0 for Load. 0 for unacknowledged writes (w=0), which have no counts to report.
wasAcknowledged boolean
Whether the writes were acknowledged
False with an unacknowledged write concern (w=0): the counts are then 0.
Metrics
records counter
countNumber of documents processed in the bulk operation; when a write fails, those applied before it
requests.count counter
countNumber of bulk requests sent to MongoDB
