MongoDB Slow Query and Collection Scan Profiler

In high-scale MongoDB applications, missing indexes represent the primary cause of sudden cluster CPU spikes, replica set failovers, and cascading connection timeouts. When a query targets an unindexed field, the MongoDB query planner executes a full collection scan (COLLSCAN), inspecting every document in the collection from disk.

MongoDB includes an internal database profiler that logs slow operations and unindexed plan summaries to the system.profile capped collection. However, unless teams actively query this collection, unindexed operations go unnoticed until they cause production degradation.

This blueprint establishes an automated database profiler sentinel for MongoDB. Running every 15 minutes (or on-demand), it queries system.profile using an aggregation pipeline to detect operations executing with planSummary: /COLLSCAN/ that exceed your latency threshold. When detected, it dispatches an actionable Slack alert with the offending namespace, execution latency, and index creation guidance.

How it works

1. Scheduled Profiling Audit: The scheduled_profiler_audit trigger (io.kestra.plugin.core.trigger.Schedule) executes every 15 minutes, or runs on-demand via the Kestra UI.
2. Profiler Aggregation: The audit_profiler_entries task (io.kestra.plugin.mongodb.Aggregate) inspects system.profile using an aggregation pipeline, matching records where planSummary contains COLLSCAN and millis breaches slow_ms_threshold.
3. Violation Gate: The evaluate_profiler_violations flowable task (io.kestra.plugin.core.flow.If) branches based on whether unindexed queries were returned.
4. Slack Alert Dispatch: If violations exist, notify_dba_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) delivers an actionable alert card with the namespace, operation type, duration, and indexing advice.
5. Nominal Logging: If no slow unindexed operations are found, log_nominal_status records nominal status in execution logs.
6. Audit Manifest: The export_profiler_manifest task records telemetry outputs for DBA dashboards.

What you get

• Automated detection of unindexed queries before they cause production cluster lockups.
• Visibility into slow queries executing full collection scans.
• Direct Slack notifications detailing the offending collection and client IP.
• Zero human overhead for continuous MongoDB query optimization audits.

Who it is for

• MongoDB Database Administrators governing production replica sets and sharded clusters.
• Backend engineers identifying missing compound or single-field indexes.
• Site Reliability Engineers monitoring database CPU and latency spikes.

Why orchestrate this with Kestra

Monitoring MongoDB profiler collections typically requires setting up external log forwarders or custom monitoring agents. Kestra provides declarative, serverless orchestration: it connects directly via native MongoDB drivers, schedules regular inspections, conditionally alerts via Slack, securely handles connection strings, and maintains execution audit records.

Inputs

Name	Type	Default	Description
database	STRING	production	MongoDB database where profiling is enabled.
slow_ms_threshold	INT	100	Execution duration in milliseconds beyond which a query is flagged.
slack_channel	STRING	#dba-alerts	Slack channel destination for MongoDB query performance alerts.

Expected outputs

• {{ outputs.audit_profiler_entries.rows }}: Array of profiler records containing ns, op, planSummary, exec_time_ms, and client.
• {{ outputs.audit_profiler_entries.size }}: Total number of unindexed slow operations detected.
• {{ outputs.evaluate_profiler_violations }}: Result of conditional branch evaluation.
• {{ outputs.export_profiler_manifest.value }}: Structured JSON telemetry manifest recording execution timestamp and status.

Prerequisites

• A MongoDB instance (version 4.4, 5.0, 6.0, 7.0, or 8.0) with profiling enabled (db.setProfilingLevel(1, { slowms: 100 })).
• A MongoDB connection string with read privileges on the target database and its system.profile collection.
• A Slack Incoming Webhook URL targeting your database alert channel.

Secrets

• MONGODB_URI: MongoDB connection URI (e.g. mongodb+srv://<username>:<password>@cluster.mongodb.net/?retryWrites=true&w=majority).
• SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for sending alert notifications.

Quick start

1. Enable profiling on your target MongoDB database: db.setProfilingLevel(1, { slowms: 100 }).
2. Configure MONGODB_URI and SLACK_WEBHOOK_URL in your Kestra namespace secrets.
3. Import this flow YAML into your Kestra instance.
4. Click Execute in the UI to run an initial profiler audit.
5. Review execution outputs to inspect slow query plans and collection scan frequencies.

Common pitfalls and troubleshooting

• Profiling Must Be Enabled: By default, MongoDB profiling level is 0 (disabled). Run db.setProfilingLevel(1, { slowms: 100 }) in mongosh prior to executing this blueprint.
• Capped Collection Lifecycle: system.profile is a capped collection. When it reaches its maximum size (default 1MB), older entries are overwritten. Schedule this flow frequently (e.g., every 15 minutes) to ensure complete capture.
• Atlas Shared Tier Restrictions: On shared MongoDB Atlas tiers (M0/M2/M5), profiling permissions may be restricted. Ensure you are targeting an M10+ cluster or self-hosted deployment.

How to extend

• Add automated Jira or GitHub issue creation for unindexed queries using io.kestra.plugin.github.issues.Create.
• Expand the aggregation filter to audit aggregation pipeline stages missing index support.
• Aggregate slow queries weekly and email a top-10 offender digest to the development team.

Links

• Kestra MongoDB Plugin Documentation: https://kestra.io/plugins/plugin-mongodb
• Kestra Slack Plugin Documentation: https://kestra.io/plugins/plugin-notifications
• MongoDB Database Profiler Documentation: https://www.mongodb.com/docs/manual/tutorial/manage-the-database-profiler/