Integration Monitoring for DevOps Teams
A monitor created by hand in a UI is infrastructure nobody can reproduce. Define it next to the resource it watches.
The DevOps objection to most monitoring tools is straightforward. A monitor clicked into existence in a web UI is undocumented infrastructure: not in version control, not reviewable, not reproducible, and gone when the person who made it leaves.
If your infrastructure is declarative and your monitoring is not, the monitoring will drift and you will find out during an incident.
Define monitors with the resource
The goal is that creating a scheduled job and creating its monitor are the same change, reviewed together, applied together, and destroyed together.
Use the provider's own Terraform provider where one exists rather than raw HTTP calls, because you want deletion handled too. A monitor that outlives the job it watched becomes a permanent false alarm, and permanent false alarms are how teams learn to ignore a channel.
Drift detection applies to monitoring too
You already detect drift between declared and actual infrastructure. Apply the same check to monitoring, because manual edits during an incident are exactly when monitors get changed and never changed back.
The common drift is somebody widening a grace period at 3am to stop an alert firing, then never restoring it. Six weeks later that monitor has a four-hour grace period and nobody remembers why.
Ephemeral environments need an exclusion rule
Preview environments per pull request break naive monitoring in both directions. Either you monitor them and get alerts every time a branch is deleted, or you exclude them and lose coverage on anything that only exists in preview.
The workable rule is that ephemeral environments get monitors with alerting disabled and a short retention, so failures are visible on a dashboard for debugging and nothing pages anyone.
| Environment | Monitors | Alerting | Lifecycle |
|---|---|---|---|
| Production | Full | Page on critical | Managed in code |
| Staging | Full | Ticket only | Managed in code |
| Preview / PR | Created with the env | Disabled | Destroyed with the env |
| Local | None | None | n/a |
The watcher needs to be outside the thing it watches
Self-hosted monitoring in the same cluster as the workload shares its failure domain. If the cluster degrades, the monitoring degrades with it, and the alert about the cluster never leaves the cluster.
This is the argument for at least one external check even in an otherwise self-hosted stack. It does not need to be elaborate: something outside your perimeter confirming that a heartbeat from inside is still arriving covers the case where everything internal is down at once.
The same reasoning applies to self-hosted Uptime Kuma, and is covered in Uptime Kuma alternative.
Expiry belongs in the same pipeline
Certificates issued by cert-manager renew themselves until an ACME challenge starts failing. Cloud credentials rotate until a pipeline that nobody owns stops running. Domain registrations sit entirely outside your infrastructure and expire on a date nothing in your cluster knows about.
Those dates belong in the same declarative system as everything else, with owners attached, rather than in somebody's calendar.
SensaCat exposes monitors, credentials and domains over an API so they can be managed the same way as the rest of your stack. Start free.