How to Monitor Cron Jobs in Django
Management commands, django-crontab and Celery Beat each fail differently. The environment is what breaks first.
Monitoring Django scheduled work depends on how you schedule it, and Django projects usually accumulate two or three methods over time. Start by finding out which ones you actually have.
The three you will encounter are system cron calling a management command, django-crontab, and Celery Beat. Each fails in its own way.
Management commands called from system cron
This is the most common and most reliable arrangement. The monitoring fits naturally into the command's handle method.
The zero-count branch is the part worth copying. A sync that runs perfectly and imports nothing because the supplier feed returned an empty array is the classic Django scheduled-task failure, and a plain heartbeat reports it as healthy.
The crontab entry is where it actually breaks
Cron runs with a minimal environment. It does not source your shell profile, does not activate your virtualenv, and does not know which settings module to use.
Use the virtualenv's python directly rather than activating the environment. Activation is a shell function and cron is not running a login shell.
Redirect output to a file you can actually read. Cron's default is to mail it, and on most servers there is no mail transfer agent, so the traceback that would have told you what went wrong is generated and then discarded.
django-crontab moves the problem
django-crontab writes entries into the system crontab from your CRONJOBS setting. It is convenient and it introduces a step that can silently not happen.
The entries only exist after someone runs manage.py crontab add. If a deploy adds a job to CRONJOBS and nobody runs that command on the server, the job is in your codebase and not in the crontab. The code review passes and the job never runs.
Run this as a deploy step and compare the count against CRONJOBS. It takes a minute and removes an entire class of confusion.
Celery Beat fails in two places
Beat schedules tasks and workers execute them. They are separate processes, and either can be down while the other looks fine.
If Beat is down, nothing is scheduled and the workers sit idle looking healthy. If the workers are down, Beat keeps queueing tasks that pile up in the broker, and your queue depth grows while Beat reports success at scheduling.
| Component down | What you see | Caught by |
|---|---|---|
| Beat | Workers idle, no errors, no tasks | Heartbeat on a frequent task |
| Workers | Broker queue depth growing | Heartbeat, plus a queue-depth alarm |
| Broker (Redis/RabbitMQ) | Both processes erroring | Uptime check on the broker |
| A single task raising | Celery logs the traceback | Explicit failure ping in the task |
Add a trivial Beat task on a short schedule whose only job is to ping. It proves Beat is scheduling and a worker is executing, which is the pair of facts you actually care about.
A 5-minute schedule is a reasonable default. It costs 288 pings a day, which is nothing, and it means a dead Beat process is caught within minutes rather than whenever your least frequent real task was next due.
Related guides
For non-Django Python see monitoring cron jobs in Python. Background is in heartbeat monitoring.