Monitoring Celery with Flower

Audience: Ops and backend engineers running Celery workers on RabbitMQ. Golden rule: Flower is an ops dashboard. Product job status always comes from your `jobs` table / GET /jobs/{id} : never from Flower.

TL;DR

Celery workers ◄── events / inspect ──► Flower (ops UI)
 │
 └── still emit metrics/logs to Prometheus/your stack

Contents

1. What Flower is for
2. What Flower is not for
3. Install and run
4. Essential configuration
5. Security (mandatory)
6. What to watch in the UI
7. Enable Celery events
8. Flower + metrics (better together)
9. Deploy sketch
10. Troubleshooting
11. Checklist

---

1. What Flower is for

Flower is a real-time web monitor for Celery:

• List workers (alive, concurrency, queues)
• See active, reserved, scheduled tasks
• Inspect task success/failure history (when events are enabled)
• Basic rates and task runtime views
• Optional revoke / shutdown controls (treat as dangerous in prod)

Use it during incidents: “Are workers connected? Is a task stuck active? Did failures spike?”

---

2. What Flower is not for

See 15 Observability.

---

3. Install and run

# same app env as workers
pip install flower

# PSEUDOCODE : module path to Celery app instance
celery -A app.workers.celery_app.celery_app flower \
 --port=5555 \
 --basic_auth=ops_user:strong_password

# PSEUDOCODE : app/workers/celery_app.py must be importable
from celery import Celery
celery_app = Celery("app", broker=settings.celery_broker_url)

Compose / process list:

api | worker | beat (1) | flower (1, private) | rabbitmq | redis | db

---

4. Essential configuration

# PSEUDOCODE : richer RabbitMQ view (management plugin + credentials)
celery -A app.workers.celery_app.celery_app flower \
 --broker_api=https://user:pass@rabbitmq:15672/api/

Environment variables (common):

CELERY_BROKER_URL=amqps://...
FLOWER_BASIC_AUTH=ops_user:strong_password
FLOWER_PORT=5555

---

5. Security (mandatory)

Never put Flower on the public internet without controls.

# PSEUDOCODE : internal reverse proxy sketch
# listen only on internal LB
location /flower/ {
 auth_request /oauth2/auth; # or htpasswd
 proxy_pass http://flower:5555/;
 proxy_set_header Host $host;
 proxy_set_header X-Forwarded-Proto https;
}

Same rules for RabbitMQ Management UI.

---

6. What to watch in the UI

Workers tab

Tasks

During incidents

1. Any workers online?
2. Active task stuck for too long? (compare to soft_time_limit)
3. Failure rate jumping?
4. Then check RabbitMQ depth/DLQ and app metrics : not Flower alone

---

7. Enable Celery events

Flower relies on Celery events for live task visibility.

# PSEUDOCODE : celery config (workers)
celery_app.conf.update(
 worker_send_task_events=True, # workers emit events
 task_send_sent_event=True, # optional: task-sent events
)

# workers must not disable events
celery -A app.workers.celery_app.celery_app worker -E -Q jobs.default,jobs.io
# -E is --task-events

Without events, Flower may show workers but sparse/empty task history.

---

8. Flower + metrics (better together)

Minimum production set:

1. Flower (private) for humans
2. Prometheus metrics: enqueue, success, fail, runtime, depth, consumers
3. Alerts: zero consumers, DLQ > 0, success drop, SLO breach
4. JSON logs with job_id + task_id

---

9. Deploy sketch

# PSEUDOCODE : docker-compose fragment
services:
 flower:
 image: your-app:${TAG}
 command: >
 celery -A app.workers.celery_app.celery_app flower
 --port=5555
 --basic_auth=${FLOWER_BASIC_AUTH}
 environment:
 CELERY_BROKER_URL: ${CELERY_BROKER_URL}
 # no public ports in production : only attach to internal network
 networks: [internal]
 depends_on: [rabbitmq]
 restart: unless-stopped

Kubernetes:

• Deployment replicas: 1 is enough for Flower
• Service ClusterIP only
• Ingress with auth + TLS, or no Ingress (port-forward / VPN)
• Resource limits modest (CPU/memory grow with --persistent and task volume)

---

10. Troubleshooting

---

11. Checklist

• [ ] Flower runs as its own process/container (not inside API)
• [ ] Same Celery app module / broker as workers
• [ ] Task events enabled on workers (-E / config)
• [ ] Private network only
• [ ] Authentication enabled (basic or SSO)
• [ ] TLS at the edge
• [ ] Documented: Flower ≠ user job status
• [ ] RabbitMQ depth/DLQ/consumer metrics + alerts still configured
• [ ] Optional --broker_api only over TLS with locked-down creds
• [ ] Revoke/admin actions limited by policy
• [ ] Runbook link from dashboard (scale workers, redrive DLQ)

---

Quick commands

# start
celery -A app.workers.celery_app.celery_app flower --port=5555 --basic_auth=user:pass

# worker with events
celery -A app.workers.celery_app.celery_app worker -E -Q jobs.default,jobs.io -c 4

# never: expose 5555 on 0.0.0.0 to the internet without auth

---

Prometheus: 19 Integrate Flower with Prometheus

See also

• 15 Observability, metrics, and Flower : health, SLOs, alerts
• 05 Celery + RabbitMQ
• 14 Celery Beat
• 17 Celery RL + lock pipeline
• 09 Errors / DLQ

External

• Flower documentation
• Celery monitoring
• Celery events