Guide 19

Integrate Flower with Prometheus

Audience: Teams running Celery + Flower who want scrapeable metrics and alerts. Policy: Flower UI stays private. Prometheus scrapes an internal metrics endpoint. Product job status still comes from the `jobs` table.

TL;DR

Observability: Flower, Prometheus, app metrics

Piece	Role
Flower `/metrics`	Celery-centric Prometheus metrics (workers, tasks)
Prometheus	Scrapes Flower (+ RabbitMQ exporter + app metrics)
Grafana / Alertmanager	Dashboards and pages
Not enough alone	Queue depth/DLQ → RabbitMQ exporter; SLOs → app/job metrics

text

Celery workers ──events──► Flower ──/metrics──► Prometheus ──► Grafana / Alertmanager
RabbitMQ ──exporter──► rabbitmq-exporter ──/metrics──┘
API/workers ──custom counters/histograms──────────┘

---

1. Architecture

text

┌────────────┐ ┌─────────────┐ ┌────────────┐
│ workers │ │ Flower │ │ Prometheus │
│ (-E) │────►│ :5555 │────►│ scrape │
└────────────┘ │ /metrics │ └─────┬──────┘
 └─────────────┘ │
┌────────────┐ ┌─────────────┐ │
│ RabbitMQ │────►│ RMQ exporter│───────────┤
└────────────┘ └─────────────┘ │
┌────────────┐ ┌─────────────┐ │
│ API/worker │────►│ app /metrics│───────────┘
│ (custom) │ └─────────────┘
└────────────┘

Flower answers: worker liveness, task counts/rates from Celery’s view. RabbitMQ answers: depth, consumers, DLQ. Your app answers: enqueue, job SLOs, business outcomes.

---

2. Enable Flower metrics

Flower exposes Prometheus metrics when the Prometheus client is available and the metrics endpoint is enabled (Flower 1.0+ / 2.x).

bash

pip install flower prometheus-client

bash

# PSEUDOCODE : run Flower (internal only)
celery -A app.workers.celery_app.celery_app flower \
 --port=5555 \
 --basic_auth="${FLOWER_BASIC_AUTH}" \
 --address=0.0.0.0

Metrics URL (cluster-internal):

text

http://flower:5555/metrics

Workers still need task events so Flower sees task activity:

bash

celery -A app.workers.celery_app.celery_app worker -E -Q jobs.default,jobs.io

python

celery_app.conf.update(
 worker_send_task_events=True,
 task_send_sent_event=True,
)

> Note: Metric names can vary slightly by Flower version. Hit /metrics once and copy the actual series names into dashboards. Below uses common flower_* patterns : adjust if your build differs.

Optional: scrape without basic-auth friction

Prefer network policy (Prometheus in the same mesh scrapes Flower Service) over disabling auth.

If Prometheus must scrape past basic auth:

yaml

# PSEUDOCODE : prometheus.yml scrape with basic auth
basic_auth:
 username: ops_user
 password: strong_password

Or terminate auth at the UI path only and leave /metrics on a separate internal listener (if you customize the proxy). Simplest: ClusterIP + basic_auth in scrape config.

---

3. Prometheus scrape config

yaml

# PSEUDOCODE : prometheus.yml
global:
 scrape_interval: 15s

scrape_configs:
 - job_name: flower
 metrics_path: /metrics
 static_configs:
 - targets: ["flower:5555"]
 labels:
 service: celery-flower
 env: production
 basic_auth:
 username: ops_user
 password_file: /etc/prometheus/flower_password

 - job_name: rabbitmq
 static_configs:
 - targets: ["rabbitmq-exporter:9419"]

 - job_name: api
 static_configs:
 - targets: ["api:8000"]
 metrics_path: /metrics

 - job_name: worker-app
 # if workers expose a small HTTP metrics port
 static_configs:
 - targets: ["worker-metrics:9100"]

Kubernetes ServiceMonitor sketch:

yaml

# PSEUDOCODE
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
 name: flower
spec:
 selector:
 matchLabels:
 app: flower
 endpoints:
 - port: http
 path: /metrics
 interval: 15s
 basicAuth:
 username:
 name: flower-auth
 key: user
 password:
 name: flower-auth
 key: password

---

4. Useful Flower metrics

Inspect live output:

bash

curl -u ops_user:pass http://flower:5555/metrics | head

Common series (verify names on your version):

Metric (typical)	Meaning
`flower_worker_online` / worker up gauges	Worker presence
`flower_events_total` / task event counters	Event throughput
`flower_task_runtime_seconds` (histogram/summary)	Task runtime
Task succeeded/failed counters by `task` label	Outcome rates
Active task gauges	In-flight work

Relabel noisy labels (full args, ids) if cardinality explodes : prefer task name, queue, worker, state.

---

5. What else to scrape

Flower alone misses broker truth:

Source	Why
rabbitmq_exporter / RMQ prometheus plugin	`queue_messages`, consumers, DLQ depth
App metrics on API	`jobs_enqueued_total`, 429s
App metrics on workers	`jobs_succeeded_total`, lock/RL requeues
postgres (optional)	Job table lag queries via exporter/sql exporter

text

# PSEUDOCODE : app counters (prometheus_client)
jobs_enqueued_total{type=}
jobs_succeeded_total{type=}
jobs_failed_total{type=}
job_runtime_seconds{type=}
job_time_to_start_seconds{type=}
job_time_to_complete_seconds{type=}

See 15 Observability.

---

6. Example PromQL

promql

# PSEUDOCODE : adapt metric names to your /metrics output

# Task success rate (5m) by task name
sum(rate(flower_task_succeeded_total[5m])) by (task)
/
clamp_min(
 sum(rate(flower_task_succeeded_total[5m])) by (task)
 +
 sum(rate(flower_task_failed_total[5m])) by (task),
 1e-9
)

# Workers online (if gauge exists)
sum(flower_worker_online)

# RabbitMQ backlog (exporter)
sum(rabbitmq_queue_messages{queue=~"jobs\\..*"}) by (queue)

# Consumers
sum(rabbitmq_queue_consumers{queue=~"jobs\\..*"}) by (queue)

# App SLO: p95 time to complete
histogram_quantile(0.95, sum(rate(job_time_to_complete_seconds_bucket[10m])) by (le, type))

---

7. Alert rules

yaml

# PSEUDOCODE : alert rules (tune names/thresholds)
groups:
 - name: celery-flower
 rules:
 - alert: CeleryWorkersDown
 expr: sum(flower_worker_online) == 0
 for: 2m
 labels:
 severity: critical
 annotations:
 summary: No Celery workers visible to Flower

 - alert: RabbitMQZeroConsumers
 expr: sum(rabbitmq_queue_consumers{queue="jobs.default"}) == 0
 for: 2m
 labels:
 severity: critical
 annotations:
 summary: Zero consumers on jobs.default

 - alert: RabbitMQDLQNotEmpty
 expr: sum(rabbitmq_queue_messages{queue=~".*dlq.*"}) > 0
 for: 5m
 labels:
 severity: high
 annotations:
 summary: DLQ has messages

 - alert: CeleryTaskFailureRateHigh
 expr: |
 (
 sum(rate(flower_task_failed_total[5m]))
 /
 clamp_min(sum(rate(flower_task_succeeded_total[5m])) + sum(rate(flower_task_failed_total[5m])), 1e-9)
 ) > 0.05
 for: 10m
 labels:
 severity: high
 annotations:
 summary: Task failure rate > 5%

 - alert: JobQueueBacklogHigh
 expr: sum(rabbitmq_queue_messages{queue="jobs.default"}) > 1000
 for: 10m
 labels:
 severity: medium
 annotations:
 summary: jobs.default depth high

Map to checklist severities: zero consumers Critical, DLQ High, success drop High, SLO Medium (15).

---

8. Grafana dashboard ideas

Row 1 : Celery (Flower): workers online, task success/fail rate, runtime p95 Row 2 : Broker (RMQ): depth by queue, consumers, DLQ Row 3 : App: enqueue rate, time-to-start/complete p95, 429s Row 4 : Dependencies: DB errors, vendor latency

Link runbooks: scale workers, redrive DLQ, check Beat, open Flower UI (VPN).

---

9. Security

Risk	Control
Metrics leak task names / hostnames	Internal scrape only; careful labels
Flower UI exposure	Private + auth + TLS (18)
Scrape credentials	K8s Secret / password_file; not in git
High cardinality	Drop `task_id` / args from metric labels

Prometheus should scrape ClusterIP Flower, not a public Ingress /metrics.

---

10. Compose sketch

yaml

# PSEUDOCODE
services:
 flower:
 image: your-app:${TAG}
 command: >
 celery -A app.workers.celery_app.celery_app flower
 --port=5555
 --basic_auth=${FLOWER_BASIC_AUTH}
 environment:
 CELERY_BROKER_URL: ${CELERY_BROKER_URL}
 networks: [internal]
 # expose 5555 only on internal network

 prometheus:
 image: prom/prometheus:latest
 volumes:
 - ./deploy/prometheus.yml:/etc/prometheus/prometheus.yml:ro
 - ./deploy/flower_password:/etc/prometheus/flower_password:ro
 networks: [internal]

 rabbitmq-exporter:
 image: kbudde/rabbitmq-exporter:latest
 environment:
 RABBIT_URL: http://rabbitmq:15672
 RABBIT_USER: monitor
 RABBIT_PASSWORD: ${RMQ_MONITOR_PASSWORD}
 networks: [internal]

---

11. Troubleshooting

Symptom	Fix
`/metrics` 404	Upgrade Flower; ensure `prometheus-client` installed; check Flower version docs
Empty/zero series	Workers not using `-E`; Flower not connected to broker
401 on scrape	Add `basic_auth` to scrape_config
Cardinality explosion	Remove high-cardinality labels; lower history
Workers up in Flower UI but bad alerts	Prefer RMQ consumer metrics as source of truth for “can we drain?”
Success rate wrong	Combine with app `jobs_*` counters from DB-backed outcomes

---

12. Checklist

[ ] prometheus-client installed in Flower environment
[ ] Flower /metrics reachable internally
[ ] Prometheus scrape job for Flower (auth if required)
[ ] Workers send task events (-E)
[ ] RabbitMQ exporter (or plugin) for depth/consumers/DLQ
[ ] App metrics for enqueue + job SLOs
[ ] Alert rules: workers down, zero consumers, DLQ, failure rate, backlog
[ ] Grafana dashboard with Flower + RMQ + app rows
[ ] Flower UI still private; metrics not public
[ ] Document metric name mapping for your Flower version

---

Quick verify

bash

# internal
curl -u ops_user:pass http://flower:5555/metrics | grep -E 'flower_|celery_' | head

# prometheus targets UI: flower state = UP

---

External

Flower documentation
Prometheus scrape config
Celery monitoring
RabbitMQ Prometheus plugin / community exporters
Alternatives if Flower metrics are insufficient: dedicated celery-exporter sidecars (still pair with RMQ + app metrics)

Integrate Flower with Prometheus

TL;DR

Contents

1. Architecture

2. Enable Flower metrics

Optional: scrape without basic-auth friction

3. Prometheus scrape config

4. Useful Flower metrics

5. What else to scrape

6. Example PromQL

7. Alert rules

8. Grafana dashboard ideas

9. Security

10. Compose sketch

11. Troubleshooting

12. Checklist

Quick verify

See also

External