Skip to content

Broker Selection Guide

How to choose the right broker for your tasks_worker deployment.

Quick comparison

memory local postgres rabbitmq
Setup None None PostgreSQL RabbitMQ server
Message persistence ❌ RAM ❌ RAM ✅ DB ✅ Queue
Multiple worker nodes ❌ ❌ single-host ✅ ⚠️ partial
Distributed task locks ❌ ❌ ✅ DB ❌ in-memory/node
Distributed worker registry ❌ ❌ ✅ DB ❌ in-memory/node
Cluster-wide concurrency limits ❌ ❌ ✅ ❌ per-node only
Dead-letter queue ✅ ✅ ✅ ✅
Best for Tests / dev Single-host multi-process Production High-throughput / routing

When to use each broker

memory — in-process, no setup

BROKER_TYPE = "memory"
- Zero dependencies - Messages lost on process restart - Use for: unit tests, local development, single-process apps

local — multi-process on one host

BROKER_TYPE = "local"
BROKER_DSN = "127.0.0.1:50050"   # optional
- Uses multiprocessing.BaseManager for IPC - All workers on the same machine share state via the manager server - Call broker.setup() before starting workers to launch the manager - Use for: staging, local multi-process setups

BROKER_TYPE = "postgres"
BROKER_DSN = "postgresql+psycopg://user:pass@host/db"  # optional; uses FastPluggy core DB if omitted
- Multiple workers on different hosts can compete atomically via SELECT … FOR UPDATE SKIP LOCKED - Worker heartbeats, task locks, and running counters are all DB-persisted - Stale workers auto-cleaned after postgres_worker_ttl_seconds (default: 86400) - Use for: any production deployment; the simplest path to horizontal scaling

rabbitmq — advanced message routing

BROKER_TYPE = "rabbitmq"
BROKER_DSN = "amqp://guest:guest@localhost:5672/"
- AMQP naturally distributes messages across competing consumers - Durable queues survive server restarts - Limitation: worker registry, task locks, and concurrency counters are in-memory per node — not shared across a cluster. Two nodes can exceed per-topic concurrency limits or run the same exclusive task simultaneously. - Use for: deployments that already run RabbitMQ and need advanced exchange/routing features; single-node or when lock correctness is not critical

Choosing between postgres and rabbitmq for multi-node

Question Answer
Do you need cluster-wide exclusive task locks? → postgres
Do you need cluster-wide concurrency limits per topic? → postgres
Do you already run RabbitMQ and need topic-exchange routing? → rabbitmq
Simplest production setup, minimal ops overhead? → postgres (reuses FastPluggy DB)

Configuration reference

Setting Default Description
BROKER_TYPE memory Broker backend: memory, local, postgres, rabbitmq
BROKER_DSN — Connection string (optional for postgres — falls back to core DB)
postgres_worker_ttl_seconds 86400 PostgresBroker: seconds before stale worker records are cleaned up