The producer/consumer pattern is the backbone of asynchronous systems. A producer creates messages and pushes them into a queue. A consumer pulls messages out and processes them. Neither knows about the other. This separation lets you scale each side independently, handle failures gracefully, and smooth out traffic spikes.
The basic pattern
// Producer: creates work items
class OrderProducer {
constructor(queue) {
this.queue = queue;
}
async submitOrder(order) {
// Validate and save order first (synchronous, user-facing)
const savedOrder = await db.orders.create(order);
// Then publish for async processing
await this.queue.publish('orders.new', {
orderId: savedOrder.id,
items: savedOrder.items,
userId: savedOrder.userId,
timestamp: Date.now()
});
return savedOrder; // User gets immediate response
}
}
// Consumer: processes work items
class OrderConsumer {
async start() {
await queue.subscribe('orders.new', async (message) => {
try {
const { orderId, items } = message.data;
await this.reserveInventory(orderId, items);
await this.calculateShipping(orderId);
await this.sendConfirmationEmail(orderId);
message.ack(); // Tell queue: processing succeeded
} catch (err) {
console.error(`Failed to process order ${message.data.orderId}:`, err);
message.nack(); // Tell queue: processing failed, redeliver
}
});
}
}The producer does not care if the consumer is fast, slow, or temporarily down. It drops the message and moves on. The consumer processes at its own pace. If messages pile up, you add more consumers.
Competing consumers (load balancing)
When one consumer cannot keep up with the message rate, you add more. Multiple consumers pull from the same queue, and each message goes to exactly one of them. This is called the competing consumer pattern because the consumers "compete" for the next message.
// Three competing consumers on the same queue
// Each message is delivered to exactly ONE consumer
// Consumer 1
queue.subscribe('emails.send', async (msg) => {
await sendEmail(msg.data);
msg.ack();
});
// Consumer 2 (same queue, same logic, different process)
queue.subscribe('emails.send', async (msg) => {
await sendEmail(msg.data);
msg.ack();
});
// Consumer 3
queue.subscribe('emails.send', async (msg) => {
await sendEmail(msg.data);
msg.ack();
});
// Queue distributes messages round-robin:
// Message 1 -> Consumer 1
// Message 2 -> Consumer 2
// Message 3 -> Consumer 3
// Message 4 -> Consumer 1
// ...This is horizontal scalingWhat is horizontal scaling?Adding more machines to handle increased load, rather than upgrading a single machine to be more powerful. for message processing. Need more throughputWhat is throughput?The number of requests or operations a system can handle per unit of time, like requests per second.? Add more consumers. A consumer crashes? The others pick up the slack, and the dead consumer's unacknowledged messages get redelivered.
Fan-out (pub/subWhat is pub/sub?A messaging pattern where senders publish events to a channel and any number of listeners receive them in real time.)
Sometimes you want every subscriber to receive every message. When an order is placed, the inventory service needs to know, the analytics service needs to know, and the notification service needs to know. Each of them needs its own copy.
// Fan-out: every subscriber gets every message
// RabbitMQ: use a fanout exchange
// Kafka: use different consumer groups
// SQS: use SNS topic -> multiple SQS queues
// Publisher
await exchange.publish('order.placed', orderData);
// Subscriber 1: Inventory service (gets ALL order.placed messages)
exchange.subscribe('order.placed', 'inventory-queue', async (msg) => {
await reserveStock(msg.data.items);
msg.ack();
});
// Subscriber 2: Analytics service (also gets ALL order.placed messages)
exchange.subscribe('order.placed', 'analytics-queue', async (msg) => {
await trackOrderMetrics(msg.data);
msg.ack();
});
// Subscriber 3: Notification service (also gets ALL messages)
exchange.subscribe('order.placed', 'notification-queue', async (msg) => {
await sendOrderNotification(msg.data.userId);
msg.ack();
});The key difference: in competing consumers, adding a consumer splits the work. In fan-out, adding a subscriber adds a new copy of all messages.
| Aspect | Competing consumers | Fan-out |
|---|---|---|
| Message delivery | Each message to ONE consumer | Each message to ALL subscribers |
| Purpose | Load balancing, parallel processing | Broadcasting, event notification |
| Add a consumer | Work splits across more workers | New subscriber gets its own copy |
| Use case | Email sending, image processing | Order events, audit logging |
| Queue model | Single queue, multiple readers | Exchange/topic with multiple queues |
| Failure impact | Other consumers pick up slack | Only that subscriber's queue backs up |
Consumer groups (Kafka concept)
Kafka introduced consumer groups, which elegantly combine both patterns. A consumer group is a named set of consumers that together process all messages from a topic. Within a group, messages are distributed (competing). Across groups, messages are duplicated (fan-out).
// Kafka consumer groups: best of both worlds
// Group "inventory-service" (3 instances for load balancing)
// Each order goes to exactly ONE instance in this group
const inventoryConsumer = new KafkaConsumer({
groupId: 'inventory-service',
topics: ['orders']
});
// Group "analytics-service" (2 instances)
// Each order ALSO goes to exactly ONE instance in this group
const analyticsConsumer = new KafkaConsumer({
groupId: 'analytics-service',
topics: ['orders']
});
// Result:
// Order #1 -> inventory-service (instance 2) AND analytics-service (instance 1)
// Order #2 -> inventory-service (instance 1) AND analytics-service (instance 2)
// Order #3 -> inventory-service (instance 3) AND analytics-service (instance 1)Each group sees every message (fan-out). Within each group, messages are distributed among instances (competing). This is how you scale individual services independently while ensuring every service gets the data it needs.
Message acknowledgment
Acknowledgment (ack) is how consumers tell the queue that processing is done. Without it, the queue has no idea whether the consumer actually handled the message or crashed mid-way.
// Manual acknowledgment (most control)
queue.subscribe('tasks', async (message) => {
try {
const result = await processTask(message.data);
await saveResult(result);
message.ack(); // "I'm done, remove this message from the queue"
} catch (err) {
if (isRetryable(err)) {
message.nack(); // "I failed, put it back for another attempt"
} else {
message.reject(); // "I failed and retrying won't help, send to DLQ"
}
}
});Three acknowledgment actions:
| Action | Meaning | Queue behavior |
|---|---|---|
ack() | Processing succeeded | Remove message permanently |
nack() | Processing failed, retry | Redeliver to same or different consumer |
reject() | Processing failed permanently | Send to dead letter queue (DLQ) |
Visibility timeout
When a consumer picks up a message, the queue hides it from other consumers for a set period. This is the visibility timeout. If the consumer does not acknowledge before the timeout expires, the queue assumes the consumer died and makes the message visible again.
// SQS-style visibility timeout
const consumer = new SQSConsumer({
queueUrl: 'https://sqs.us-east-1.amazonaws.com/123/orders',
visibilityTimeout: 30, // seconds
handleMessage: async (message) => {
// You have 30 seconds to process and ack
// For long-running tasks, extend the timeout
if (willTakeLong(message)) {
await sqs.changeMessageVisibility({
ReceiptHandle: message.ReceiptHandle,
VisibilityTimeout: 120 // extend to 2 minutes
});
}
await processOrder(message.Body);
// Auto-ack on success (SQS consumer library handles this)
}
});Getting the visibility timeout right matters:
| Timeout | Too short | Too long |
|---|---|---|
| Effect | Message redelivered while still being processed | If consumer dies, message stuck invisible for a long time |
| Result | Duplicate processing | Delayed retry, messages appear "stuck" |
| Fix | Set timeout to 2-3x expected processing time | Use heartbeat/extension for long tasks |
A common pattern is to set a moderate default (30-60 seconds) and have the consumer extend the timeout for messages that need more time. This gives you fast redelivery for crashes while supporting long-running tasks.
Scaling considerations
The producer/consumer pattern gives you independent scaling knobs. Here is how to think about it:
// Monitoring and scaling decisions
async function evaluateScaling(queueMetrics) {
const {
messageRate, // Messages published per second
processingRate, // Messages consumed per second
queueDepth, // Current messages waiting
avgProcessingTime, // Seconds per message
consumerCount // Current active consumers
} = queueMetrics;
// Queue growing? Need more consumers
if (messageRate > processingRate) {
const deficit = messageRate - processingRate;
const consumersNeeded = Math.ceil(deficit * avgProcessingTime);
console.log(`Need ${consumersNeeded} more consumers to keep up`);
}
// Queue too deep? Might need to scale producers down or consumers up
if (queueDepth > 10000) {
console.warn(`Queue depth at ${queueDepth}, consider scaling consumers`);
}
// Over-provisioned? Scale down to save costs
if (queueDepth === 0 && processingRate > messageRate * 2) {
console.log('Consumers idle, consider scaling down');
}
}The beauty of this pattern is that producers and consumers scale independently. During a sale event, you might have the same number of web servers (producers) but triple the order-processing consumers. During off-hours, you scale consumers down to save money. The queue absorbs the difference.