Redis as a Job Queue — BullMQ and Sidekiq for Reliable Asynchronous Processing

Your web app sends a confirmation email and the server freezes for 10 seconds. Or importing a CSV with 50,000 rows leaves the user staring at a loading spinner. At Meteora Web, we see this all the time: synchronous tasks that kill user experience and waste resources. The fix? Move heavy work out of the request/response cycle.

A Redis-backed job queue does exactly that: accept a request, enqueue it, respond immediately, and process the work in the background. Two battle-tested libraries are BullMQ (Node.js) and Sidekiq (Ruby). This guide covers how they work, when to use them, and how to set them up for production.

Why use Redis for job queues?

Redis is more than a cache. Its in-memory data structures — lists, sorted sets — make it perfect for atomic, persistent queues. With LPUSH/BRPOP you can create a FIFO queue in three lines. But production needs retries, priorities, scheduling, rate limiting. That’s why libraries like BullMQ and Sidekiq exist: they build robust abstractions on top of Redis to handle failures and scale.

We often use Redis for queues because it’s already in the stack (cache, sessions). No extra infrastructure. And we believe in owning your stack instead of renting it. With Redis, your data stays under control.

What's the difference between BullMQ and Sidekiq?

Practical question: if your backend is Node.js, BullMQ is the natural choice. If you use Ruby on Rails, Sidekiq is the de facto standard. Both rely on Redis and offer similar features: persistent jobs, retries, priorities, scheduling, recurring jobs. But there are key differences.

BullMQ (Node.js)

BullMQ is the modern version of Bull. Written in TypeScript, it leverages Promises and Workers. It supports concurrent queues, sandboxed job execution (separate processes), and has an open-source admin panel called Arena. We use it in Laravel projects with Vue (Node for async tasks) and it works well in clusters with Redis replication.

Sidekiq (Ruby)

Sidekiq is mature (since 2012), extremely performant, with a default thread pool that processes jobs concurrently. It has a built-in web interface, plugins for scheduling like sidekiq-cron, and native integration with Active Job in Rails. The catch: Sidekiq is multi-threaded, so your Ruby code must be thread-safe.

How to set up a queue with BullMQ?

Let's walk through a real example. Suppose we need to generate a PDF after purchase and email it. With BullMQ we create a queue and a worker.

// Install: npm install bullmq ioredis
const { Queue, Worker, QueueScheduler } = require('bullmq');

// Redis connection
const connection = { host: 'localhost', port: 6379 };

// Queue for PDFs
export const pdfQueue = new Queue('pdf-generation', { connection });

// Add a job
await pdfQueue.add('generate', {
  orderId: 123,
  userId: 456,
  email: 'customer@example.com'
}, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 1000 },
  removeOnComplete: { age: 3600 }
});

// Worker
const worker = new Worker('pdf-generation', async job => {
  const { orderId, userId, email } = job.data;
  // Simulate PDF generation (slow operation)
  console.log(`Generating PDF for order ${orderId}`);
  await generatePDF(orderId);
  // Send email
  await sendEmail(email, 'Here is your PDF');
  console.log('Job completed');
}, { connection, concurrency: 5 });

worker.on('completed', job => console.log(job.id, 'done'));
worker.on('failed', (job, err) => console.error(job.id, err));

// Scheduler for delayed or repeated jobs
new QueueScheduler('pdf-generation', { connection });

With this setup, if generation fails, BullMQ retries with exponential backoff. Jobs persist in Redis until completion or TTL. We always add removeOnComplete to avoid memory bloat.

How to implement a worker with Sidekiq?

In Rails, setup is even simpler. Use Sidekiq with Active Job.

# Gemfile
gem 'sidekiq'
gem 'sidekiq-cron' # for scheduling

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

# app/workers/pdf_generator_worker.rb
class PdfGeneratorWorker
  include Sidekiq::Worker
  sidekiq_options retry: 3, backtrace: true

  def perform(order_id, user_id, email)
    generate_pdf(order_id)
    send_email(email, 'Here is your PDF')
  end
end

# From controller
PdfGeneratorWorker.perform_async(order.id, user.id, user.email)

To run Sidekiq in production: bundle exec sidekiq. Mount Sidekiq::Web in your routes for the admin console. Pro tip: set concurrency based on your CPU cores and monitor latency with sidekiq-monitor.

How to handle errors and retries in Redis queues?

Error handling is critical. In BullMQ, set attempts and backoff per job. In Sidekiq, use retry and options like retry_count. We always recommend logging all failures and alerting the team. A job that fails after 3 attempts should go to a dead letter queue (DLQ). BullMQ has removeOnFail, Sidekiq has DeadSet.

// BullMQ – dead letter handling
worker.on('failed', async (job, err) => {
  await deadLetterQueue.add('failed', job.data, {
    attempts: 1,
    backoff: { type: 'fixed', delay: 0 }
  });
});

How to monitor the Redis queue status?

You need a dashboard. For BullMQ, Arena is open-source and lightweight. For Sidekiq, the built-in web UI is already comprehensive. In production, we add alerts on key metrics: queue length, stale jobs, failure rate. With Redis, you can expose these metrics to Prometheus via sidekiq-prometheus or bull-mq-prometheus.

What to do next

1. Identify async load in your app: emails, file generation, webhooks, data processing. Move it to a Redis queue.
2. Pick the right library: BullMQ for Node.js, Sidekiq for Ruby. Both can coexist on the same Redis instance with separate queues.
3. Configure retries with backoff and a dead letter queue. Don’t leave jobs pending forever.
4. Monitor the queue with a dashboard and alert metrics. Redis can become a bottleneck if not sized correctly.
5. Test in staging concurrency and behavior under worker crash. Use Redis persistence (AOF + RDB) to avoid data loss.

For more on the Redis ecosystem, read our pillar guide on Redis and caching strategies. If you have practical questions on BullMQ or Sidekiq, reach out.