Project Setup with Next.js 15 and BullMQ v5
Start by creating a Next.js 15 project with TypeScript support. Run the following command:
npx create-next-app@latest my-queue-app --typescript --tailwind --eslint --app --yesNavigate into the project and install the core dependencies. Use BullMQ v5.1.2, the Redis v4 client, and supporting libraries:
npm install bullmq@5.1.2 redis@4.6.14 @types/bullmq
npm install drizzle-orm postgres
npm install @bull-board/api@5.0.0 @bull-board/express@5.0.0 express@4.18.2Create a lib/redis.ts file to centralize the connection. This ensures all queue, worker, and dashboard components use identical settings and reuse the client:
import { Redis } from 'redis';
export const redisConnection = new Redis({
host: process.env.REDIS_HOST || 'localhost',
port: parseInt(process.env.REDIS_PORT || '6379'),
password: process.env.REDIS_PASSWORD,
maxRetriesPerRequest: null,
enableReadyCheck: false,
});
redisConnection.on('error', (err) => console.error('Redis Client Error', err));The maxRetriesPerRequest: null setting is required for BullMQ compatibility with the Redis v4 client. Add these environment variables to your .env.local file and never commit them.
Creating and Configuring Your First Queue
Define a dedicated queue for email processing. Create lib/queues/emailQueue.ts:
import { Queue } from 'bullmq';
import { redisConnection } from '../redis';
export const emailQueue = new Queue('email-queue', {
connection: redisConnection,
defaultJobOptions: {
attempts: 3,
backoff: {
type: 'exponential',
delay: 1000,
},
removeOnComplete: {
age: 3600 * 24 * 7, // 7 days
},
removeOnFail: {
age: 3600 * 24 * 30, // 30 days
},
},
});The defaultJobOptions apply to every job added to this queue unless overridden. The exponential backoff starts at 1 second and doubles on each retry. Retention settings prevent Redis memory from growing unbounded by automatically cleaning completed and failed jobs after the specified age in seconds.
Add a job from a Next.js Route Handler. In app/api/signup/route.ts:
import { NextResponse } from 'next/server';
import { emailQueue } from '@/lib/queues/emailQueue';
export async function POST(request: Request) {
const { userId, email } = await request.json();
await emailQueue.add('welcome-email', {
userId,
email,
template: 'welcome',
variables: { name: 'Alex' }
}, {
priority: 1,
delay: 5000, // 5 second delay for demonstration
});
return NextResponse.json({ success: true });
}This pattern decouples the HTTP response from the email delivery, allowing the API to return immediately while the job runs asynchronously.
Building the Worker to Process Jobs
The worker runs separately from your Next.js request handlers. Create workers/emailWorker.ts:
import { Worker, Job } from 'bullmq';
import { redisConnection } from '../lib/redis';
async function processWelcomeEmail(job: Job) {
const { email, template, variables } = job.data;
console.log(`Processing ${template} email for ${email}`);
// Simulate sending via Resend or Nodemailer
await new Promise(resolve => setTimeout(resolve, 1500));
await job.updateProgress(50);
// Add database update here (see Drizzle section below)
console.log('Email sent successfully');
return { delivered: true, messageId: 'mock-123' };
}
const worker = new Worker('email-queue', processWelcomeEmail, {
connection: redisConnection,
concurrency: 10,
limiter: {
max: 50,
duration: 10000, // max 50 jobs per 10 seconds
},
});
worker.on('completed', (job) => {
console.log(`Job ${job.id} completed`);
});
worker.on('failed', (job, err) => {
console.error(`Job ${job?.id} failed with error ${err.message}`);
});
worker.on('progress', (job, progress) => {
console.log(`Job ${job.id} is ${progress}% complete`);
});Run this worker with ts-node workers/emailWorker.ts during development. The concurrency: 10 controls how many jobs process simultaneously per worker instance. The limiter prevents hammering external APIs like email providers that enforce rate limits.
Adding a Monitoring Dashboard with Bull Board
Install the dashboard to visualize queues without building your own UI. Create app/api/queue-dashboard/route.ts for Express integration inside Next.js:
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';
import { emailQueue } from '@/lib/queues/emailQueue';
import express from 'express';
const serverAdapter = new ExpressAdapter();
serverAdapter.setBasePath('/api/queue-dashboard');
createBullBoard({
queues: [new BullMQAdapter(emailQueue)],
serverAdapter: serverAdapter,
});
const app = express();
app.use('/api/queue-dashboard', serverAdapter.getRouter());
export { app as GET, app as POST };Access the dashboard at http://localhost:3000/api/queue-dashboard. Protect it in production using Next.js middleware that checks for an admin session or API key:
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';
export function middleware(request: NextRequest) {
const token = request.headers.get('x-queue-token');
if (token !== process.env.QUEUE_SECRET_TOKEN) {
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
}
return NextResponse.next();
}
export const config = {
matcher: '/api/queue-dashboard/:path*',
};This dashboard shows active jobs, failed jobs, throughput, and allows manual retry or removal of stuck jobs.
Advanced Job Patterns: Flows, Priorities, and Repeating Jobs
BullMQ Flows allow orchestrating complex multi-step processes. For a report generation pipeline that depends on data export followed by PDF creation, define a parent flow:
import { FlowProducer } from 'bullmq';
const flow = new FlowProducer({ connection: redisConnection });
await flow.add({
name: 'generate-report',
queueName: 'report-queue',
data: { userId: 42, period: 'monthly' },
children: [
{
name: 'export-data',
queueName: 'data-queue',
data: { userId: 42 },
},
{
name: 'create-pdf',
queueName: 'pdf-queue',
data: { format: 'landscape' },
}
]
});The children execute first. The parent job only runs after all children complete successfully. This dependency graph prevents partial reports.
Set job priority for time-sensitive tasks. Lower numbers process first:
await notificationQueue.add('push', data, { priority: 1 }); // urgent
await emailQueue.add('digest', data, { priority: 10 }); // batchFor recurring jobs like daily analytics aggregation, use repeatable jobs:
await analyticsQueue.add('daily-summary', {}, {
repeat: {
pattern: '0 2 * * *', // every day at 2am
},
});The cron syntax follows the standard used by BullMQ's built-in scheduler.
Integrating BullMQ Jobs with Drizzle ORM
Most production jobs interact with a database. Combine BullMQ with Drizzle ORM for type-safe queries inside workers. First configure Drizzle:
// lib/db.ts
import { drizzle } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
const client = postgres(process.env.DATABASE_URL!);
export const db = drizzle(client);Inside a worker that updates user records after sending an email:
import { db } from '../lib/db';
import { users } from '../lib/schema';
import { eq } from 'drizzle-orm';
async function processOnboardingEmail(job: Job) {
const { userId } = job.data;
// Business logic...
await sendEmail(...);
await db.update(users)
.set({
onboardingEmailSent: true,
updatedAt: new Date()
})
.where(eq(users.id, userId));
return { status: 'completed' };
}Wrap database operations in transactions for jobs that perform multiple writes. This prevents partial updates when a job fails midway:
await db.transaction(async (tx) => {
await tx.update(...);
await tx.insert(...);
});Connection pooling is handled automatically by the postgres.js driver. Keep worker concurrency lower than your database's maximum connections to avoid exhausting the pool.
Error Handling, Retries, and Dead Letter Queues
BullMQ automatically retries failed jobs according to the attempts setting. For custom backoff strategies, define a function:
backoff: (attemptsMade: number, type: string, err: Error) => {
if (err.message.includes('rate-limit')) {
return 5000 * Math.pow(2, attemptsMade); // aggressive backoff
}
return 1000 * attemptsMade;
}After all retries are exhausted, move the job to the failed set. Create a dedicated dead-letter queue by listening to the failed event and re-adding to another queue for manual inspection:
worker.on('failed', async (job, err) => {
if (job) {
await deadLetterQueue.add('failed-job', {
originalJob: job.data,
originalQueue: job.queueName,
error: err.message,
stack: err.stack,
failedAt: new Date().toISOString()
});
}
});This pattern keeps your main queues clean while preserving context for debugging.
Deployment Strategies for Next.js + BullMQ Applications
Vercel does not support long-running worker processes. Deploy the Next.js frontend to Vercel and the worker separately. Use Railway, Render, or Fly.io for the worker.
With Docker, create a multi-stage Dockerfile that can run either the Next.js app or the worker based on an environment variable:
# Dockerfile.worker
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
CMD ["npm", "run", "worker"]Define the script in package.json:
"scripts": {
"worker": "ts-node workers/emailWorker.ts"
}For Redis, use a managed service such as Upstash Redis or AWS ElastiCache. Upstash offers a serverless Redis that scales to zero when idle, reducing costs for low-traffic startups.
Scale horizontally by running multiple identical worker instances. BullMQ coordinates via Redis, so adding more workers automatically distributes jobs. Set different concurrency values based on available CPU and memory on each instance.
Performance Tuning and Monitoring
Monitor Redis memory usage with the INFO memory command. BullMQ stores job data as hashes. Large payloads increase memory pressure. For jobs carrying large objects, store the payload in PostgreSQL and pass only an ID to the queue:
await emailQueue.add('newsletter', { payloadId: savedPayload.id });Use Prometheus metrics with the bullmq-prometheus exporter to track queue depth, job duration, and failure rates. Alert when the delayed set grows beyond 500 jobs or failure rate exceeds 2% over a 5-minute window.
Adjust Redis configuration for better performance under load. Increase timeout and tune tcp-keepalive. For high-throughput systems, consider Redis Cluster, though this adds operational complexity.
Securing Queues and Preventing Abuse
Never expose queue management endpoints publicly. Always use the middleware shown earlier. Sanitize all data added to jobs to prevent injection attacks if job data is later rendered or executed.
Rate-limit job creation from your API routes using the sliding window technique with Upstash Rate Limit or a simple Redis counter. This prevents a single compromised user from flooding your queues.
For sensitive jobs, encrypt PII before adding to Redis. Decrypt only inside the worker process which can run in a more restricted network environment.
Implement job uniqueness to prevent duplicate work:
await emailQueue.add('welcome-email', data, {
deduplication: {
id: `welcome-${userId}`,
ttl: 3600
}
});Complete Worked Example: Batch Invoice Generation System
Consider a SaaS application that generates monthly invoices for all customers. The complete flow involves:
- Fetching active subscriptions from the database
- Calculating usage totals
- Creating PDF invoices
- Sending invoices via email
- Recording the batch in an audit log
Implement this as a Flow with a parent job that spawns child jobs per customer to allow parallel processing while maintaining an audit trail.
The parent job code looks like this:
async function createMonthlyInvoices() {
const customers = await db.select().from(subscriptions).where(...);
const children = customers.map(customer => ({
name: 'single-invoice',
queueName: 'invoice-queue',
data: {
customerId: customer.id,
month: '2025-01'
}
}));
const flow = await flowProducer.add({
name: 'monthly-invoice-batch',
queueName: 'batch-queue',
data: { totalCustomers: customers.length, month: '2025-01' },
children
});
return flow.job.id;
}The single-invoice worker calculates line items using Drizzle, generates a PDF with pdf-lib, uploads it to S3, then sends the email with a secure download link. On completion it updates the invoice status in the database within a transaction.
This architecture processes 200 customers in parallel across workers while the parent job only completes when every invoice has been delivered, giving you a single source of truth for the entire batch.
By separating concerns into small, focused jobs you gain retry isolation. A failure for one customer does not affect others. The parent flow can implement compensation logic if needed.
Common Pitfalls and How to Avoid Them
One frequent issue is Redis connection limits. Both your Next.js API routes and workers open connections. Use a shared connection pool and monitor connected_clients.
Another pitfall is storing large JSON blobs in job data. Redis performs best with payloads under 512KB. Move large data to object storage and pass references.
Timezone handling in repeatable jobs can cause unexpected execution times. Always use UTC for cron patterns and store timezone information with the job data if localization is required.
Finally, test your workers independently of the web application. Write Jest tests that instantiate a test queue with a mocked Redis instance using redis-mock or by pointing to a test Redis container spun up with Testcontainers.
Following these patterns will give you a robust, observable, and scalable background job system that grows with your Next.js application from prototype to millions of daily jobs.