Robust Background Processing with Laravel Horizon and Queues

Background jobs are essential for any modern application. Whether it’s sending welcome emails, processing large CSV uploads, or generating reports, offloading heavy tasks to a queue keeps your UI snappy.

But simply pushing jobs to a queue isn’t enough. You need to manage failures, batch jobs together, and monitor throughput.

1. Advanced Job Configuration

Laravel allows fine-grained control over how a job behaves when it fails or takes too long.

class ProcessVideo implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    // How many times the job may be attempted.
    public $tries = 5;

    // The number of seconds to wait before retrying the job.
    public $backoff = 30;

    // The maximum number of unhandled exceptions to allow before failing.
    public $maxExceptions = 3;
}

2. Job Batching

Laravel’s Job Batching allows you to execute a batch of jobs and then perform an action when the entire batch has completed successfully. This is perfect for chunking large imports.

use IlluminateSupportFacadesBus;
use Throwable;

$batch = Bus::batch([
    new ProcessPodcast($podcast1),
    new ProcessPodcast($podcast2),
    new ProcessPodcast($podcast3),
])->then(function (Batch $batch) {
    // All jobs completed successfully...
    Log::info('Podcast processing complete!');
})->catch(function (Batch $batch, Throwable $e) {
    // First batch job failure detected...
})->finally(function (Batch $batch) {
    // The batch has finished executing...
})->dispatch();

You can easily track the progress of this batch (e.g., to show a progress bar in the UI) by inspecting the batch ID.

3. Rate Limiting Jobs

If you are interacting with third-party APIs (like a CRM or email service), you must respect their rate limits, or your jobs will crash. You can easily rate limit jobs using Laravel’s Redis integration.

use IlluminateSupportFacadesRedis;

public function handle()
{
    Redis::throttle('any_key')->allow(10)->every(60)->then(function () {
        // Job logic...
    }, function () {
        // Could not obtain lock... release job back to queue
        return $this->release(10);
    });
}

4. Laravel Horizon

If you are using Redis for your queues, Laravel Horizon is an absolute necessity. Horizon provides a beautiful dashboard and code-driven configuration for your Redis queues.

With Horizon, you can:

• View real-time metrics on job throughput and runtime.
• Auto-scale worker processes based on queue length (e.g., spinning up more workers if the emails queue gets backed up).
• Monitor job failures with detailed stack traces.

To install:

composer require laravel/horizon
php artisan horizon:install

By leveraging these advanced techniques, you guarantee that your application processes data reliably in the background, regardless of load spikes.