Integrating LLMs into Monoliths: Decoupling with Queues

API Latency and Web Thread Starvation

Calling LLM APIs (like Gemini or OpenAI) takes time (often 2-5 seconds). If these calls are executed synchronously inside your web server’s request thread, the thread remains blocked, leading to gateway timeouts under moderate traffic.

Case Study: Monolithic Portal Outages

A company added an AI text analyzer to their main application. During peak hours, the web server ran out of available thread pools, causing the entire site to go offline with HTTP 502 Bad Gateway errors.

The Bug: Synchronous API Request Blocking

The analyzer was called directly inside the controller endpoint:

// Blocked Controller Thread
public function analyzeText(Request $request) {
    $text = $request->input('text');
    $analysis = $this->llmClient->call($text); // Blocks here for 5 seconds!
    return response()->json($analysis);
}

The Fix: Decoupling with Redis & Queue Workers

We decoupled the AI integration by shifting LLM execution to asynchronous background workers using a Redis queue:

// 1. Controller pushes job to queue and returns instantly
public function analyzeText(Request $request) {
    $jobId = uniqid('job_');
    Queue::push(new ProcessTextAnalysis($jobId, $request->input('text')));
    return response()->json(['status' => 'processing', 'job_id' => $jobId]);
}

// 2. Background worker processes the job asynchronously
class ProcessTextAnalysis {
    public function handle() {
        $result = $this->llmClient->call($this->text);
        $this->saveResult($this->jobId, $result);
    }
}

This async architecture kept web controller response times under 20ms and eliminated server crash loops.