Stop Blocking Your Users: Queue Long-Running Laravel AI Agents with Batching, Retries, and Rate Limits

An LLM call does not belong in most HTTP request cycles.

A provider may take 5 to 30 seconds to return a response. During that time, your PHP-FPM worker remains occupied. A few simultaneous requests can exhaust the worker pool, increase REST API latency, and trigger upstream timeouts.

Laravel gives you a better model. Accept the request, create an agent_runs record, dispatch a job, and return 202 Accepted. The user interface can poll for progress or listen for a completion event.

This approach uses the same queue primitives that make Laravel a productive PHP web framework. You get retries, batching, rate limits, failure handling, and observability without building a separate worker system.

Why LLM calls block your API

An LLM request is a network-bound operation with unpredictable duration. The model may need to generate thousands of tokens, call tools, or retry internally before returning a response.

If your controller calls an agent directly, the request remains open:

public function store(Request $request)
{
    $response = (new SupportAgent)
        ->prompt($request->string('message'));

    return response()->json([
        'answer' => (string) $response,
    ]);
}

That code is simple, but the PHP-FPM process cannot serve another request until the provider responds. Under load, workers queue behind slow AI calls. Your unrelated endpoints become slow as well.

Long requests also collide with several timeout layers:

• PHP-FPM request limits
• Nginx or load balancer timeouts
• API gateway timeouts
• Provider HTTP timeouts
• Client-side timeouts

The fix is not always a larger timeout. The fix is moving the long operation outside the request cycle.

Store the run before dispatching the job

Create a table that represents the lifecycle of an agent run. Keep the user-facing state in your database rather than trying to infer it from queue internals.

Schema::create('agent_runs', function (Blueprint $table) {
    $table->id();
    $table->foreignId('user_id')->constrained()->cascadeOnDelete();
    $table->string('status')->default('queued');
    $table->text('input');
    $table->json('output')->nullable();
    $table->json('usage')->nullable();
    $table->json('error_context')->nullable();
    $table->unsignedInteger('attempts')->default(0);
    $table->unsignedInteger('duration_ms')->nullable();
    $table->timestamp('started_at')->nullable();
    $table->timestamp('completed_at')->nullable();
    $table->timestamps();
});

Your REST endpoint can now respond immediately:

use App\Jobs\RunSupportAgent;
use App\Models\AgentRun;
use Illuminate\Http\Request;

public function store(Request $request)
{
    $data = $request->validate([
        'message' => ['required', 'string', 'max:20000'],
    ]);

    $run = AgentRun::create([
        'user_id' => $request->user()->id,
        'input' => $data['message'],
        'status' => 'queued',
    ]);

    RunSupportAgent::dispatch($run->id);

    return response()->json([
        'id' => $run->id,
        'status' => $run->status,
        'status_url' => route('agent-runs.show', $run),
    ], 202);
}

The 202 status tells the client that the request was accepted but is not complete. This is a clean foundation when you build a REST API with PHP.

Run the Laravel AI SDK inside a queued job

Laravel’s AI SDK provides a consistent interface for agents and providers. The job owns the slow operation.

namespace App\Jobs;

use App\Ai\Agents\SupportAgent;
use App\Events\AgentRunCompleted;
use App\Models\AgentRun;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Middleware\RateLimited;
use Illuminate\Queue\Middleware\WithoutOverlapping;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Log;
use Throwable;

class RunSupportAgent implements ShouldQueue
{
    use Queueable;

    public int $timeout = 180;
    public int $tries = 5;
    public array $backoff = [10, 30, 90];

    public function __construct(
        public int $runId,
    ) {}

    public function middleware(): array
    {
        return [
            new RateLimited('ai-provider'),
            (new WithoutOverlapping("agent-run:{$this->runId}"))
                ->releaseAfter(15)
                ->expireAfter(240),
        ];
    }

    public function handle(): void
    {
        $run = AgentRun::findOrFail($this->runId);

        if ($run->status === 'completed') {
            return;
        }

        $run->update([
            'status' => 'running',
            'started_at' => $run->started_at ?? now(),
            'attempts' => $run->attempts + 1,
        ]);

        $started = microtime(true);

        $response = (new SupportAgent)->prompt(
            $run->input,
            timeout: 150,
        );

        $durationMs = (int) ((microtime(true) - $started) * 1000);

        DB::transaction(function () use ($run, $response, $durationMs) {
            $run->update([
                'status' => 'completed',
                'output' => ['text' => (string) $response],
                'usage' => data_get($response, 'usage'),
                'duration_ms' => $durationMs,
                'completed_at' => now(),
            ]);
        });

        AgentRunCompleted::dispatch($run->fresh());
    }

    public function failed(?Throwable $exception): void
    {
        AgentRun::whereKey($this->runId)->update([
            'status' => 'failed',
            'error_context' => [
                'type' => $exception ? $exception::class : null,
                'message' => $exception?->getMessage(),
                'failed_at' => now()->toISOString(),
            ],
        ]);

        Log::error('AI agent run failed', [
            'run_id' => $this->runId,
            'exception' => $exception?->getMessage(),
        ]);
    }
}

The job-level $timeout should be shorter than the queue connection’s retry_after value. Otherwise, a worker may retry a job while the original process is still running.

The $tries and $backoff values also matter. Rate limiting and overlap middleware release jobs back to the queue, and those releases consume attempts. Set the values high enough for your expected delays.

The failed() method runs after the job exhausts its retries. Persist the error context so the UI and support team can distinguish provider failures, timeouts, invalid input, and application errors.

Protect against duplicate runs

Retries are normal. Duplicate side effects are not.

For a job dispatched outside a batch, implement ShouldBeUnique when only one run for a resource should be queued:

use Illuminate\Contracts\Queue\ShouldBeUnique;

class RunSupportAgent implements ShouldQueue, ShouldBeUnique
{
    public int $uniqueFor = 3600;

    public function uniqueId(): string
    {
        return "support-agent:{$this->runId}";
    }
}

Use WithoutOverlapping when several jobs may exist but only one should execute for a given run, document, customer, or provider. The middleware uses an atomic lock and releases competing jobs.

There is an important distinction: unique job constraints do not apply to jobs inside Bus::batch(). Deduplicate batch inputs before dispatching. Use WithoutOverlapping for runtime protection within a batch.

Rate-limit provider calls gracefully

OpenAI, Anthropic, and other providers can return 429 responses. Retrying immediately increases pressure and can extend an outage.

Define a named limiter:

use Illuminate\Cache\RateLimiting\Limit;
use Illuminate\Support\Facades\RateLimiter;

public function boot(): void
{
    RateLimiter::for('ai-provider', function () {
        return Limit::perMinute(60)->by('shared-ai-provider');
    });
}

Attach it to the job with RateLimited:

use Illuminate\Queue\Middleware\RateLimited;

public function middleware(): array
{
    return [
        (new RateLimited('ai-provider'))->releaseAfter(30),
    ];
}

Tune the limit to your provider account and model. You may define separate limiters for OpenAI and Anthropic, or segment limits by tenant.

For repeated provider exceptions, Laravel’s ThrottlesExceptions middleware can pause a job after a threshold. Pair it with retryUntil() when you want a time window instead of a fixed attempt count.

Batch thousands of embeddings or summaries

A single agent run is only one use case. AI workloads often involve thousands of independent tasks.

Use Bus::batch() to fan out document summaries or embeddings. A batch tracks progress and supports completion callbacks.

namespace App\Jobs;

use App\Ai\Agents\DocumentSummarizer;
use App\Models\Document;
use Illuminate\Bus\Batchable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;

class SummarizeDocument implements ShouldQueue
{
    use Batchable, Queueable;

    public int $timeout = 180;
    public int $tries = 4;
    public array $backoff = [15, 45, 120];

    public function __construct(
        public int $documentId,
    ) {}

    public function handle(): void
    {
        if ($this->batch()?->cancelled()) {
            return;
        }

        $document = Document::findOrFail($this->documentId);

        if ($document->summary !== null) {
            return;
        }

        $response = (new DocumentSummarizer)->prompt($document->content);

        $document->update([
            'summary' => (string) $response,
        ]);
    }
}

Dispatch the batch from a service or controller:

use App\Jobs\SummarizeDocument;
use App\Models\Document;
use Illuminate\Bus\Batch;
use Illuminate\Support\Facades\Bus;
use Throwable;

$jobs = Document::query()
    ->whereNull('summary')
    ->pluck('id')
    ->unique()
    ->map(fn (int $id) => new SummarizeDocument($id))
    ->all();

$batch = Bus::batch($jobs)
    ->name('Generate document summaries')
    ->allowFailures()
    ->then(function (Batch $batch) {
        // All jobs completed successfully.
    })
    ->catch(function (Batch $batch, Throwable $exception) {
        // Record the first batch failure.
    })
    ->finally(function (Batch $batch) {
        // Mark the parent operation as finished.
    })
    ->dispatch();

For very large datasets, do not load every document ID during an HTTP request. Dispatch loader jobs that add work to the current batch in chunks. Laravel supports adding jobs from within a batch using $this->batch()->add(...).

The same pattern works for embeddings:

use Laravel\Ai\Embeddings;

$response = Embeddings::for([$document->content])
    ->generate();

$document->update([
    'embedding' => $response->embeddings[0],
]);

Chain multi-step agent pipelines

Some workflows need ordered stages rather than parallel work. For example, summarize a document, translate the summary, then store the final result.

Use Bus::chain():

use App\Jobs\StoreTranslatedSummary;
use App\Jobs\SummarizeDocument;
use App\Jobs\TranslateSummary;
use Illuminate\Support\Facades\Bus;

Bus::chain([
    new SummarizeDocument($document->id),
    new TranslateSummary($document->id, language: 'fr'),
    new StoreTranslatedSummary($document->id),
])->dispatch();

If one job fails, later jobs do not run. Add a catch() callback when you need to update the parent agent_runs record or notify the user.

For partial completion, make every stage idempotent. Store a stage result before moving forward. On retry, check whether that stage already completed. This prevents a provider call or database write from running twice after a worker timeout.

Return results through polling or events

Your polling endpoint can remain small:

public function show(Request $request, AgentRun $run)
{
    abort_unless($run->user_id === $request->user()->id, 404);

    return response()->json([
        'id' => $run->id,
        'status' => $run->status,
        'output' => $run->status === 'completed'
            ? $run->output
            : null,
        'error' => $run->status === 'failed'
            ? $run->error_context
            : null,
    ]);
}

Expose routes such as:

Route::post('/agent-runs', [AgentRunController::class, 'store']);
Route::get('/agent-runs/{run}', [AgentRunController::class, 'show'])
    ->name('agent-runs.show');

Polling is dependable and easy to support. For a more responsive interface, broadcast an AgentRunCompleted event with Laravel’s broadcasting tools. Reverb and Echo can notify the browser without repeated requests.

Measure tokens, duration, and queue health

AI workloads need more than application logs. Record the run ID, provider, model, token usage, duration, attempt count, and failure type.

The AI SDK exposes lifecycle events such as AgentPrompted and EmbeddingsGenerated. Use them to centralize usage logging rather than duplicating instrumentation across jobs.

For queue health, Horizon provides Redis queue throughput, runtime, wait time, failures, tags, and worker configuration. Give AI jobs their own queue when they should not compete with emails or user-facing tasks.

Telescope is useful during development. Its job, batch, HTTP client, exception, and log watchers help trace a run from dispatch to provider response.

Keep the request fast

A reliable AI feature has a short request path:

1. Validate the input.
2. Create an agent_runs record.
3. Dispatch a queue job.
4. Return 202 Accepted.
5. Poll or broadcast progress.
6. Persist the final result and usage.
7. Retry transient failures with backoff.
8. Mark permanent failures with useful context.

That design keeps PHP-FPM available for normal traffic while workers handle the slow part. Laravel’s queues, batching, middleware, and AI SDK give you the core PHP developer tools to build the workflow without adding unnecessary infrastructure.

The principle is simple: accept quickly, process deliberately, and make every AI step safe to retry.