Provider Failover in Laravel: Keep Your AI Features Alive When OpenAI or Anthropic Goes Down

AI features inherit the availability of their providers.

A valid OpenAI request can still fail because of a timeout, rate limit, regional outage, or temporary overload. Anthropic can experience the same issues. A production application should treat provider failure as a normal operating condition.

The Laravel AI SDK supports provider failover directly. You can pass an ordered provider array, and the SDK moves to the next provider when a failover-eligible error occurs:

$response = (new SupportAgent)->prompt(
    'Summarize this support ticket.',
    provider: [
        Lab::OpenAI,
        Lab::Anthropic,
        Lab::Ollama,
    ],
);

For simple cases, this may be enough. Production systems usually need more control. They need per-tenant limits, retries, circuit-breaker cooldowns, queue backoff, monitoring, and tests that exercise failure paths.

This tutorial builds that layer around Laravel AI SDK.

Configure OpenAI, Anthropic, and Ollama

Install the SDK if it is not already present:

composer require laravel/ai
php artisan vendor:publish --provider="Laravel\Ai\AiServiceProvider"
php artisan migrate

Set credentials for the hosted providers:

OPENAI_API_KEY=your-openai-key
ANTHROPIC_API_KEY=your-anthropic-key

# Ollama can run locally or behind an internal network address.
OLLAMA_BASE_URL=http://127.0.0.1:11434

Define the providers in config/ai.php:

<?php

return [
    'providers' => [
        'openai' => [
            'driver' => 'openai',
            'key' => env('OPENAI_API_KEY'),
        ],

        'anthropic' => [
            'driver' => 'anthropic',
            'key' => env('ANTHROPIC_API_KEY'),
        ],

        'ollama' => [
            'driver' => 'ollama',
            'url' => env('OLLAMA_BASE_URL'),
        ],
    ],
];

The model names belong in application configuration because they are part of your resilience policy:

// config/ai-resilience.php

return [
    'providers' => [
        'openai',
        'anthropic',
        'ollama',
    ],

    'models' => [
        'openai' => env('AI_OPENAI_MODEL', 'gpt-4o-mini'),
        'anthropic' => env('AI_ANTHROPIC_MODEL', 'claude-haiku-4-5-20251001'),
        'ollama' => env('AI_OLLAMA_MODEL', 'llama3.1'),
    ],

    'timeout' => 30,

    'retries' => 2,

    'cooldown_seconds' => 60,
];

Use Lab::OpenAI, Lab::Anthropic, and Lab::Ollama when you want enum-backed provider names:

use Laravel\Ai\Enums\Lab;

Add a failover service

Keep provider selection out of controllers and jobs. A service gives every AI feature the same behavior.

This example:

• skips providers in their cooldown window;
• retries transient errors;
• adds exponential backoff with jitter;
• moves to the next provider;
• records failover events in structured logs;
• supports a local Ollama fallback.

<?php

namespace App\Services;

use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Log;
use Laravel\Ai\Enums\Lab;
use Throwable;

class AiFailover
{
    public function __construct(
        private SupportAgent $agent,
    ) {}

    public function generate(string $tenantId, string $prompt): string
    {
        $providers = collect(config('ai-resilience.providers'))
            ->reject(fn (string $provider) => Cache::has($this->cooldownKey($provider)))
            ->values()
            ->all();

        if ($providers === []) {
            throw new \RuntimeException('No AI providers are currently available.');
        }

        $lastException = null;

        foreach ($providers as $providerIndex => $provider) {
            $attempts = config('ai-resilience.retries') + 1;

            for ($attempt = 0; $attempt < $attempts; $attempt++) {
                try {
                    $response = $this->agent->prompt(
                        $prompt,
                        provider: $this->lab($provider),
                        model: config("ai-resilience.models.$provider"),
                        timeout: config('ai-resilience.timeout'),
                    );

                    if ($providerIndex > 0) {
                        Log::notice('ai.provider.failover_succeeded', [
                            'tenant_id' => $tenantId,
                            'provider' => $provider,
                        ]);
                    }

                    Cache::forget($this->cooldownKey($provider));

                    return (string) $response;
                } catch (Throwable $exception) {
                    $lastException = $exception;

                    if (! $this->isTransient($exception)) {
                        throw $exception;
                    }

                    if ($attempt + 1 < $attempts) {
                        $baseDelay = 2 ** $attempt;
                        $jitter = random_int(0, 250);
                        usleep(($baseDelay * 1000 + $jitter) * 1000);
                    }
                }
            }

            Cache::put(
                $this->cooldownKey($provider),
                true,
                now()->addSeconds(config('ai-resilience.cooldown_seconds'))
            );

            Log::warning('ai.provider.failover', [
                'tenant_id' => $tenantId,
                'failed_provider' => $provider,
                'next_provider' => $providers[$providerIndex + 1] ?? null,
                'exception' => get_class($lastException),
            ]);
        }

        throw $lastException ?? new \RuntimeException('AI generation failed.');
    }

    private function cooldownKey(string $provider): string
    {
        return "ai-provider-cooldown:$provider";
    }

    private function lab(string $provider): Lab
    {
        return match ($provider) {
            'openai' => Lab::OpenAI,
            'anthropic' => Lab::Anthropic,
            'ollama' => Lab::Ollama,
        };
    }

    private function isTransient(Throwable $exception): bool
    {
        $class = class_basename($exception);
        $message = strtolower($exception->getMessage());
        $code = (int) $exception->getCode();

        return in_array($code, [408, 429, 500, 502, 503, 504], true)
            || str_contains($class, 'RateLimited')
            || str_contains($class, 'Overloaded')
            || str_contains($class, 'Connection')
            || str_contains($class, 'Timeout')
            || str_contains($message, 'timed out')
            || str_contains($message, 'temporarily unavailable');
    }
}

The SDK already classifies several provider failures as failover-eligible. The service above adds policies the SDK does not own: same-provider retries, jitter, cooldowns, and tenant context.

Do not retry malformed prompts, invalid tool arguments, authentication failures, or schema errors. Retrying those requests adds cost without improving availability.

Use a circuit-breaker cooldown

The cache flag acts as a simple circuit breaker.

When OpenAI fails repeatedly, the service stores a cooldown key. Other requests skip OpenAI for 60 seconds and go directly to Anthropic. After the key expires, OpenAI receives traffic again.

Use a shared cache such as Redis in production:

// config/cache.php

'limiter' => 'redis',

A shared cache matters when multiple PHP workers or servers process requests. Otherwise, each worker will maintain a separate view of provider health.

For stricter half-open behavior, use a short cooldown and allow one probe request after expiry. Clear the cooldown after a successful response.

Rate-limit each tenant

Provider failover should not allow one tenant to consume the entire backup capacity. Laravel’s RateLimiter works well as route middleware.

<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\RateLimiter;
use Symfony\Component\HttpFoundation\Response;

class LimitTenantAi
{
    public function handle(Request $request, Closure $next): Response
    {
        $tenantId = (string) $request->user()->tenant_id;
        $key = "ai-tenant:$tenantId";

        $allowed = RateLimiter::attempt(
            $key,
            30,
            fn () => true,
            60,
        );

        if (! $allowed) {
            return response()->json([
                'message' => 'AI request limit exceeded.',
                'retry_after' => RateLimiter::availableIn($key),
            ], 429);
        }

        return $next($request);
    }
}

Apply this middleware to AI routes. Keep the limit separate from provider limits. Your application controls tenant usage. The provider controls account-level usage.

Queue expensive generations

Long summaries, document processing, and batch content generation should not block an HTTP request. Laravel queues provide retries and delayed backoff for these workloads.

<?php

namespace App\Jobs;

use App\Services\AiFailover;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Queue\Middleware\ThrottlesExceptions;

class GenerateSummary implements ShouldQueue
{
    use Queueable;

    public int $tries = 5;
    public int $timeout = 180;

    public function __construct(
        public string $tenantId,
        public int $documentId,
        public string $content,
    ) {}

    public function handle(AiFailover $ai): void
    {
        $summary = $ai->generate(
            $this->tenantId,
            "Summarize this document in five bullet points:\n\n{$this->content}",
        );

        // Persist the summary for the document.
    }

    public function backoff(): array
    {
        return [5, 15, 45, 120];
    }

    public function middleware(): array
    {
        return [
            (new ThrottlesExceptions(3, 300))
                ->by("ai-tenant:{$this->tenantId}")
                ->backoff(10),
        ];
    }
}

The service handles short in-process retries. The job handles longer recovery windows. Set the worker timeout below the queue connection’s retry_after value to avoid duplicate processing.

Keep streaming responses useful

The AI SDK supports streaming and provider failover:

use App\Ai\Agents\SupportAgent;
use Laravel\Ai\Enums\Lab;
use Illuminate\Support\Facades\Route;

Route::get('/chat/stream', function () {
    return (new SupportAgent)->stream(
        'Explain the current incident status.',
        provider: [
            Lab::OpenAI,
            Lab::Anthropic,
            Lab::Ollama,
        ],
    );
});

Failover works when the provider fails before the stream begins. It cannot transparently resume a response after the client has already received partial tokens.

For that reason:

• send a provider handoff event before content when possible;
• retry only before the first chunk is emitted;
• avoid appending a second complete answer to a partial answer;
• let the client replace the stream if a reconnect occurs;
• include a request ID so duplicate generations can be detected.

For critical streaming features, buffer the first provider’s response until you have received enough content to commit it. This increases latency but prevents broken partial output.

Observe failover with Nightwatch

Failover is useful only when you know it is happening.

The service emits structured log records such as ai.provider.failover. Laravel Nightwatch observes application logs, outgoing requests, cache interactions, jobs, and exceptions.

Track at least:

• tenant ID;
• failed provider;
• selected fallback provider;
• model;
• retry count;
• request duration;
• exception class;
• estimated token cost.

Create a Nightwatch alert for repeated failover events. A single fallback may be normal. A sustained increase usually means a provider outage, a bad deployment, or an account quota problem.

Never log prompts or customer content by default. Use request IDs and hashed tenant identifiers when the prompt may contain sensitive information.

Test the failure path

A happy-path fake does not prove failover works. Your test must force the primary provider to fail.

The AI SDK provides agent fakes for deterministic unit tests:

use App\Ai\Agents\SupportAgent;
use Laravel\Ai\Prompts\AgentPrompt;

SupportAgent::fake(function (AgentPrompt $prompt) {
    return 'Fallback response.';
});

$response = (new SupportAgent)->prompt('Test prompt.');

SupportAgent::assertPrompted('Test prompt.');

For the failover service, inject a small gateway so tests can model provider-specific failures:

final class FakeAiGateway
{
    public function __construct(
        private array $responses,
    ) {}

    public function generate(string $provider, string $prompt): string
    {
        $response = array_shift($this->responses[$provider]);

        if ($response instanceof Throwable) {
            throw $response;
        }

        return $response;
    }
}

Then test a 503-style failure followed by a successful Anthropic response:

use Illuminate\Support\Facades\Cache;
use Tests\TestCase;

test('fails over when the primary provider is unavailable', function () {
    Cache::fake();

    $gateway = new FakeAiGateway([
        'openai' => [new RuntimeException('Provider unavailable')],
        'anthropic' => ['Generated by the fallback provider.'],
    ]);

    $service = new AiFailoverUsingGateway($gateway);

    expect($service->generate('tenant-1', 'Write a summary.'))
        ->toBe('Generated by the fallback provider.');

    Cache::assertPut('ai-provider-cooldown:openai', true);
});

At the HTTP boundary, use Http::fake when your configured transport uses Laravel’s HTTP client:

Http::fake([
    'api.openai.com/*' => Http::response([], 503),
    'api.anthropic.com/*' => Http::response([
        'content' => [['text' => 'Fallback response.']],
    ], 200),
]);

Test each important branch:

• OpenAI returns 429.
• OpenAI times out.
• OpenAI returns a 5xx response.
• Anthropic succeeds after OpenAI fails.
• All providers fail.
• A cooldown prevents OpenAI from being called.
• A non-transient error does not trigger fallback.
• A queued job retries with the expected backoff.
• A tenant exceeds its rate limit.

Account for cost and latency

Failover is not free.

A retry adds latency. A second provider may charge a different price. A local model may reduce cost but produce weaker reasoning, shorter context handling, or less reliable structured output.

Define fallback tiers by feature:

• critical support responses: OpenAI → Anthropic;
• low-risk summaries: OpenAI → Anthropic → Ollama;
• structured financial actions: fail closed instead of using a weaker model;
• background content drafts: queue the work and tolerate higher latency.

The right policy depends on the result’s business impact. Provider failover should keep your application available without hiding meaningful quality or cost changes from you or your users.