Laravel AI Failover: Switch Providers When an LLM Times Out

Your Laravel app calls an LLM provider, the request hangs for eight seconds, then twenty, then finally dies with a timeout. The user sees a spinner. Your first instinct is simple: “Call the backup provider if the first one fails.”

That instinct is right, but the implementation is where systems get messy.

A naive failover can double your AI costs, hide real bugs, send sensitive prompts to an unapproved provider, retry invalid requests forever, or fail over after you have already started streaming output to the user. LLM failover is not just exception handling. It is a product, latency, cost, and reliability decision.

This article walks through a production-oriented approach to Laravel AI failover: how to switch LLM providers when one times out, when not to switch, and how to build the system so the fallback actually helps instead of becoming a second outage.

TL;DR

If you only take away a few rules:

Do not fail over on every exception. Timeouts and provider outages are failover candidates. Bad input, content policy refusals, and schema violations usually are not.
Put a provider-agnostic contract in front of your LLM calls. Failover logic should not be scattered across controllers and jobs.
Give each provider its own timeout budget. Do not let the primary provider consume the entire user-facing deadline.
Use a circuit breaker. Otherwise, a failing provider will keep receiving traffic and dragging down every request.
For streaming, fail over only before the first visible token. Once the user sees output, seamless provider switching is mostly over.
Validate fallback output. A faster but incompatible answer is still a broken feature.
Test failover with fake providers. Do not wait for a real provider outage to discover that your fallback path was never exercised.

📋 Table of Contents

1. A Timeout Is a Business Decision, Not Just an Exception
2. Put a Contract Between Laravel and the Model
3. The First Failover Version Everyone Writes, and Why It Hurts
4. Give Each Provider a Smaller Timeout Than You Think
5. Add a Circuit Breaker Before You Add More Providers
6. Rate Limits Need a Different Failover Rule Than Timeouts
7. Streaming Changes When Failover Is Allowed
8. Make the Fallback Model Speak the Same Product Language
9. Make Failover Observable and Testable
Choosing the Right Failover Strategy
What I Would Actually Ship

1. A Timeout Is a Business Decision, Not Just an Exception

Scenario:
Your product has a chat assistant. A user asks a question. Your app calls the primary LLM provider. The provider is having a bad day. The request hangs. Your HTTP client waits for the default timeout, maybe thirty seconds, maybe sixty. The user refreshes. Now you have two slow requests instead of one.

Why it matters:
The provider’s timeout is not the same thing as your user’s patience. A provider may be technically “working” while still being useless for your UX. If your interactive feature needs an answer in five to eight seconds, letting a provider hang for thirty seconds is not resilience. It is delayed failure.

Solution:
Decide what kind of request you are handling before deciding how long to wait.

There are usually three broad classes:

Interactive requests
The user is waiting. Latency matters more than completeness. You may prefer streaming, a shorter timeout, and a faster fallback model.
Background jobs
Nobody is staring at a spinner. You can tolerate longer timeouts, retries, queues, and exponential backoff.
Critical structured operations
The LLM output feeds another system: JSON extraction, tool calling, classification, moderation, database writes. Correctness matters more than raw speed.

These need different failover behavior.

For an interactive Laravel route, you might define something like this:

// config/llm.php

return [
    'interactive_deadline_seconds' => 8,

    'providers' => [
        'primary' => [
            'name' => 'primary',
            'timeout_seconds' => 5,
            'connect_timeout_seconds' => 2,
        ],

        'fallback' => [
            'name' => 'fallback',
            'timeout_seconds' => 3,
            'connect_timeout_seconds' => 2,
        ],
    ],
];

The important part is not the exact numbers. The important part is that the primary provider does not get the whole budget.

If your interactive request has an eight-second budget and the primary provider gets eight seconds, there is no time left for failover. The fallback may succeed technically, but the user has already abandoned the request.

💡 Practical note:
For user-facing AI features, I would rather have a partial or degraded answer quickly than a perfect answer after the user has refreshed three times.

2. Put a Contract Between Laravel and the Model

Scenario:
You started with one LLM provider. The call lives inside a controller, maybe a service class, maybe a job. Then you add a backup provider. Now you have if ($provider === 'openai') in one place, match ($driver) in another, and a third place that assumes the response shape is always the same.

Why it matters:
Failover logic needs a clean seam. If your application code knows too much about provider-specific request formats, error formats, and response fields, switching providers becomes a bug farm.

Solution:
Define a small contract.

You do not need a massive abstraction. Start with something deliberately narrow:

namespace App\Ai;

final readonly class Prompt
{
    public function __construct(
        public array $messages,
        public ?string $system = null,
        public ?string $jsonSchema = null,
        public bool $interactive = true,
    ) {}
}

final readonly class CompletionResult
{
    public function __construct(
        public string $content,
        public string $provider,
        public ?string $model,
        public ?string $finishReason,
        public array $usage = [],
    ) {}
}

interface CompletionProvider
{
    public function name(): string;

    public function supports(Prompt $prompt): bool;

    public function complete(Prompt $prompt): CompletionResult;
}

This is intentionally boring. That is good.

The supports() method matters more than it may look. Not every provider can handle every request. For example:

final class StructuredJsonProvider implements CompletionProvider
{
    public function supports(Prompt $prompt): bool
    {
        if ($prompt->jsonSchema !== null && ! $this->supportsStructuredJson) {
            return false;
        }

        return true;
    }
}

If your primary provider supports strict JSON schema output but your fallback does not, you do not want to fail over blindly and then crash your downstream parser.

⚠️ Gotcha:
Avoid putting provider-specific options into the shared Prompt object unless your failover layer knows how to translate them. Things like tool calling, response format, embedding dimensions, and max token limits are common sources of silent incompatibility.

A provider implementation might use Laravel’s HTTP client directly:

namespace App\Ai\Providers;

use App\Ai\CompletionProvider;
use App\Ai\CompletionResult;
use App\Ai\Exceptions\LlmTimeoutException;
use App\Ai\Exceptions\LlmUnavailableException;
use App\Ai\Prompt;
use Illuminate\Http\Client\ConnectionException;
use Illuminate\Support\Facades\Http;

final class HttpCompletionProvider implements CompletionProvider
{
    public function __construct(
        private readonly string $name,
        private readonly string $endpoint,
        private readonly string $apiKey,
        private readonly int $timeoutSeconds,
        private readonly int $connectTimeoutSeconds,
    ) {}

    public function name(): string
    {
        return $this->name;
    }

    public function supports(Prompt $prompt): bool
    {
        return true;
    }

    public function complete(Prompt $prompt): CompletionResult
    {
        try {
            $response = Http::withToken($this->apiKey)
                ->timeout($this->timeoutSeconds)
                ->connectTimeout($this->connectTimeoutSeconds)
                ->post($this->endpoint, [
                    'model' => config("llm.providers.{$this->name}.model"),
                    'messages' => $prompt->messages,
                    'temperature' => 0.2,
                ]);
        } catch (ConnectionException $e) {
            throw new LlmTimeoutException(
                "LLM provider {$this->name} timed out.",
                previous: $e
            );
        }

        if ($response->status() === 429) {
            throw new LlmUnavailableException(
                "LLM provider {$this->name} returned rate limit."
            );
        }

        if ($response->serverError()) {
            throw new LlmUnavailableException(
                "LLM provider {$this->name} returned {$response->status()}."
            );
        }

        $body = $response->json();

        return new CompletionResult(
            content: data_get($body, 'choices.0.message.content', ''),
            provider: $this->name,
            model: data_get($body, 'model'),
            finishReason: data_get($body, 'choices.0.finish_reason'),
            usage: data_get($body, 'usage', []),
        );
    }
}

The exact payload will differ by provider. That is fine. The point is that your Laravel application depends on CompletionProvider, not on the quirks of one vendor API.

Once this contract exists, failover becomes a composition problem instead of a copy-paste problem.

3. The First Failover Version Everyone Writes, and Why It Hurts

Scenario:
You add a backup provider with the most obvious code possible:

try {
    return $primary->complete($prompt);
} catch (Throwable $e) {
    return $fallback->complete($prompt);
}

It works in the happy path. Then production teaches you the hard lessons.

Why it matters:
This version fails over on everything: invalid requests, malformed JSON, unsupported parameters, content policy refusals, authentication errors, and even bugs in your own code. It also has no memory. If the primary provider is down, every request still pays the primary timeout first.

Solution:
Classify failures.

A useful mental model:

Failure type	Fail over?	Why
Connection timeout	Usually yes	Provider did not respond in time.
Slow first token	Yes, if interactive	User experience is being damaged.
500/502/503/529-style provider error	Usually yes	Provider-side instability.
429 rate limit	Sometimes	Could be temporary, but watch cost and abuse.
400 invalid request	No	The request itself is likely wrong.
Authentication error	No	Failover will not fix bad credentials.
Content policy refusal	Usually no	You probably do not want to silently bypass safety controls.
Output validation failure	Maybe	Depends on whether the fallback can do better.
Partial streamed response	Usually no	You cannot seamlessly replace output already shown.

Your failover service should catch only the exceptions that represent failover-worthy conditions.

Define explicit exceptions:

namespace App\Ai\Exceptions;

use RuntimeException;

final class LlmTimeoutException extends RuntimeException {}

final class LlmRateLimitedException extends RuntimeException {}

final class LlmUnavailableException extends RuntimeException {}

final class LlmOutputInvalidException extends RuntimeException {}

Then build a failover service:

namespace App\Ai;

use App\Ai\Exceptions\LlmRateLimitedException;
use App\Ai\Exceptions\LlmTimeoutException;
use App\Ai\Exceptions\LlmUnavailableException;

final class LlmFailover
{
    /**
     * @param array<int, CompletionProvider> $providers
     */
    public function __construct(
        private readonly array $providers,
    ) {}

    public function complete(Prompt $prompt): CompletionResult
    {
        $lastException = null;

        foreach ($this->providers as $provider) {
            if (! $provider->supports($prompt)) {
                continue;
            }

            try {
                return $provider->complete($prompt);
            } catch (LlmTimeoutException | LlmUnavailableException $e) {
                $lastException = $e;
                report($e);
                continue;
            } catch (LlmRateLimitedException $e) {
                $lastException = $e;
                report($e);
                continue;
            }
        }

        throw $lastException ?? new LlmUnavailableException(
            'No LLM provider was available.'
        );
    }
}

This is still simple, but it is much safer than catching Throwable.

🧠 The important part:
Failover is not “try everything until something works.” It is “try the next provider only when the failure indicates that another provider might plausibly succeed.”

If the request is invalid, the fallback provider may fail too, but now you have paid twice and delayed the real error.

4. Give Each Provider a Smaller Timeout Than You Think

Scenario:
Your interactive route has an eight-second budget. Your primary provider has a ten-second timeout. Technically, the provider is configured. Practically, failover cannot happen inside the user-facing deadline.

Why it matters:
Timeouts are additive in the worst way. If the first provider gets ten seconds and the second gets ten seconds, your “eight-second experience” can become a twenty-second failure.

Solution:
Allocate timeout budgets explicitly.

For interactive requests:

$prompt = new Prompt(
    messages: $request->messages(),
    interactive: true,
);

$providers = [
    new HttpCompletionProvider(
        name: 'primary',
        endpoint: config('llm.providers.primary.endpoint'),
        apiKey: config('llm.providers.primary.key'),
        timeoutSeconds: 5,
        connectTimeoutSeconds: 2,
    ),
    new HttpCompletionProvider(
        name: 'fallback',
        endpoint: config('llm.providers.fallback.endpoint'),
        apiKey: config('llm.providers.fallback.key'),
        timeoutSeconds: 3,
        connectTimeoutSeconds: 2,
    ),
];

A few rules work well:

The primary provider gets the largest slice, but not the whole slice.
The fallback provider should be fast.
Connection timeout should be much smaller than total timeout.
Background jobs can use larger timeouts than interactive routes.

For background work, the trade-off changes. If you are summarizing a support ticket in a queue job, waiting thirty or sixty seconds may be acceptable. In that case, failover can be slower and more thorough.

// config/llm.php

return [
    'background_timeout_seconds' => 45,

    'background_providers' => [
        'primary' => [
            'timeout_seconds' => 30,
        ],

        'fallback' => [
            'timeout_seconds' => 30,
        ],
    ],
];

This is also where queues become valuable. Instead of blocking a web request, you can dispatch the AI call:

GenerateSupportSummary::dispatch($ticket);

Then the job can retry, fail over, back off, and eventually land in a failed job queue if nothing works.

⚠️ Production warning:
If your HTTP client retries automatically and your failover layer also retries, you can accidentally multiply latency and cost. Choose one place to own retry policy.

5. Add a Circuit Breaker Before You Add More Providers

Scenario:
Your primary provider is down. Every request still tries it first, waits for the timeout, then falls back. Your fallback provider now handles all traffic, but every user still pays the primary timeout tax. Worse, if the fallback starts to strain under the extra load, your whole system degrades.

Why it matters:
Failover without memory is expensive. A circuit breaker gives your system a short-term memory: if a provider keeps failing, stop sending traffic to it for a while.

Solution:
Use a simple circuit breaker backed by Laravel’s cache or rate limiter.

A practical implementation can be surprisingly small:

namespace App\Ai;

use Illuminate\Support\Facades\RateLimiter;

final class ProviderCircuit
{
    public function __construct(
        private readonly string $provider,
        private readonly int $failureThreshold = 5,
        private readonly int $cooldownSeconds = 60,
    ) {}

    private function key(): string
    {
        return "llm-circuit:{$this->provider}";
    }

    public function allow(): bool
    {
        return ! RateLimiter::tooManyAttempts(
            $this->key(),
            $this->failureThreshold
        );
    }

    public function recordFailure(): void
    {
        RateLimiter::hit($this->key(), $this->cooldownSeconds);
    }

    public function recordSuccess(): void
    {
        RateLimiter::clear($this->key());
    }
}

Then integrate it into the failover loop:

public function complete(Prompt $prompt): CompletionResult
{
    $lastException = null;

    foreach ($this->providers as $provider) {
        if (! $provider->supports($prompt)) {
            continue;
        }

        $circuit = new ProviderCircuit($provider->name());

        if (! $circuit->allow()) {
            continue;
        }

        try {
            $result = $provider->complete($prompt);
            $circuit->recordSuccess();

            return $result;
        } catch (LlmTimeoutException | LlmUnavailableException $e) {
            $circuit->recordFailure();
            $lastException = $e;
            report($e);
        } catch (LlmRateLimitedException $e) {
            $lastException = $e;
            report($e);
        }
    }

    throw $lastException ?? new LlmUnavailableException(
        'No LLM provider was available.'
    );
}

This is not a perfect distributed circuit breaker. It is intentionally simple. But it already fixes the most common production problem: repeatedly paying the cost of a known-bad provider.

If you run multiple Laravel servers, use a shared cache store such as Redis. If each web server has its own local array cache, each process will independently rediscover the outage.

A few production refinements worth considering:

Use per-provider, per-route circuit keys if different features use different models.
Add half-open behavior by allowing one probe request after the cooldown.
Avoid tripping the circuit on invalid input or auth errors.
Track circuit state changes as events, not just silent cache writes.

💡 Practical note:
A circuit breaker should usually trip on repeated timeouts and 5xx errors. It should be more cautious with rate limits, because a 429 may mean “slow down” rather than “this provider is dead.”

6. Rate Limits Need a Different Failover Rule Than Timeouts

Scenario:
Your primary provider returns 429 Too Many Requests. Your failover logic treats this exactly like a timeout and immediately calls the backup provider. It works, until your traffic increases and you start burning through the backup provider’s quota too.

Why it matters:
A timeout often means “this provider cannot respond right now.” A rate limit often means “you are sending too much traffic, too fast, or you have exhausted quota.” Those are related, but not identical.

If you fail over too aggressively on rate limits, you can turn a capacity problem into a multi-provider capacity problem.

Solution:
Separate rate-limit handling from timeout handling.

A reasonable starting point:

try {
    return $provider->complete($prompt);
} catch (LlmTimeoutException | LlmUnavailableException $e) {
    $circuit->recordFailure();
    $lastException = $e;
    report($e);
} catch (LlmRateLimitedException $e) {
    // Do not immediately treat this as a dead provider.
    // Maybe fail over, maybe back off, depending on context.
    $lastException = $e;
    report($e);
}

For interactive requests, failing over on rate limit can still be acceptable because the user is waiting. For background jobs, I would usually prefer:

Respect Retry-After if the provider gives it.
Use exponential backoff with jitter.
Keep the job in the queue instead of immediately hammering another provider.
Only fail over if the rate limit persists or the job is time-sensitive.

Example in a queued job:

public function backoff(): array
{
    return [5, 30, 120, 600];
}

public function retryUntil(): int
{
    return now()->addMinutes(30)->getTimestamp();
}

For rate-limited interactive requests, you can also choose a product-level fallback instead of another LLM provider:

Return a cached answer if appropriate.
Return a shorter answer.
Tell the user the assistant is busy.
Queue the request and notify when complete.
Downgrade to a cheaper or faster model.

This is where failover stops being purely technical.

If your primary model is expensive and your fallback is cheap, failing over on every rate limit may quietly change your cost model. If your fallback model is less capable, failing over may reduce quality in a way users notice.

🔍 Why this matters:
The best failover policy is not always “another provider.” Sometimes it is “another expectation.”

7. Streaming Changes When Failover Is Allowed

Scenario:
Your chat UI streams tokens to the browser. The primary provider starts responding, then stalls mid-stream. You want to fail over to the backup provider, but the user has already seen three paragraphs of output.

Why it matters:
Once output has been visible to the user, seamless failover is mostly gone. You cannot silently replace a partial answer with a different provider’s answer without creating a confusing experience. The model may phrase things differently, contradict itself, or lose the previous context.

Solution:
For streaming, divide the failure into two phases:

Before first visible token
Failover is still possible.
After first visible token
Failover becomes a recovery problem, not a seamless switch.

For the first phase, you can apply timeout logic to the initial connection and first chunk.

Conceptually:

public function stream(Prompt $prompt): Generator
{
    foreach ($this->providers as $provider) {
        if (! $provider->supports($prompt)) {
            continue;
        }

        try {
            $stream = $provider->openStream($prompt);

            // Wait for first chunk with a short timeout.
            $firstChunk = $this->awaitFirstChunk($stream);

            yield $firstChunk;

            foreach ($stream as $chunk) {
                yield $chunk;
            }

            return;
        } catch (LlmTimeoutException $e) {
            report($e);
            continue;
        }
    }

    throw new LlmUnavailableException('No streaming provider available.');
}

The exact implementation depends on your provider client, HTTP streaming strategy, and how you push chunks to the frontend. The principle is stable: do not commit to a provider until it has produced something usable.

Once you have emitted output, your options are more limited:

Show an error and allow the user to retry.
Preserve the partial response and ask whether they want a regenerated answer.
Switch to a background retry if the operation can be completed asynchronously.
Log the interruption and degrade gracefully.

Do not pretend the stream never happened.

Streaming also changes timeout budgets. A provider that takes four seconds to start streaming but then produces tokens quickly may be better than a provider that starts in one second but stalls later. For interactive UX, time to first token is often more important than total completion time.

Track these separately:

provider_first_token_latency_ms
provider_total_completion_ms
provider_stream_interruption_count
provider_timeout_before_first_token_count

🚨 Production warning:
If your frontend has already rendered tokens, failover should usually become an explicit recovery action, not a hidden retry.

8. Make the Fallback Model Speak the Same Product Language

Scenario:
Your primary model returns clean JSON. Your fallback model returns valid JSON most of the time, but occasionally wraps it in Markdown, adds commentary, or uses slightly different field names. Your failover “works,” but now your feature randomly breaks.

Why it matters:
A fallback provider is only useful if the rest of your system can consume its output. If the fallback changes the shape, tone, length, or reliability of the response, you have not solved the problem. You have moved it.

Solution:
Treat output compatibility as part of failover design.

There are several compatibility concerns to handle:

1. Structural compatibility

If your app expects JSON, validate it.

final class JsonCompletionValidator
{
    public function validate(CompletionResult $result): array
    {
        $decoded = json_decode($result->content, true);

        if (json_last_error() !== JSON_ERROR_NONE) {
            throw new LlmOutputInvalidException(
                'LLM response was not valid JSON.'
            );
        }

        return $decoded;
    }
}

If you use a JSON schema, validate against the schema too. Do not assume the model will follow instructions perfectly just because the provider claims structured output support.

2. Capability compatibility

If the prompt uses tool calling, the fallback must support compatible tool calling. If the prompt requires long context, the fallback must have enough context window. If the prompt requires vision input, the fallback must accept images.

Your supports() method should enforce this:

public function supports(Prompt $prompt): bool
{
    if ($prompt->requiresToolCalling && ! $this->supportsToolCalling) {
        return false;
    }

    if ($prompt->requiresVision && ! $this->supportsVision) {
        return false;
    }

    return true;
}

3. Product tone compatibility

This is softer but still real. If your assistant is customer-facing, a fallback model that suddenly becomes terse, overly formal, or overly cheerful can feel like a bug.

For visible AI features, it is worth running a small evaluation suite against fallback models:

Does it follow the system prompt?
Does it refuse unsafe requests appropriately?
Does it produce the expected format?
Is the answer quality acceptable?
Does it stay within token limits?
Does it handle edge cases in your language or domain?

4. Legal and privacy compatibility

Failover can accidentally move data to a provider with different retention, training, regional hosting, or compliance characteristics.

Before enabling automatic failover, ask:

Is the fallback approved for this data classification?
Does the fallback provider store prompts?
Does it train on submitted data?
Is it allowed for regulated or customer-identifiable data?
Does your data processing agreement cover this route?

This is not just a legal concern. It is an architecture concern. If certain prompts cannot go to certain providers, your failover service needs routing rules.

final class SensitivePromptRouter
{
    public function allowedProviders(Prompt $prompt): array
    {
        if ($prompt->containsPhi) {
            return [$this->approvedProvider];
        }

        return $this->defaultProviders;
    }
}

⚠️ Gotcha:
Do not let failover quietly bypass your own data governance rules. A provider that is fast and cheap is not automatically allowed to receive every prompt.

9. Make Failover Observable and Testable

Scenario:
You ship failover. It seems fine. Then one morning you discover that 80% of traffic has been going to the expensive fallback for two days because the primary provider’s API key was misconfigured.

Why it matters:
Failover is a reliability feature, but without observability it can become a silent cost and quality regression. You need to know when failover happens, why it happened, and whether the fallback is actually helping.

Solution:
Log structured events and write tests that simulate failures.

At minimum, track:

llm_request_started
llm_request_succeeded
llm_request_failed
llm_failover_attempted
llm_failover_succeeded
llm_failover_exhausted
llm_circuit_opened
llm_circuit_closed
llm_output_validation_failed

Useful context includes:

report(new LlmFailoverAttempted(
    provider: $provider->name(),
    nextProvider: $nextProvider?->name(),
    reason: $e::class,
    interactive: $prompt->interactive,
));

If you use Laravel’s logging channels, consider a dedicated llm channel:

// config/logging.php

'llm' => [
    'driver' => 'daily',
    'path' => storage_path('logs/llm.log'),
    'level' => 'debug',
],

Then you can log without drowning your main application logs:

Log::channel('llm')->warning('LLM failover attempted', [
    'provider' => $provider->name(),
    'reason' => $e::class,
    'interactive' => $prompt->interactive,
]);

For metrics, if you have a metrics pipeline, these are the ones I would care about most:

Metric	Why it matters
Provider timeout rate	Detects failing providers early.
Time to first token	Critical for streaming UX.
Failover rate	Shows how often the primary is not enough.
Fallback success rate	Tells you whether the fallback is useful.
Output validation failure rate	Detects incompatible fallback output.
Cost per request by provider	Prevents surprise bills.
Circuit breaker state	Shows whether a provider is being skipped.

Testing is just as important.

Do not test only the successful provider. Test the failure paths.

test('it fails over when the primary provider times out', function () {
    $prompt = new Prompt(messages: [
        ['role' => 'user', 'content' => 'Hello'],
    ]);

    $primary = Mockery::mock(CompletionProvider::class);
    $primary->shouldReceive('name')->andReturn('primary');
    $primary->shouldReceive('supports')->andReturn(true);
    $primary->shouldReceive('complete')
        ->andThrow(new LlmTimeoutException('primary timed out'));

    $fallback = Mockery::mock(CompletionProvider::class);
    $fallback->shouldReceive('name')->andReturn('fallback');
    $fallback->shouldReceive('supports')->andReturn(true);
    $fallback->shouldReceive('complete')
        ->andReturn(new CompletionResult(
            content: 'Hello from fallback',
            provider: 'fallback',
            model: 'fallback-model',
            finishReason: 'stop',
        ));

    $failover = new LlmFailover([$primary, $fallback]);

    $result = $failover->complete($prompt);

    expect($result->provider)->toBe('fallback')
        ->and($result->content)->toBe('Hello from fallback');
});

Also test the cases where failover should not happen:

test('it does not fail over on invalid output when fallback would not help', function () {
    // Arrange providers to throw LlmOutputInvalidException.

    // Assert the exception is not swallowed by failover.
});

If you use Laravel’s HTTP fake, you can also simulate HTTP-level failures:

Http::fake([
    'api.primary.example.com/*' => Http::response(null, 500),
    'api.fallback.example.com/*' => Http::response([
        'choices' => [
            [
                'message' => ['content' => 'fallback answer'],
                'finish_reason' => 'stop',
            ],
        ],
    ], 200),
]);

A good failover test suite should answer questions like:

What happens when the primary times out?
What happens when the fallback times out?
What happens when all providers are unavailable?
What happens when the primary returns invalid JSON?
What happens when the fallback does not support the requested capability?
What happens when the circuit breaker is open?
What happens during a streaming stall before first token?
What happens after the first token has already been emitted?

💡 Practical note:
If your fallback path is never tested, it is not a fallback. It is a rumor.

Choosing the Right Failover Strategy

There is no single correct failover design. The right approach depends on the type of AI feature you are building.

Here is a practical comparison.

Strategy	Best for	Complexity	Main risk
Retry same provider once	Simple features, low stakes	Low	Can double latency.
Immediate provider chain	Basic interactive fallback	Low/Medium	Can amplify cost and load.
Timeout-budgeted failover	User-facing features	Medium	Requires careful budgeting.
Circuit breaker plus failover	Production systems	Medium	State needs shared cache at scale.
Queue-based retry and failover	Background processing	Medium/High	Delayed user feedback.
Streaming first-token failover	Chat and streaming UI	High	Harder to implement and test.
External AI gateway failover	Multi-team or multi-app systems	High	Adds infrastructure dependency.

For a small Laravel app with one AI feature, I would start with:

A provider contract.
Explicit timeouts.
Failover on timeout and provider unavailability.
Simple circuit breaker.
Structured logging.

For a product with real-time chat, I would add:

Streaming-first design.
Time-to-first-token budgets.
Failover only before first visible chunk.
Degraded UI states for mid-stream failures.
Output validation.

For background AI jobs, I would lean more on:

Queues.
Exponential backoff.
Longer timeouts.
Dead-letter handling.
Cost and quota monitoring.

For highly regulated data, I would add:

Provider allowlists by data classification.
Feature flags for failover eligibility.
Audit logging.
Explicit routing for sensitive prompts.
Manual fallback for cases where automatic switching is not allowed.

What I Would Actually Ship

If I were building a Laravel AI feature in production today, my default would not be “call the backup provider whenever anything throws.”

I would ship something more deliberate:

A single CompletionProvider interface.
Controllers and jobs would never know which provider is being used.
Explicit timeout budgets.
Interactive requests would have a short total deadline. Each provider would get a smaller slice of that deadline.
A small failover service.
It would only fail over on timeout, provider outage, and selected rate-limit conditions.
A circuit breaker backed by Redis.
If a provider keeps failing, traffic should stop hitting it for a cooldown period.
Separate handling for streaming.
Failover before the first visible token, explicit recovery after that.
Output validation.
Especially for JSON, tool calls, and anything that feeds another system.
Observability from day one.
I would want to know when failover happens, how often, how much it costs, and whether the fallback output passes validation.
Feature flags.
Automatic failover should be switchable per feature, per prompt class, and sometimes per tenant.

The most important mental shift is this: LLM failover is not a retry policy. It is a routing policy.

You are routing requests across providers with different latency, cost, quality, privacy rules, capabilities, and failure modes. Laravel gives you the right primitives — service container, HTTP client, cache, queues, logging, testing — but the architecture is yours to design.

Get that architecture right, and a provider timeout becomes a boring operational event instead of a user-visible outage.