Vision in Laravel: Building Multimodal AI Features with the AI SDK

A user uploads a receipt. Another sends a screenshot of an error. A field technician photographs a damaged component.

Your Laravel application can now send those images to an AI agent for analysis. The Laravel AI SDK normalizes the provider-specific details behind a small, expressive API.

In this tutorial, you will build a reusable ImageAnalyzer agent. It will accept an uploaded image, send it to a vision-capable model, and return structured analysis.

The examples target Laravel 13 and the Laravel AI SDK v0.11 era. You can review the official AI SDK documentation and the Laravel AI SDK repository for the complete API.

Vision is a model capability: Choose the right model

Vision is not available simply because a provider supports text generation.

The configured model must accept multimodal input. A text-only model cannot inspect an attached screenshot, even when its provider supports vision elsewhere.

The Laravel AI SDK currently supports image attachments with providers including:

• OpenAI
• Anthropic
• Gemini

Each provider has different model names, image formats, limits, and pricing. Select a multimodal model explicitly in config/ai.php, then test it with the images your application expects to receive.

The SDK gives you one interface. The model still determines what the request can understand.

You can install the SDK with Composer:

composer require laravel/ai

php artisan vendor:publish \
    --provider="Laravel\Ai\AiServiceProvider"

php artisan migrate

Configure the provider credentials in your environment. Then set a vision-capable model as the provider's default text model, or pass the provider and model when prompting the agent.

Build the ImageAnalyzer agent: Keep vision logic reusable

Create an agent with Artisan:

php artisan make:agent ImageAnalyzer --structured

The agent should own the instructions and output shape. Controllers should handle HTTP concerns, such as validation and storage.

Create app/Ai/Agents/ImageAnalyzer.php:

<?php

namespace App\Ai\Agents;

use Illuminate\Contracts\JsonSchema\JsonSchema;
use Laravel\Ai\Contracts\Agent;
use Laravel\Ai\Contracts\HasStructuredOutput;
use Laravel\Ai\Promptable;
use Stringable;

class ImageAnalyzer implements Agent, HasStructuredOutput
{
    use Promptable;

    public function instructions(): Stringable|string
    {
        return <<<'PROMPT'
            You analyze attached images for a Laravel application.

            Describe only what the image supports.
            Transcribe visible text carefully.
            Mark uncertain values as uncertain.
            Never invent missing details.
        PROMPT;
    }

    public function schema(JsonSchema $schema): array
    {
        return [
            'summary' => $schema->string()->required(),

            'visible_text' => $schema->array()
                ->items($schema->string())
                ->required(),

            'objects' => $schema->array()
                ->items($schema->string())
                ->required(),

            'confidence' => $schema->string()
                ->enum(['low', 'medium', 'high'])
                ->required(),
        ];
    }
}

The Agent contract defines the agent. The Promptable trait provides methods such as prompt, stream, and queue.

HasStructuredOutput keeps the response predictable. This matters when another part of your application needs to save, search, or route the result.

Attach an image with Laravel\Ai\Files\Image:

use App\Ai\Agents\ImageAnalyzer;
use Laravel\Ai\Files\Image;

$response = (new ImageAnalyzer)->prompt(
    'Analyze this image and extract the visible details.',
    attachments: [
        Image::fromPath('/var/app/uploads/photo.jpg'),
    ],
);

$summary = $response['summary'];

Images can come from several sources:

use Laravel\Ai\Files\Image;

// Local path
Image::fromPath('/var/app/uploads/photo.jpg');

// Laravel filesystem disk
Image::fromStorage('uploads/photo.jpg', disk: 'private');

// Remote URL
Image::fromUrl('https://example.com/photo.jpg');

// Uploaded file
Image::fromUpload($request->file('photo'));

For request uploads, storing the file first is usually the better production pattern. The stored path survives retries and queued jobs, and it gives you a stable audit trail.

$path = $request->file('photo')->store('vision', 'private');

$response = (new ImageAnalyzer)->prompt(
    'Describe the attached photo.',
    attachments: [
        Image::fromStorage($path, disk: 'private'),
    ],
);

Provider selection: Use a multimodal configuration

You can use the default provider configured by the SDK. You can also select a provider and model per request.

use Laravel\Ai\Enums\Lab;

$response = (new ImageAnalyzer)->prompt(
    'Read the receipt and identify the merchant, date, and total.',
    provider: Lab::Gemini,
    model: 'your-multimodal-model',
    attachments: [
        Image::fromStorage($path, disk: 'private'),
    ],
);

Replace the model name with a currently available multimodal model from your provider. Keep this value in configuration rather than scattering it through controllers.

A useful configuration pattern separates vision models from ordinary text models:

// config/services.php

return [
    'ai' => [
        'vision_provider' => env('AI_VISION_PROVIDER', 'openai'),
        'vision_model' => env('AI_VISION_MODEL'),
    ],
];

Your application can then choose a cheaper model for simple classification and a more capable model for dense diagrams or difficult receipts.

Build a REST endpoint: Accept an image and return JSON

A vision feature fits naturally into an API. This is a practical way to build rest api with php while keeping the AI layer inside a dedicated agent.

Define a route in routes/api.php:

use App\Http\Controllers\VisionController;
use Illuminate\Support\Facades\Route;

Route::post('/vision/analyze', [VisionController::class, 'store']);

Create the controller:

<?php

namespace App\Http\Controllers;

use App\Ai\Agents\ImageAnalyzer;
use Illuminate\Http\Request;
use Illuminate\Validation\Rules\File;
use Laravel\Ai\Files\Image;

class VisionController extends Controller
{
    public function store(Request $request)
    {
        $validated = $request->validate([
            'image' => [
                'required',
                File::image()
                    ->types(['jpg', 'jpeg', 'png', 'webp'])
                    ->max(5120),
            ],
            'instruction' => ['nullable', 'string', 'max:1000'],
        ]);

        $path = $request->file('image')->store('vision', 'private');

        $response = (new ImageAnalyzer)->prompt(
            $validated['instruction']
                ?? 'Analyze the attached image.',
            attachments: [
                Image::fromStorage($path, disk: 'private'),
            ],
        );

        return response()->json([
            'summary' => $response['summary'],
            'visible_text' => $response['visible_text'],
            'objects' => $response['objects'],
            'confidence' => $response['confidence'],
        ]);
    }
}

A request can now send a multipart upload:

curl -X POST https://example.test/api/vision/analyze \
    -H "Accept: application/json" \
    -F "image=@receipt.jpg" \
    -F "instruction=Extract the merchant, date, subtotal, tax, and total."

The same endpoint can support product photos, screenshots, diagrams, and identity documents. Use the instruction to describe the task. Keep the agent instructions stable and focused.

For highly specific workflows, create separate agents. A ReceiptAnalyzer can return currency and line items. A SupportScreenshotAnalyzer can return error codes and likely causes.

Practical applications: Match the prompt to the workflow

Support-ticket screenshot triage

Users often send screenshots instead of copying error messages. A vision agent can extract the visible error, identify the application area, and suggest a routing label.

$response = (new ImageAnalyzer)->prompt(
    <<<'PROMPT'
        Inspect this support screenshot.

        Extract visible error messages.
        Identify the likely product area.
        Return a short triage summary.
        Do not claim a root cause unless the screenshot supports it.
    PROMPT,
    attachments: [
        Image::fromStorage($path, disk: 'private'),
    ],
);

You can save the structured response to the ticket, show it to a support agent, or use it to suggest a queue. Keep a human in the loop when the analysis changes ticket priority or customer communication.

Receipt extraction

Receipts provide a clear structured-output use case. Ask for exact fields and require uncertainty when text is unreadable.

$response = (new ImageAnalyzer)->prompt(
    'Extract the merchant, purchase date, currency, subtotal, tax, total, and line items. '
    .'Return null or an empty list when a value is not visible.',
    attachments: [
        Image::fromStorage($path, disk: 'private'),
    ],
);

For a production receipt workflow, define a dedicated schema with numeric totals and an array of line-item objects. Then validate the returned values before creating accounting records.

Product and diagram understanding

Vision models can classify product photos, identify visible components, and summarize architecture diagrams. They can also answer focused questions about screenshots of dashboards or code editors.

Prompt for one job at a time. “Describe everything” produces less reliable output than “Identify the three visible status indicators and return their labels.”

Upload safety: Validate before sending

Treat uploaded images as untrusted input.

Laravel validation should check the file type and size before the image reaches your provider. File::image() checks that the upload is an image, while types() and max() narrow the accepted input.

use Illuminate\Validation\Rules\File;

$request->validate([
    'image' => [
        'required',
        File::image()
            ->types(['jpg', 'jpeg', 'png', 'webp'])
            ->max(5120),
    ],
]);

The max(5120) value is measured in kilobytes. That example limits uploads to approximately 5 MB.

Provider limits vary. Anthropic, for example, caps image inputs at around 5 MB in common request flows. Other providers may apply different limits for dimensions, formats, image count, or encoded request size.

Use the strictest limit that supports your feature. Resize very large images before sending them. Strip metadata when privacy requires it. Reject animated or unsupported formats unless your selected model handles them.

Store private uploads on a private disk. Do not expose permanent public URLs for receipts, identity documents, or customer screenshots. Delete temporary files after analysis when retention is not required.

You should also consider:

• Rate limiting the endpoint.
• Authorizing who can view uploaded files.
• Scanning uploads when your threat model requires it.
• Logging provider request IDs without logging sensitive image content.
• Recording model, provider, latency, and token usage.
• Redacting personal data from application logs.

Laravel's validation documentation and filesystem documentation cover the framework pieces around this boundary.

Image understanding versus image generation: Use different APIs

Vision input and image generation solve different problems.

Laravel\Ai\Files\Image represents an image attached to a prompt. The model inspects that image and returns text or structured data.

Laravel\Ai\Image generates a new image from a text prompt. It can also use attachments as reference images for transformations or style transfer.

use Laravel\Ai\Files;
use Laravel\Ai\Image;

$image = Image::of(
    'Create a clean square illustration of this product for a social post.'
)
    ->attachments([
        Files\Image::fromStorage($path, disk: 'private'),
    ])
    ->square()
    ->generate('gemini');

Image generation supports OpenAI, Gemini, and xAI. Anthropic supports vision input in this workflow, but it is not an image-generation provider in the Laravel AI SDK.

You can also generate without a reference image:

$generated = Image::of(
    'A bright editorial illustration of a Laravel application analyzing a receipt.'
)
    ->square()
    ->generate('openai');

$path = $generated->store('generated-images', 'public');

Keep the distinction clear:

• Use Files\Image with an agent to understand an image.
• Use Laravel\Ai\Image to create an image.
• Use attachments() on Laravel\Ai\Image for reference-based generation.

Production notes: Queue heavy work and measure usage

Synchronous vision requests are suitable for small interactive workflows. Large images and complex prompts can take longer, so consider queueing the analysis.

Store the upload first. Dispatch a job with its storage path. The job can call the agent with Image::fromStorage(...), save the structured response, and notify the frontend when processing finishes.

Streaming can improve the experience for long textual responses. It does not make the image upload itself instant, but it lets users see analysis as it arrives.

Track latency, failures, provider errors, and usage. Laravel AI SDK events such as AgentPrompted, AgentFailed, and AgentStreamed can feed your application logs or monitoring pipeline. Laravel Nightwatch can help you observe the surrounding application behavior.

Finally, test without calling a real provider. The SDK supports faking agents and asserting that prompts were sent:

use App\Ai\Agents\ImageAnalyzer;

ImageAnalyzer::fake([
    [
        'summary' => 'A printed receipt.',
        'visible_text' => ['Total 42.00'],
        'objects' => ['receipt'],
        'confidence' => 'high',
    ],
]);

$response = (new ImageAnalyzer)->prompt(
    'Analyze the image.',
    attachments: [
        Image::fromStorage('vision/receipt.jpg', disk: 'private'),
    ],
);

ImageAnalyzer::assertPrompted('Analyze the image.');

Vision becomes practical when it stays close to normal Laravel design. Validate the upload, store it safely, attach it to a focused agent, and return data your application can trust and review.

If you build a useful image workflow with the Laravel AI SDK, we’d love to hear how you approach it.