RAG in Laravel: Building a Document Q&A System with the AI SDK and Vector Search

A generic AI model knows a lot. It does not know your private product manuals, internal policies, or customer documentation.

Retrieval-Augmented Generation (RAG) closes that gap. Your application retrieves relevant document chunks, adds them to the prompt, and asks the model to answer from that context.

This post builds that workflow with Laravel, PostgreSQL, pgvector, and the Laravel AI SDK. The result is a document Q&A API that can answer questions using your own data.

The architecture has five stages:

1. Split documents into focused chunks.
2. Generate an embedding for each chunk.
3. Store the chunks and vectors in PostgreSQL.
4. Retrieve the closest chunks for each question.
5. Feed the retrieved context into an AI SDK agent.

Laravel provides the application structure, database layer, queues, validation, and deployment tools. The Laravel AI SDK provides a consistent interface for embeddings and text generation.

Choose the retrieval architecture

There are two practical RAG paths in the Laravel AI SDK.

The first stores vectors in your own database. You control chunking, metadata, filters, authorization, and retrieval. This is the approach used here.

The second uses provider-managed vector stores. You upload documents to a provider, then expose the SDK’s FileSearch tool to an agent.

Use database-backed vectors when your application needs strict tenant isolation, custom filters, citations, or full control over ingestion. Use provider-managed stores when you want to avoid maintaining the indexing pipeline.

This guide uses PostgreSQL with pgvector. Laravel’s semantic search documentation covers the same database capabilities.

Install the AI SDK and prepare PostgreSQL

Install the SDK with Composer:

composer require laravel/ai

Publish its configuration and migrations:

php artisan vendor:publish --provider="Laravel\Ai\AiServiceProvider"
php artisan migrate

Configure an embedding provider in your environment. The exact model depends on your provider and config/ai.php.

OPENAI_API_KEY=your-api-key

The vector column must use the same number of dimensions as the embedding model. The examples below use 1,536 dimensions.

For local PostgreSQL installations, enable pgvector. Managed PostgreSQL services may already include it. Laravel Cloud provides PostgreSQL environments with vector support available.

Create a table for document chunks

A RAG system should store chunks rather than entire documents. Each row represents a focused piece of source content.

Create a model and migration:

php artisan make:model DocumentChunk -m

Use the following migration:

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration
{
    public function up(): void
    {
        Schema::ensureVectorExtensionExists();

        Schema::create('document_chunks', function (Blueprint $table) {
            $table->id();
            $table->string('source');
            $table->text('content');
            $table->json('metadata')->nullable();
            $table->vector('embedding', dimensions: 1536)->index();
            $table->timestamps();
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('document_chunks');
    }
};

The vector index creates an HNSW index with cosine distance. That keeps similarity queries responsive as the table grows.

Cast the vector and metadata fields in the model:

<?php

namespace App\Models;

use Illuminate\Database\Eloquent\Model;

class DocumentChunk extends Model
{
    protected $fillable = [
        'source',
        'content',
        'metadata',
        'embedding',
    ];

    protected function casts(): array
    {
        return [
            'metadata' => 'array',
            'embedding' => 'array',
        ];
    }
}

Run the migration:

php artisan migrate

Split documents into useful chunks

Embedding an entire book produces a broad representation of many unrelated ideas. Retrieval becomes less precise.

Instead, split documents into chunks that each cover one topic. Markdown headings and paragraph boundaries make good starting points. Keep chunks small enough to retrieve specific facts, but large enough to preserve context.

A simple paragraph-based chunker works for an initial implementation:

<?php

namespace App\Services;

class DocumentChunker
{
    public function split(
        string $text,
        int $maxCharacters = 4000,
        int $overlap = 400,
    ): array {
        $paragraphs = preg_split(
            '/\R{2,}/',
            trim($text),
            -1,
            PREG_SPLIT_NO_EMPTY
        );

        $chunks = [];
        $current = '';

        foreach ($paragraphs as $paragraph) {
            $candidate = $current === ''
                ? $paragraph
                : $current."\n\n".$paragraph;

            if ($current !== '' && strlen($candidate) > $maxCharacters) {
                $chunks[] = $current;

                $current = substr($current, -$overlap)."\n\n".$paragraph;
                continue;
            }

            $current = $candidate;
        }

        if ($current !== '') {
            $chunks[] = $current;
        }

        return $chunks;
    }
}

This implementation uses character counts for clarity. Production systems should consider a token-aware strategy. Markdown parsers can also preserve headings and section breadcrumbs in metadata.

Store metadata such as the source filename, heading, tenant ID, document version, or access scope. That metadata becomes useful during filtering and citation rendering.

Generate and store embeddings

The AI SDK’s Embeddings class supports batching. Batch generation reduces network overhead compared with making one API request per chunk.

A console command can ingest Markdown files from storage:

php artisan make:command IngestDocuments

The core ingestion flow looks like this:

<?php

namespace App\Console\Commands;

use App\Models\DocumentChunk;
use App\Services\DocumentChunker;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Storage;
use Laravel\Ai\Embeddings;

class IngestDocuments extends Command
{
    protected $signature = 'rag:ingest';

    protected $description = 'Embed and store application documents';

    public function handle(DocumentChunker $chunker): int
    {
        foreach (Storage::disk('local')->files('docs') as $path) {
            $text = Storage::disk('local')->get($path);
            $chunks = $chunker->split($text);

            if ($chunks === []) {
                continue;
            }

            $response = Embeddings::for($chunks)->generate();

            foreach ($chunks as $index => $content) {
                DocumentChunk::create([
                    'source' => $path,
                    'content' => $content,
                    'metadata' => [
                        'hash' => hash('sha256', $content),
                    ],
                    'embedding' => $response->embeddings[$index],
                ]);
            }
        }

        $this->info('Document ingestion complete.');

        return self::SUCCESS;
    }
}

Run it with:

php artisan rag:ingest

Before inserting new rows, production ingestion should usually delete or update the previous version of the same document. A content hash helps you skip unchanged chunks.

Embedding generation is external work. Do not run it inside an upload request for large files. Dispatch a queued job after the upload completes. Laravel’s queue system keeps slow embedding operations away from web requests.

You can also enable embedding caching in config/ai.php. The AI SDK caching documentation explains how cached vectors are keyed by provider, model, dimensions, and input content.

Retrieve relevant chunks for a question

At query time, embed the user’s question with the same embedding model used during ingestion.

use App\Models\DocumentChunk;
use Laravel\Ai\Embeddings;

$question = 'How long can a customer request a refund?';

$queryEmbedding = Embeddings::for([$question])
    ->generate()
    ->embeddings[0];

$chunks = DocumentChunk::query()
    ->whereVectorSimilarTo(
        'embedding',
        $queryEmbedding,
        minSimilarity: 0.35
    )
    ->limit(6)
    ->get();

whereVectorSimilarTo compares the query vector with stored vectors using cosine similarity. Results are filtered by the minimum similarity and ordered by relevance.

You can combine vector search with normal database constraints:

$chunks = DocumentChunk::query()
    ->where('team_id', $request->user()->team_id)
    ->whereVectorSimilarTo(
        'embedding',
        $queryEmbedding,
        minSimilarity: 0.35
    )
    ->limit(6)
    ->get();

This is important for multi-tenant systems. Never allow a question to retrieve chunks from another customer’s knowledge base.

Laravel also accepts a plain string instead of a precomputed embedding:

$chunks = DocumentChunk::query()
    ->whereVectorSimilarTo('embedding', $question)
    ->limit(6)
    ->get();

Using an explicit embedding makes the pipeline easier to inspect. It also lets you log embedding latency and reuse the vector in a later retrieval step.

Feed retrieved context into the AI SDK

The retrieved chunks now need to become model context. Keep the prompt strict. Tell the model to use only the supplied context and to admit when the context does not contain an answer.

Laravel’s anonymous agent API works well for a deterministic retrieval flow:

use function Laravel\Ai\agent;

$context = $chunks
    ->values()
    ->map(function (DocumentChunk $chunk, int $index) {
        $source = $chunk->source;

        return "--- Source {$index}: {$source} ---\n"
            .$chunk->content;
    })
    ->implode("\n\n");

$instructions = <<<PROMPT
You answer questions about the application's documents.

Use only the context provided below.
Do not rely on general knowledge.
If the context does not contain enough information, say that you do not know.
Do not invent policies, dates, prices, or product details.

Context:
{$context}
PROMPT;

$response = agent(
    instructions: $instructions,
    messages: [],
    tools: [],
)->prompt($question);

$answer = (string) $response;

This approach performs retrieval before the language model call. It is predictable and easy to debug. You can log the question, similarity threshold, returned sources, and final answer.

The AI SDK also provides a SimilaritySearch tool. That approach lets an agent decide when to search and which query to use. It is useful for conversational workflows, but direct retrieval is often better for a document Q&A endpoint where every answer must be grounded.

Expose the workflow as a REST API

Laravel is a practical PHP web framework for this workflow because authentication, validation, queues, Eloquent, and JSON responses share one application layer.

A controller can combine the retrieval and generation steps:

<?php

namespace App\Http\Controllers;

use App\Models\DocumentChunk;
use Illuminate\Http\Request;
use Laravel\Ai\Embeddings;
use function Laravel\Ai\agent;

class DocumentQuestionController
{
    public function __invoke(Request $request)
    {
        $data = $request->validate([
            'question' => ['required', 'string', 'max:2000'],
        ]);

        $queryEmbedding = Embeddings::for([$data['question']])
            ->generate()
            ->embeddings[0];

        $chunks = DocumentChunk::query()
            ->where('team_id', $request->user()->team_id)
            ->whereVectorSimilarTo(
                'embedding',
                $queryEmbedding,
                minSimilarity: 0.35
            )
            ->limit(6)
            ->get();

        $context = $chunks
            ->map(fn (DocumentChunk $chunk) => $chunk->source."\n".$chunk->content)
            ->implode("\n\n---\n\n");

        $response = agent(
            instructions: <<<PROMPT
Answer only from the supplied document context.
If the answer is not present, say so clearly.

Context:
{$context}
PROMPT,
            messages: [],
            tools: [],
        )->prompt($data['question']);

        return response()->json([
            'answer' => (string) $response,
            'sources' => $chunks->pluck('source')->unique()->values(),
        ]);
    }
}

Register the endpoint:

use App\Http\Controllers\DocumentQuestionController;
use Illuminate\Support\Facades\Route;

Route::middleware('auth:sanctum')->post(
    '/document-questions',
    DocumentQuestionController::class
);

This gives you a clean foundation for building a REST API with PHP. Add rate limiting, authorization policies, request tracing, and structured response fields as the feature matures.

Improve retrieval quality in production

A working RAG pipeline is only the beginning. Retrieval quality usually depends more on ingestion than on the final prompt.

Use these improvements as your dataset grows:

• Preserve headings and source locations in chunk metadata.
• Re-embed only documents whose content hash changed.
• Queue ingestion and retry provider failures.
• Filter by tenant, team, locale, publication status, or document version.
• Return source references so users can verify answers.
• Add a reranking pass when the initial results contain several similar chunks.
• Log retrieved chunks and evaluate answers against a fixed question set.
• Keep embedding model and dimension changes behind a reindexing plan.

The Laravel AI SDK supports multiple embedding providers, reranking, testing fakes, and provider failover. Laravel’s broader ecosystem adds the remaining pieces, from Sanctum authentication to Horizon queue monitoring and Nightwatch application observability.

RAG works when retrieval is deliberate. Store focused chunks, enforce data boundaries, and give the model only the context it needs. With Laravel and the AI SDK, those decisions fit naturally into the same tools you already use to build modern applications.