Vectorless RAG: When LLM Reasoning Replaces Vector Search
Vector databases are the default retrieval backbone for RAG systems — but they are not the only option, and in many enterprise scenarios, they are not the right one. Vectorless RAG replaces similarity search with direct LLM reasoning over document summaries. This article breaks down how it works, when it outperforms traditional approaches, and how it changes the architecture of agentic AI systems.
The Limitation Vector Search Was Never Designed to Solve
Standard RAG makes a foundational assumption: that the semantic meaning of a user query, when converted into a vector, will be geometrically close to the vectors of the document chunks that contain the answer. In practice, this assumption holds reasonably well for straightforward factual lookups. It breaks down reliably for anything more complex.
Consider an analyst asking: "What were the key risk factors that contributed to the Q3 underperformance, and how do they relate to the credit risk thresholds updated in the 2024 policy amendment?" This query spans multiple documents, requires cross-referencing, and its intent is not well-captured by a single embedding vector. A similarity search will retrieve chunks that are semantically adjacent to individual phrases in the query — but not chunks that are jointly relevant, nor chunks that a human would recognise as necessary context for the full answer.
This is the gap Vectorless RAG — also described as reasoning-based RAG — is designed to close. Instead of asking "which chunks are most similar to this query?", it asks the LLM directly: "given what you know about this question, which documents should we retrieve to answer it?" That is a fundamentally different, and in many cases substantially more reliable, operation.
Traditional RAG vs. Vectorless RAG
Traditional RAG
- ·Embed query → cosine similarity
- ·Top-k chunk retrieval
- ·Vector database required
- ·Chunking artifacts affect results
- ·Struggles with multi-hop queries
Vectorless RAG
- ·LLM reasons over doc summaries
- ·Full-page targeted retrieval
- ·No vector index needed
- ·Preserves document structure
- ·Natural multi-document reasoning
How Vectorless RAG Works: The PageIndex Architecture
The most established implementation pattern is the PageIndex approach. The name is deliberate: instead of fragmenting documents into sub-paragraph chunks and embedding them, the system indexes at the page or document level — preserving natural document structure — and uses LLM reasoning to select which pages to retrieve rather than similarity scoring to surface which chunks are close.
The pipeline has four stages: indexing, summary-based retrieval planning, targeted document fetch, and generation. Each is distinct from its traditional RAG counterpart.
The Vectorless RAG Pipeline
Indexing
LLM generates a structured summary for each document page or section
Reasoning
LLM reads all summaries and selects which pages are relevant to the query
Retrieval
Full text of selected pages is fetched — no vector similarity, no chunking
Generation
LLM generates the answer grounded in the full, structurally intact page content
Stage 1: Structured Document Indexing
Rather than embedding chunks, the system passes each page or logical document section through an LLM with a structured summarisation prompt. The prompt instructs the model to extract: the primary topic, key entities (names, dates, figures, codes), the type of content (policy clause, financial table, procedure, definition), and any explicit cross-references to other documents or sections.
The resulting summaries are stored in a lightweight index — a simple JSON file, a relational database, or a document store. No vector database, no embedding model, no similarity index. The index is human-readable and auditable by default, which matters in regulated environments where retrieval decisions may need to be explained.
This indexing step is done once and updated incrementally as documents change — the same update patterns (delta indexing, soft deletes, versioned summaries) that apply to vector indexes apply here. The difference is that a changed summary is re-generated by the LLM rather than re-embedded.
Stage 2: Reasoning-Based Retrieval Planning
When a query arrives, the system constructs a prompt that presents the LLM with the full summary index — or a filtered subset of it — and asks it to reason about which pages are necessary to answer the question. This is the core mechanism that separates Vectorless RAG from everything else.
The retrieval planning prompt typically includes: the user query in full, all document summaries with page identifiers, explicit instructions to think about what information is needed before selecting pages, and a structured output format specifying which page IDs to retrieve and why.
Retrieval Planning Prompt Structure
System: You are a retrieval planning assistant. Your task is to identify which document pages contain information needed to answer the user's question.
Document Index:
[Page 4 — AML Policy v4.2, Section 3]: Defines escalation thresholds for PEP alerts. Covers risk tiers 1–3...
[Page 17 — Risk Framework 2024]: Updates to credit exposure limits and cross-border transaction flags...
Query: What is the escalation process for a Tier-2 PEP alert flagged under the new credit risk framework?
Task: List the page IDs required to answer this query and explain why each is needed. Return as JSON.
The LLM returns a structured selection — e.g., { "pages": [4, 17], "reasoning": "Page 4 defines the PEP escalation tiers; Page 17 contains the 2024 update to credit risk thresholds that the query cross-references" }. This reasoning trace is itself auditable. You know exactly why documents were selected, not just which ones were.
Stage 3: Full-Page Retrieval
The system fetches the complete text of the selected pages from a document store — not reconstructed chunks, but the original intact content, preserving tables, numbered lists, cross-reference markers, and formatting context that chunking destroys. This structural integrity is one of the most practically significant advantages of the approach, particularly for legal contracts, financial reports, and policy documents where exact phrasing, table values, and section numbering carry meaning.
Stage 4: Grounded Generation
The generation stage is similar to standard RAG: construct a prompt with retrieved content and the user query, instruct the LLM to answer using only provided documents, and apply faithfulness validation. The key difference is the quality of the input: instead of a collection of extracted chunks that may or may not be coherent together, the LLM receives full page content with intact structure and known provenance. Hallucination risk decreases because the grounding is richer and the model is less likely to fill structural gaps with parametric knowledge.
Where Vectorless RAG Outperforms Vector-Based Retrieval
Multi-Hop and Cross-Document Reasoning
When answering a question requires synthesising information from two or more documents — for example, a compliance question spanning a policy document and its 2024 amendment — LLM reasoning naturally identifies both sources as jointly relevant. Vector similarity, operating on independent chunks, cannot model this relationship.
Structured and Tabular Documents
Financial tables, regulatory schedules, and contract annexes lose meaning when chunked. A clause referencing 'the thresholds in Table 3' has no retrieval value as an isolated chunk. Vectorless RAG preserves the page context that makes structured content intelligible.
Auditable Retrieval in Regulated Industries
The reasoning trace from the retrieval planning step is a native audit trail: every answer can be explained by which documents were selected and why. This is not a post-hoc explanation — it is part of the retrieval mechanism itself. For financial services, healthcare, and legal applications this can be a decisive advantage.
Small-to-Medium Document Corpora
Vectorless RAG works most effectively when the entire summary index fits within a single LLM context window — typically corpora up to a few hundred pages. For larger corpora, hierarchical strategies (summarise summaries, partition by domain) extend the approach, but the complexity grows.
Low-Overlap Terminology Domains
In domains where user query language and document language diverge significantly — clinical queries against regulatory text, for instance — semantic embedding often finds false positives based on surface similarity. LLM reasoning is more robust to this vocabulary gap because it understands intent rather than measuring vector proximity.
Honest Tradeoffs: Where Vector Search Remains the Right Choice
Vectorless RAG is not a universal replacement. The tradeoffs are real and in certain configurations they are disqualifying.
Cost Per Query
The retrieval planning step requires an LLM call over all document summaries — at minimum one additional inference per query, sometimes two (plan + validate). For high-throughput consumer applications, this cost structure is often prohibitive. Vector search is orders of magnitude cheaper per query.
Latency
LLM reasoning is sequential and adds meaningful latency to the retrieval path. Vector similarity search in a well-indexed database can return results in single-digit milliseconds. Vectorless retrieval planning adds seconds. For real-time applications this is a hard constraint.
Scale of Document Corpus
When the document corpus grows to tens or hundreds of thousands of pages, passing all summaries to the LLM in a single context becomes infeasible. Strategies like two-stage retrieval (coarse filtering then reasoning) partially address this, but at the cost of the architecture's simplicity. At very large scale, vector search plus reranking remains the more pragmatic choice.
Rapidly Changing Content
If documents update at high frequency, re-generating LLM summaries for changed pages creates a real-time indexing burden. Vector embedding updates are faster and cheaper per document update than LLM-based re-summarisation.
Vectorless RAG in Agentic AI Systems
The relationship between Vectorless RAG and agentic architectures is significant and worth examining separately. In a standard RAG pipeline, retrieval is a fixed, single-pass operation: query in, chunks out, generate. In an agentic system, the agent can iterate — retrieve, reason, identify gaps, retrieve again.
Vectorless RAG integrates naturally with this loop. The retrieval planning step is already an act of reasoning — the agent can extend it: after receiving an initial answer, it can inspect the confidence and completeness of the response, identify which claims are insufficiently grounded, and issue a follow-up retrieval plan targeting the specific gaps. This iterative, reasoning-driven retrieval is harder to implement cleanly with vector search, where you cannot ask the index "what did I miss?"
Agentic Retrieval Loop with Vectorless RAG
This pattern — plan, retrieve, reflect, re-plan — is what makes Vectorless RAG especially well-suited to agentic architectures built on long-context, reasoning-capable models. As context windows expand and inference costs decrease, the case for reasoning-based retrieval over similarity-based retrieval in enterprise settings will only strengthen.
Choosing the Right Approach: A Decision Framework
Use Vectorless RAG when…
Stick with vector search when…
The most robust enterprise systems will not choose one pattern exclusively. A hybrid architecture — vector search for broad corpus recall, Vectorless RAG for precision retrieval on shortlisted documents, agentic iteration for complex multi-hop queries — combines the throughput advantages of vector search with the reasoning advantages of LLM-based selection.
Key Takeaways
Vectorless RAG replaces vector similarity with LLM reasoning to plan retrieval — asking which documents are needed rather than which chunks are similar.
The PageIndex pattern builds a summary index of document pages, then uses the LLM to select relevant pages by reasoning over those summaries rather than computing cosine distances.
Structural document integrity is preserved: full pages are retrieved, not reconstructed chunks — a decisive advantage for tables, clauses, and cross-referenced enterprise documents.
The retrieval reasoning trace is a native audit trail, not post-hoc explanation. This matters significantly in regulated industries where retrieval decisions must be defensible.
Agentic architectures benefit most: the reasoning loop can iterate, identify gaps, and issue follow-up retrieval plans in a way that vector search — which has no concept of 'what did I miss' — cannot.
Vectorless RAG is not a universal replacement. At high query volume, large corpus scale, or where sub-second latency is required, vector search remains the practical choice.
Retrieval is not a solved problem — it is an evolving design space. The shift from similarity search to reasoning-based selection reflects a broader pattern in AI systems: as models become more capable reasoners, the logic that was hardcoded into infrastructure can move into the model itself. The question is not which approach is correct, but which failure modes you are most exposed to.
Dr. Nabanita Sinha
Associate Director | AI & Consulting · Author · Mentor
