PageIndex vs Vector RAG: Reasoning Over a Table of Contents Instead of Embeddings

[!NOTE] TL;DR

Vector RAG retrieves what looks like the question. In long financial filings, manuals, contracts and specs, the answer often sits in a section that doesn't look like the question at all.

PageIndex (VectifyAI, MIT-licensed) replaces the vector index with a hierarchical table-of-contents tree. An LLM reads the tree and decides which sections to open, the way you skim a ToC.

What you gain: retrieval that follows document structure and cross-references, and citations that point to a section and a page range.

What you pay: several LLM calls per query, added latency, and a dependency on documents having usable structure (and text, so scanned PDFs need OCR first).

For a corpus with many documents, use both: BM25 or vectors to pick the documents, then tree reasoning inside each one.

I spend a lot of my time turning messy documents into structure: long PDFs, drawing sets, spreadsheets that someone exported from a system nobody remembers. On those documents I have watched vector RAG fail the same way over and over. It returns a confident top-5 of chunks that are on topic and still useless.

PageIndex is VectifyAI's open-source take on a fix. Their README calls it "vectorless, reasoning-based RAG", and its first line is the thesis of this post: similarity ≠ relevance.

I'll walk through why that's true, how tree-based retrieval works, and how to build a small version yourself in plain Python. Then I'll be honest about where it breaks and when I'd pick each approach.

Why similarity is not relevance

Embedding search answers one question: which chunks are closest to the query in vector space? That is a good proxy for relevance when the answer is phrased like the question. Think FAQs, support tickets, short articles.

Professional documents are different. Ask a 10-K "What interest rate applies to the term loan?" and the nearest chunks will probably be:

the glossary entry that defines "term loan",
an MD&A paragraph saying interest rates rose this year,
a risk factor about loan covenants.

All three are similar. None of them contains the rate. The rate sits in a debt footnote, maybe in a table with almost no prose around it, maybe behind a "see Appendix G" pointer. Similarity search misses what is relevant but not similar, and returns what is similar but not relevant.

Vector RAG ranks chunks by closeness to the query. The three hits are on topic, but the page that holds the answer, a dense debt footnote, never reaches the top k (illustrative 10-K).

This isn't a tuning problem you can fix with a better embedding model. A chunk that has been cut out of its section has lost the thing that made it relevant: where it sits in the document. A human analyst doesn't search a 10-K by similarity. They open the table of contents, go to Item 8, find the debt note, and read it.

What PageIndex does instead

PageIndex works in two steps (from the README):

Index: build a tree-structure index for each document.
Retrieve: an LLM searches that tree agentically, reasoning about which branch to open.

There is no vector DB and no fixed-size chunking. The "chunks" are the document's own sections. Each tree node carries a title, the page range it covers, a node id, and a summary. At query time the model sees the outline, not a ranked list of fragments, so it can apply the domain knowledge you'd apply yourself ("interest rate on debt lives in the debt footnote, not the risk factors").

A few things the README states that I think matter:

The tree structure is extracted from the document layout without an LLM. The index model only summarises and refines it, so the README recommends a basic, cheap model for indexing and the strongest model you can afford for search.
The open-source local mode handles text-based PDFs with page-level citations. OCR and image understanding are listed as Cloud features.
VectifyAI reports 98.7% accuracy on FinanceBench with their Mafin 2.5 system built on PageIndex, against roughly 50% for vector RAG in their chart. That is their own benchmark claim; the evaluation is published in their Mafin2.5-FinanceBench repo.

The package has a Python client (pip install -U pageindex). I'm not going to copy its API here, because SDK surfaces change and the README is the place to get it right. Everything below is my own illustrative implementation of the same idea, small enough to read in one sitting. It is not PageIndex's code.

Building the tree

Tree building has three jobs:

Find the structure. Use a printed table of contents if the document has one. Otherwise infer headings from layout: font size, bold, numbering like 4.2.1, or the # levels a PDF-to-markdown parser emits.
Assign page ranges. A section runs from its heading's page until the next heading at the same or a higher level starts.
Summarise nodes. A one or two line summary per node lets the search model choose between "Note 12 Leases" and "Note 14 Debt" without reading either.

Tree building is mostly deterministic: find headings, nest them, compute page ranges. The LLM only writes short node summaries, once per document.

Here is the data structure and the builder. I assume a parser has already given you one markdown string per page, which most PDF-to-markdown tools can do.

from __future__ import annotations
 
import re
from dataclasses import dataclass, field
 
HEADING = re.compile(r"^(#{1,6})\s+(.+?)\s*$")
 
 
@dataclass
class Node:
    node_id: str
    title: str
    level: int
    start_page: int
    end_page: int = 0
    summary: str = ""
    children: list[Node] = field(default_factory=list)
 
 
def extract_headings(pages: list[str]) -> list[tuple[int, str, int]]:
    headings = []
    for page_no, text in enumerate(pages, start=1):
        for line in text.splitlines():
            if match := HEADING.match(line):
                headings.append((len(match.group(1)), match.group(2), page_no))
    return headings
 
 
def build_tree(headings: list[tuple[int, str, int]], last_page: int) -> Node:
    root = Node("0000", "ROOT", level=0, start_page=1, end_page=last_page)
    stack = [root]
    for i, (level, title, page) in enumerate(headings, start=1):
        while stack[-1].level >= level:
            stack.pop().end_page = page  # boundary page is shared on purpose
        node = Node(f"{i:04d}", title, level, page)
        stack[-1].children.append(node)
        stack.append(node)
    for still_open in stack[1:]:
        still_open.end_page = last_page
    return root
 
 
def section_text(pages: list[str], node: Node) -> str:
    return "\n".join(pages[node.start_page - 1 : node.end_page])

The stack walk is the whole trick. When a new heading arrives, every open section at the same or deeper level is closed at that page. Sections share their boundary page, because a heading in the middle of page 97 means page 97 belongs to both. Overlap is cheaper than losing half a table.

Summaries are one cheap call per node, done once at index time:

def summarise(node: Node, pages: list[str], llm, max_chars: int = 6000) -> None:
    for child in node.children:
        summarise(child, pages, llm, max_chars)
    if node.level > 0:
        text = section_text(pages, node)[:max_chars]
        node.summary = llm(
            f"Summarise this section in one line (under 20 words), naming the "
            f"concrete facts it contains.\n\nTitle: {node.title}\n\n{text}"
        ).strip()

[!TIP] Ask for summaries that name the facts a section contains ("term loan rate, maturity, covenants"), not what it is about ("discusses debt"). The search model picks branches from these lines, and topical summaries bring back the same similarity problem one level up.

Serialising the tree for the prompt

The search model needs the outline in as few tokens as possible. One line per node, indented by depth, with the id and page range. Never dump the whole tree as JSON; the braces and quotes cost tokens and add nothing.

def outline(node: Node, max_depth: int = 2, _depth: int = 0):
    for child in node.children:
        summary = f" :: {child.summary}" if child.summary else ""
        pages = f"pp. {child.start_page}-{child.end_page}"
        yield f"{'  ' * _depth}[{child.node_id}] {child.title} ({pages}){summary}"
        if _depth + 1 < max_depth:
            yield from outline(child, max_depth, _depth + 1)
 
 
def index_tree(root: Node) -> tuple[dict[str, Node], dict[str, Node]]:
    by_id, parent, stack = {}, {}, [root]
    while stack:
        node = stack.pop()
        by_id[node.node_id] = node
        for child in node.children:
            parent[child.node_id] = node
            stack.append(child)
    return by_id, parent

max_depth matters. For a 300-page filing, the full tree can run to hundreds of nodes. Show the top level always, plus the subtree around wherever the model currently is, and the prompt stays small no matter how big the document gets.

Tree-search retrieval

This is the part that makes it "reasoning-based". The model gets the question and the outline, and picks an action: open a section to see its children, read a section's text, go back up, or stop. It can backtrack when a branch turns out to be wrong, and it can jump straight to a node it saw referenced in text it read ("see Appendix G").

Tree search on an illustrative 10-K: the model narrows from the top level to the debt note, reads the term-loan subsection, follows its "see Appendix G" pointer, and answers with section and page citations.

Here is the loop. llm(prompt) -> str is whatever client you use. The model replies with one JSON object per step, and I validate it at the boundary, because models will sometimes wrap JSON in prose or invent node ids.

import json
 
ACTIONS = {"open", "read", "back", "answer"}
 
PROMPT = """You are navigating a document's table of contents to answer a question.
Question: {question}
 
Top-level sections:
{top}
 
You are inside: {here}
Its subsections:
{local}
 
Sections read so far:
{read}
 
Recent steps:
{log}
 
Reply with ONE JSON object: {{"action": "open|read|back|answer", "node_id": "...", "reason": "..."}}
- open: move into a section to see its subsections
- read: fetch a section's text
- back: return to the parent section
- answer: stop, the sections read are enough
If text you read points elsewhere (e.g. "see Appendix G"), read that node directly."""
 
 
@dataclass
class Evidence:
    node_id: str
    title: str
    start_page: int
    end_page: int
    text: str
 
 
def parse_action(raw: str) -> dict:
    match = re.search(r"\{.*\}", raw, re.DOTALL)
    if not match:
        raise ValueError("no JSON object in reply")
    action = json.loads(match.group(0))
    if action.get("action") not in ACTIONS:
        raise ValueError(f"unknown action {action.get('action')!r}")
    return action
 
 
def render_prompt(question, root, cursor, evidence, log) -> str:
    read = "\n".join(f"[{e.node_id}] {e.title}:\n{e.text}" for e in evidence)
    return PROMPT.format(
        question=question,
        top="\n".join(outline(root, max_depth=1)),
        here=f"[{cursor.node_id}] {cursor.title}",
        local="\n".join(outline(cursor, max_depth=2)) or "(leaf section)",
        read=read or "(none)",
        log="\n".join(log[-6:]) or "(none)",
    )
 
 
def tree_search(question, root, pages, llm, max_steps=10, max_reads=4,
                excerpt_chars=3000) -> tuple[list[Evidence], list[str]]:
    by_id, parent = index_tree(root)
    cursor, evidence, log = root, [], []
    for step in range(max_steps):
        try:
            action = parse_action(llm(render_prompt(question, root, cursor, evidence, log)))
        except ValueError as err:  # JSONDecodeError is a ValueError
            log.append(f"{step}: invalid reply ({err}); reply with one JSON object")
            continue
        kind, target = action["action"], by_id.get(str(action.get("node_id", "")))
        if kind == "answer":
            break
        if kind == "back":
            cursor = parent.get(cursor.node_id, root)
        elif target is None:
            log.append(f"{step}: unknown node_id {action.get('node_id')!r}")
            continue
        elif kind == "open":
            cursor = target
        elif kind == "read" and all(e.node_id != target.node_id for e in evidence):
            text = section_text(pages, target)[:excerpt_chars]
            evidence.append(Evidence(target.node_id, target.title,
                                     target.start_page, target.end_page, text))
        log.append(f"{step}: {kind} {action.get('node_id', '')} - {action.get('reason', '')}")
        if len(evidence) >= max_reads:
            break
    return evidence, log

Some decisions I'd defend:

Hard limits on steps and reads. Without them, an unsure model will wander. When it hits the limit, you answer with what you have, or say you couldn't find it. Both beat a loop that never ends.
Bad output costs a step, not the run. The error goes back into the log, so the next prompt tells the model what it did wrong.
The top level is always visible. That's what makes backtracking and cross-reference jumps cheap: the model never has to climb back up one level at a time to reach Appendix G.
excerpt_chars is a known ceiling. A long section gets cut. If your sections are big, have the model open it and read subsections instead, or split leaves by page.

The answer step is just a grounded prompt over the evidence, and the citations come for free because every piece of evidence carries its node id and page range:

def answer(question: str, evidence: list[Evidence], llm) -> str:
    context = "\n\n".join(
        f"[{e.node_id}] {e.title} (pp. {e.start_page}-{e.end_page})\n{e.text}"
        for e in evidence
    )
    return llm(
        "Answer using only the sections below. Cite every claim as "
        f"[node_id, pages]. If they don't contain the answer, say so.\n\n"
        f"{context}\n\nQuestion: {question}"
    )

A scripted dry run, no model needed

You can test the loop's control flow with a fake llm that replays fixed replies. This is how I'd unit-test it: the tree logic is deterministic, only the model isn't.

pages = [
    "# Item 7 MD&A\nRates rose this year.",
    "# Item 8 Financials\n## Note 14 Debt\nTerms below.",
    "### 14.2 Term loan\nRate per schedule, see Appendix G.",
    "# Appendix G\nTerm loan: SOFR + 2.25%.",
]
root = build_tree(extract_headings(pages), last_page=len(pages))
replies = iter([
    '{"action": "open", "node_id": "0002", "reason": "debt is in the financials"}',
    'Sure! {"action": "open", "node_id": "0003", "reason": "debt note"}',
    '{"action": "read", "node_id": "0004", "reason": "term loan"}',
    '{"action": "read", "node_id": "0005", "reason": "cross-reference"}',
    '{"action": "answer"}',
])
evidence, log = tree_search("Term loan rate?", root, pages, llm=lambda _: next(replies))
assert [e.title for e in evidence] == ["14.2 Term loan", "Appendix G"]
assert (evidence[1].start_page, evidence[1].end_page) == (4, 4)

Note the second reply has prose around the JSON. parse_action copes with that.

Traceability is the underrated win

Accuracy gets the headlines. In practice, what makes reviewers trust a system is being able to check it.

A vector hit is "chunk 4,812, similarity 0.81". A tree hit is "Note 14 Debt › 14.2 Term loan, pages 97–98, reached via Item 8". The second one tells a reviewer where to look and also why the system looked there. The search log is a readable trail of decisions, which is exactly what you want when a finance or legal user asks "where did this number come from?"

That trail also makes debugging honest. When tree search gets it wrong, you can usually point to the step: a summary that didn't mention the right fact, or a heading the parser missed. When vector search gets it wrong, you usually get to shrug at a cosine score.

A minimal vector-RAG baseline

For contrast, here is the baseline in about as few lines as it takes. embed(text) -> list[float] stands in for your embedding model.

import math
 
 
@dataclass
class Chunk:
    page: int
    text: str
 
 
def chunk_pages(pages: list[str], size: int = 800, overlap: int = 200) -> list[Chunk]:
    return [
        Chunk(page_no, text[start : start + size])
        for page_no, text in enumerate(pages, start=1)
        for start in range(0, max(len(text), 1), size - overlap)
    ]
 
 
def cosine(a: list[float], b: list[float]) -> float:
    norm = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b))
    return sum(x * y for x, y in zip(a, b)) / norm if norm else 0.0
 
 
def vector_search(question: str, chunks: list[Chunk], embed, k: int = 5) -> list[Chunk]:
    vectors = [embed(c.text) for c in chunks]  # precompute and store these in practice
    q = embed(question)
    ranked = sorted(zip(chunks, vectors), key=lambda cv: cosine(q, cv[1]), reverse=True)
    return [chunk for chunk, _ in ranked[:k]]

It's one embedding call per query and a nearest-neighbour lookup, so it runs in milliseconds and costs almost nothing. That's not a small advantage. Notice also what it throws away: the chunk knows its page, but not its section, its parent, or that it's the paragraph right after the table it describes.

Costs and failure modes

Tree search isn't free, and I don't trust posts that pretend a new retrieval method has no downside.

[!WARNING] Tree search is a sequential agent loop. Every step is a round trip to the model, and each step depends on the one before, so you can't parallelise within a single document. Budget for seconds per query, not milliseconds.

Latency and cost per query. A typical search is several LLM calls plus the answer call, and the search model needs to be a strong one. The README is explicit that the search model should be the best you can afford. Indexing is a one-time cost; querying isn't.
Documents with no structure. A 40-page email thread, a transcript, or a PDF of scanned forms has no meaningful ToC. You can infer pseudo-sections (by page, by topic shift), but at that point the tree is not giving the model much to reason with, and plain chunking is honest about that.
Scanned PDFs. No text layer means no headings. You need OCR first, and OCR errors in headings become wrong branches. The open-source local mode is scoped to text-based PDFs; the README lists OCR as a Cloud feature.
Bad structure is worse than none. If the parser flattens heading levels or misses a numbering scheme, the tree lies with confidence. Validate trees on a sample: check page ranges are monotonic and that the top level looks like the real ToC.
Tables and drawings. A node summary of "Schedule of maturities" doesn't say which years it covers. The model may need to read the node to know. That's fine, but it costs a step.
Very large corpora. Tree search works inside a document. It doesn't tell you which of 50,000 documents to open. You still need a first-stage retriever.

Head to head

	Vector RAG	Tree search (PageIndex-style)
Unit of retrieval	Fixed-size chunks	The document's own sections
Index build	Chunk + embed, no LLM needed	Parse structure, then LLM summaries per node
Query-time cost	One embedding + ANN lookup	Several sequential LLM calls
Latency	Milliseconds	Seconds
Best at	Short docs, FAQs, many small items, fuzzy topical recall	Long structured docs: filings, manuals, contracts, specs
Cross-references	Not followed	Followed by reading the referenced node
Citations	Chunk and page	Section path and page range
Explainability	A similarity score	A readable log of navigation decisions
Scales to	Millions of chunks, many documents	Deep within one document; needs a first stage across many
Breaks when	Answer isn't phrased like the question	Document has no or wrong structure, or no text layer

The hybrid I'd actually build

For anything beyond a handful of documents, I don't see this as either/or. Use the cheap, wide retriever to choose documents, and the expensive, deep one to read them.

The hybrid: a cheap first stage narrows the corpus to a few documents, tree search reads each one deeply (in parallel across documents), and the answer carries section and page citations.

The shape:

Stage 1, pick documents. BM25 plus vector search over chunks or node summaries, aggregated to the document level, plus metadata filters (company, year, doc type). This is where vectors shine: fuzzy recall over a big haystack, fast.
Stage 2, read inside. Run tree_search on the top few documents. The loop is sequential inside one document but independent across documents, so run them concurrently.
Answer. Merge evidence, answer once, cite node paths and pages.

from concurrent.futures import ThreadPoolExecutor
 
 
def hybrid_answer(question, first_stage, trees, pages_by_doc, llm, top_docs=3):
    doc_ids = first_stage(question)[:top_docs]
    with ThreadPoolExecutor(max_workers=top_docs) as pool:
        results = pool.map(
            lambda d: tree_search(question, trees[d], pages_by_doc[d], llm), doc_ids
        )
        evidence = [e for found, _log in results for e in found]
    return answer(question, evidence, llm)

In a real system you'd prefix node ids with the document id so citations stay unambiguous across documents. I left that out to keep the sketch short.

A nice side effect: node summaries are excellent things to embed for stage 1. They're short, factual, and already scoped to one section, which makes them better retrieval targets than arbitrary 800-character windows.

Why not just put the whole PDF in the context window?

For a short document, do that. It's the simplest thing that works. The README makes the cost argument against doing it for long ones: in their measurements, native PDF input cost 2.1× more than PageIndex retrieval at 52 pages and 16.6× more at 420 pages, and an 805-page document didn't fit in the context window at all. Those are VectifyAI's numbers for their setup, but the direction is obvious: full-context cost grows with document length on every question, while tree search only reads the nodes it reaches.

Decision guide

Pick vector (or BM25) RAG when:

documents are short, or each chunk stands on its own (FAQs, tickets, product listings);
you need millisecond latency or very high query volume;
questions are topical ("anything about warranty claims?") rather than precise;
documents have no structure worth navigating.

Pick tree search when:

documents are long and well structured: filings, manuals, contracts, standards, technical specs;
answers are precise, and wrong answers are expensive;
cross-references matter ("as defined in Section 2", "see Appendix G");
users need section-and-page citations they can check;
a few seconds per query is acceptable.

Use the hybrid when you have both a large corpus and long structured documents, which in my experience is most real document work.

And whichever you pick, measure it on your own questions. Write 30 to 50 questions with known answers and known pages, and score retrieval (did it read the right section?) separately from generation (did it say the right thing?). That small eval set will settle the argument faster than any blog post, this one included.

[!TIP] Most of the quality in tree search comes from the tree, not the search prompt. If results are bad, inspect the parsed headings and page ranges before touching the prompt. A missed heading level is a far more common bug than a model making a poor choice.

Closing thought

Vector RAG treated documents as bags of chunks because that was the only thing cheap enough to do at scale. Models are now good enough, and cheap enough, to read a table of contents and make a sensible choice. For long professional documents, that's closer to how the people who use these documents actually work.

It doesn't replace vectors. It moves them to where they're strong, finding the right documents, and lets reasoning handle what's inside them.