How We Build RAG at Rearway: From Basic Retrieval to Contextual Hybrid Search

If you've been following the AI space, you've probably come across terms like RAG, LLMs, LangGraph, and vector databases. Before diving into how these work, here's something important we've learned building Resolve and our other AI products at Rearway: there is no universal architecture. Every RAG implementation we build is specific to the problem it's solving.

Let's break this down from first principles.

What Is an LLM, Really?

A Large Language Model is, at its core, a next-token predictor. Given a sequence of tokens (words, characters, code), it uses learned mathematical relationships to predict what comes next. That's it. The "intelligence" emerges from doing this at scale, across an enormous amount of human-generated text.

Understanding this helps you make better architectural decisions — because the quality of what the model generates is directly tied to the quality of what you put in its context.

Why Vectors? Why a Vector Database?

If you've studied mathematics, you know that vectors are one of the most powerful ways to represent anything in the world — text, images, audio, code. What makes them especially useful for AI is this property: objects that are closer together in vector space are more similar in meaning.

When you convert your knowledge into vector embeddings, you give the LLM a way to find semantically related content fast. Instead of keyword matching, it understands meaning. "Revenue declined" and "sales dropped" end up close in vector space even though they share no words.

This is the foundation of RAG.

Do You Actually Need RAG?

Not always — and this is a point we're deliberate about at Rearway.

Anthropic's own documentation is clear on this: if your knowledge base is under roughly 200,000 tokens (about 500 pages of content), just put it in the system prompt. The simplest approach is usually the best one. We follow this principle across our products, including Resolve, which is why it can spin up a trained bot in under 60 seconds for smaller knowledge bases.

RAG becomes necessary when:

• Your knowledge base is large (500+ pages)
• Your data is dynamic and changes frequently
• You need source attribution or chunk-level traceability

For everything else, a well-crafted system prompt wins.

Classic RAG: The Baseline

When the knowledge base is too large for the context window, RAG is the answer. The process at build time:

• Break the knowledge base into chunks
• Create a vector embedding for each chunk
• Store everything in a vector database

At query time, when a user asks a question, their query is also converted into an embedding. The vector database performs a semantic similarity search to find the most relevant chunks, which are then injected into the model's prompt alongside the query.

This is Retrieval-Augmented Generation — retrieve relevant information, augment the prompt, generate the response.

The Limitation Nobody Talks About Enough

Embedding models are excellent at capturing semantic relationships, but they can miss exact matches. This is a real problem in production.

Imagine a support bot trained on technical documentation. A user asks about error code ED-1001. An embedding model might retrieve chunks about "error handling" or "system diagnostics" — semantically adjacent, but not what the user needs. They need the exact match.

This is where BM25 (Best Match 25) comes in.

BM25 is a classical lexical ranking algorithm that builds an inverted index of all terms in your knowledge base. It's particularly effective for:

• Specific error codes and identifiers
• Technical jargon and product names
• Exact phrase queries

It also handles common failure modes gracefully — applying a saturation function so a term appearing hundreds of times in one document doesn't artificially dominate results, and normalizing for document length so short and long documents compete fairly.

Hybrid Search: The Best of Both Worlds

At Rearway, we combine embeddings and BM25 in what's called hybrid search. Here's our standard pipeline:

At index time:

• Break the knowledge base into chunks
• Create semantic embeddings for each chunk
• Build a BM25 (TF-IDF) index in parallel

At query time:

• Convert the user query to an embedding
• Run a semantic search against the vector database
• Run a BM25 lexical search in parallel
• Merge the two result sets, deduplicate, and re-rank
• Take the top-K chunks and inject them into the model's context

The result is a retrieval system that handles both "explain how refunds work" (semantic) and "what is error ED-1001" (lexical) with equal accuracy.

Contextual Retrieval: Taking It Further

Standard hybrid search still has a subtle problem: chunks lose context when they're split from their source documents.

Consider a financial knowledge base with 1,000 pages of reports. A chunk that reads "The company's revenue grew by 3% over the previous quarter" is accurate, but nearly useless in isolation. Which company? Which quarter? Which year?

This is the problem Contextual Retrieval solves — an approach Anthropic has written about and one we've adopted in Resolve.

Before embedding, we run each chunk through the AI model itself to generate a contextual prefix. The transformation looks like this:

That chunk now carries enough context to be genuinely useful when retrieved. Semantic search can find it. BM25 can index the specific identifiers. And the model gets something it can reason with.

Our Full Pipeline at Rearway

Putting it all together, this is how we build RAG for production at Rearway:

Ingestion:

Query:

This is the architecture powering Resolve's ability to give precise, business-specific answers — not generic AI responses, but answers grounded in your actual content.

The Takeaway

RAG is not a single technique — it's a spectrum of architectural decisions. Start simple (system prompt). Graduate to classic RAG when scale demands it. Add BM25 when precision matters. Add contextual retrieval when your chunks lose meaning in isolation.

At Rearway, we match the architecture to the problem — not the other way around. That's what lets Resolve say it resolves, not deflects.

Rearway Technologies is an AI software company based in Kozhikode, Kerala, building intelligent products including Resolve (AI support automation), Tandoor AI (restaurant operations), and Factory Flow (smart manufacturing).