Query Transformation in RAG: How Your App Rewrites Questions Before It Searches

Say you've built a support chatbot on top of your product docs. A customer types, "why is my application not working?" Your docs have the answer, filed under "application startup failure." Same problem, different words, and the search comes back with nothing useful.

Here's the part that trips people up when they first read about this: the customer doesn't fix that, and neither do you, by hand. Your app does it. Between the moment the customer hits Enter and the moment your system searches, a small step rewrites the question into something the search can actually match. The customer never sees it. They type the way they always type and get an answer.

That step is called query transformation. In this guide we'll walk through what happens in the backend, the six techniques teams use (rewriting, expansion, conversational context, decomposition, HyDE and step-back prompting), what each one actually sends to the search, and how to tell whether it's helping.

Who does this, and when?

• Runs: on every question, after the user asks and before the search.
• Done by: your backend code. Usually one LLM call, sometimes simple rules.
• What the user sees: nothing. They type normally. The rewritten query is only used to search.

Key Takeaways

• Query transformation happens inside your app, not in the user's head. The user's original question is still the one that gets answered.
• Searching with an LLM-written "fake answer" (HyDE) lifted zero-shot retrieval quality from 44.5 to 61.3 nDCG@10 on TREC DL19 (
• Every technique comes down to one question: what text gets embedded and searched?

Where Does Query Transformation Sit in a RAG Pipeline?

A RAG system has two halves, and query transformation lives in the second one. The first half prepares your documents ahead of time. The second half runs every time someone asks a question. Rewriting the query as its own step before retrieval, the "Rewrite-Retrieve-Read" pattern, was shown to improve answers on open-domain QA (Ma et al., EMNLP, 2023).

Each box on the indexing side has its own guide: preparing documents for ingestion , chunking strategies for RAG, and embeddings and vector databases. After retrieval, reranking sorts what came back.

Why put the fix on the question side at all? Because you can tidy your documents forever and users will still type "it's broken." The vocabulary gap starts with them, so that's where the fix goes.

What Actually Happens in the Backend?

Your code takes the user's question, asks an LLM to rewrite it for search, searches with the rewrite, and then answers the original question using what it found. Four steps, one of them invisible to the user. Let's follow a single request through.

1. The user types something vague.

"My Spring Boot app won't connect to the database. What should I do?"

2. Your backend sends that to an LLM with a rewrite instruction. This is the whole "transformation." It's a prompt, not magic:

You rewrite user questions into search queries for our technical documentation.
- Keep product names, versions and error codes exactly as written.
- Use the terms our docs use. Don't add facts the user didn't give.
- Return only the rewritten query.

Conversation so far: (earlier messages mention MongoDB)
User question: My Spring Boot app won't connect to the database. What should I do?

3. The LLM returns a search query, and your app searches with it.

"Spring Boot MongoDB connection failure: connection string, authentication, network"

That line gets turned into an embedding and compared against your document chunks. It now shares vocabulary with the docs, so it actually matches.

4. The final answer responds to the original question. The retrieved chunks go to the LLM alongside what the user actually typed, not the rewrite. The rewrite was only ever a search key.

Does it have to be an LLM? No. A synonym list ("dead" → "not responding") or a template covers plenty of cases, and it's free and instant. LLMs earn their cost when the phrasing is loose, when context from earlier messages matters, or when you need something like HyDE.

Which Query Transformation Technique Fixes Which Problem?

Each technique fixes a different kind of bad question, and you, the developer, pick which ones to wire in when you build the app. Some teams add a small "router" step that picks per question. Here's the short version before the detail.

1. What Is Query Rewriting?

Query rewriting rephrases the question in clearer, more technical words without changing what it means. It's the simplest technique and the one most support tools need first, because users rarely know the exact term for what's broken.

User typed:      "Why is my API dead?"
App searches:    "Why is my API server not responding to requests?"

Nothing about the meaning changed. It's just wearing language that's more likely to match a technical doc. The one rule that matters: keep the specifics. If the user typed ERR-4021 or Spring Boot 3.2, those must survive the rewrite.

2. What Is Query Expansion?

Query expansion turns one short query into several variants and searches all of them. LLM-based expansion improved the classic BM25 keyword search by 3% to 15% on MS MARCO and TREC DL without any model training (Wang et al., Query2doc, 2023). Frameworks often ship this as a "multi query retriever."

User typed:      "MongoDB indexing"
App searches:    • How do indexes work in MongoDB?
                 • What are the different types of MongoDB indexes?
                 • How do indexes improve query performance?
                 • How can MongoDB index usage be checked?

Four searches means four result lists. So how do they become one? Your code removes duplicates and merges them, usually with reciprocal rank fusion: a chunk that ranks high in several lists ends up near the top of the combined list (Cormack et al., SIGIR, 2009). A reranker can then sort the merged list more carefully.

The risk is drift. Pile on loosely related variants and you pull in noise, not signal. Three to five variants that stay close to the original intent is a sensible starting point.

3. How Does Conversational Context Help?

Contextual transformation rewrites a follow-up question so it makes sense on its own. Nobody repeats themselves in a real conversation, and a chatbot shouldn't make them.

Earlier message: "What is an index in MongoDB?"
User typed:      "How does it improve performance?"
App searches:    "How does an index improve query performance in MongoDB?"

The search engine has no memory of the chat. Without this step, it searches for "how does it improve performance" and returns anything about performance. Your backend fills in "it" from the conversation history before searching.

What if there's no history to draw from? Then don't guess. If someone asks "why can't I connect to it?" as their first message, the honest move is to ask what "it" is. Confidently searching for the wrong thing is worse than a clarifying question.

4. What Is Query Decomposition?

Query decomposition splits a question that's really several questions into separate sub-questions, searches each one, and combines the results. It earns its keep when each part deserves its own evidence.

User typed:   "Compare MongoDB and PostgreSQL on performance, indexing,
               scalability and data modeling."
App searches: • MongoDB vs PostgreSQL performance
              • Indexing in MongoDB vs PostgreSQL
              • Scalability of MongoDB vs PostgreSQL
              • Data modeling in MongoDB vs PostgreSQL

One search for the whole question tends to return chunks about whichever topic dominates. Four searches get you evidence for all four. The results are merged the same way as expansion, and the LLM writes one answer from the combined set.

5. What Is HyDE in RAG?

HyDE (Hypothetical Document Embeddings) asks an LLM to write a fake answer to the question, then searches with that fake answer instead of the question. In the original paper, it raised zero-shot retrieval quality on TREC DL19 from 44.5 to 61.3 nDCG@10, close to a fully trained retriever at 62.2 (Gao et al., arXiv, 2022). nDCG@10 is a 0 to 100 score for how well the top 10 results are ordered.

Here's the flow, all in the backend:

1. The user asks a question.
2. Your app asks an LLM to write a plausible answer. The LLM has no access to your documents, and the answer is allowed to be wrong.
3. Your app embeds the fake answer, not the question.
4. That embedding searches your real documents.
5. The real documents that come back are used to write the actual answer. The user never sees the fake one.

User typed:   "How do I fix a MongoDB connection timeout error?"
Fake answer:  "MongoDB connection timeouts are usually caused by network latency,
               incorrect connection string parameters, or firewall rules blocking
               the port. Check the connection string's timeout settings, verify
               network connectivity to the host, and confirm the firewall allows
               traffic on the MongoDB port."

Why search with something that might be wrong? Because questions and answers don't look alike. A fake answer, even an inaccurate one, is written in the same shape and vocabulary as real documentation, so its embedding lands much closer to the real answer than the question's does.

Two technical rules apply. The fake answer must be embedded with the same embedding model you used for your documents, or the vectors can't be compared (see why). And it helps to ask for an answer about as long as one of your chunks, since that's what it's being matched against.

The costs are real: an extra, longer LLM call before every search, and the risk that a confidently wrong fake answer drags the search in the wrong direction. HyDE also works best on topics the LLM already knows something about. On a narrow internal knowledge base, its guess may look nothing like your documents.

6. What Is Step-Back Prompting?

Step-back prompting asks a broader question first, retrieves on that, and uses what comes back to answer the specific question. Researchers at Google DeepMind reported gains of 7% on MMLU Physics, 11% on Chemistry and 27% on TimeQA with this approach (Zheng et al., ICLR, 2024).

User typed:   "Why does my MongoDB query with a compound index on
               {status, created_at} return results out of order?"
App searches: "How does MongoDB use compound indexes for sorting query results?"

The step-back version drops the field names and the symptom and asks about the mechanism. That broader question matches the general documentation page, the one that never mentions the user's field names but explains the rule their query is breaking.

Decomposition narrows a question into pieces. Step-back widens it into a concept. They're sometimes combined: step back for grounding, then decompose the specifics. The risk runs the other way from expansion: go too broad and you retrieve general material that never connects back to the actual problem.

What Actually Gets Embedded in Each Technique?

Every technique above reduces to one question: what text gets converted into a vector and sent to search? Once you look at it that way, the differences stop feeling arbitrary. In our experience, it's also the fastest way to debug a pipeline that's returning odd results.

How Does Query Transformation Differ From Reranking?

Query transformation fixes the question before the search. Reranking fixes the order of results after the search. They solve different problems and often run together in the same pipeline.

Retrieval usually pulls back a batch of "probably relevant" chunks. A reranker then reads the question alongside each chunk and moves the most useful ones to the top before they go to the LLM. Our guide to reranking in RAG covers how that works. The short rule: if the right chunk never shows up in the results, rewrite the query. If it shows up but too low, rerank.

Where Does Query Transformation Go Wrong?

It goes wrong when the rewrite changes what the user meant or drops what they said. Anthropic found that adding context to each chunk before embedding cut failed retrievals by 35%, and by 49% with keyword search added (Anthropic, 2024). That's the document-side twin of this technique, and it's a reminder that both sides matter.

Where it helps:

• Vague or casual questions become searchable.
• It closes the gap between how people talk and how docs are written.
• Follow-up questions work without users repeating themselves.
• Multi-part questions get evidence for every part.

Where it bites:

• Misread intent. The rewrite searches for the wrong thing entirely.
• Lost specifics. Error codes, version numbers and product names get "cleaned up" away. Keyword search is exactly what finds those, so run the keyword side of a hybrid search on the original query, not the rewrite.
• Latency and cost. An LLM call on every question adds up at scale. HyDE adds a full generation step.
• Drift. Expansion pulls in noise. Step-back goes too broad.
• Missing information. No rewrite can find what isn't in the index. Sometimes it was in the source file and got lost when the document was parsed, which is an ingestion problem , not a query problem.

How Do You Know It's Actually Helping?

Test it the same way you'd test any retrieval change: real questions, known answers, before and after. We use the same approach in our chunking guide's test plan, and it works here unchanged.

1. Collect 30 to 50 real user questions, messy phrasing included, each with the chunk that answers it.
2. Run them with no transformation and record how often the right chunk lands in the top 5 results.
3. Turn on one technique. Run the same questions again.
4. Keep it only if the hit rate goes up enough to justify the extra latency and cost.

Change one thing at a time. If you change chunk size and add query rewriting in the same run, you won't know which one helped.

Conclusion

Query transformation closes the gap between how people ask questions and how documents are written, and it does it out of sight. Your users keep typing the way they always have. Your backend rewrites, expands, fills in context, splits, imagines an answer, or steps back, and then searches with that instead.

You don't need all six. Start by logging what users actually type and what gets retrieved. Where a vague or context-dependent question is clearly causing misses, add the technique that matches that problem, then measure. Sometimes the original question was fine all along.