RAG
A RAG chunking strategy for technical manuals
Splitting a manual into arbitrary-length pieces makes the right answer get searched in the wrong context. Preserve headings, tables and revision information.
Industrial RAG: how to measure answer quality
Evaluate answers from engineering manuals with source relevance, device context and appropriate abstention.
In short
- Retrieval success does not prove answer correctness, so each technical claim in a generated answer should be linked to a supporting passage.
- When no source covers a question, the right behavior is to name the evidence gap and ask for context, not to invent a parameter.
- A pilot acceptance record should freeze the evaluation set before changes and include definitions, comparisons, troubleshooting and out-of-scope requests.
Define the engineering question
A request such as “Why does the drive stop?” lacks the product family, fault code and operating conditions. Ask for those details before proposing settings. A matching code from another device family can produce a plausible but technically wrong answer. Include the model, document version, symptoms and checks already performed in your NQ question.
Evaluate retrieval and generation separately
Keep a test table with the question, expected document, supporting section and acceptable answer. First check whether the correct document appears in the first five results. Then link each technical claim in the generated answer to a supporting passage. Successful retrieval does not establish answer correctness. An obsolete manual may look relevant while being inapplicable.
Include unanswerable questions
Test an absent product, conflicting revisions and a question missing its fault code. The expected behavior is to describe the evidence gap and request the missing context. Inventing an exact parameter from general model knowledge is not a successful answer. A retrieval score is a ranking signal, not a calibrated probability of correctness.
Keep a pilot acceptance record
For a proposed pilot, include definitions, comparisons, troubleshooting and out-of-scope requests. Freeze the evaluation set before changes. Record the code version, collection version, latency and source links. Have two engineers review disputed answers. This is a suggested engineering workflow; it does not constitute a measured production accuracy claim.
Frequently asked
How do I evaluate RAG answers from an engineering manual?
Start with a test table holding each question, the expected document, the supporting section and an acceptable answer. Check whether the correct document appears in the first five results, then link every technical claim in the generated answer to a supporting passage. Successful retrieval alone does not establish that the answer is correct.
How should a RAG system handle questions with no answer in the documents?
Include an absent product, conflicting revisions and a question missing its fault code in the test set. The expected behavior is to describe the evidence gap and request the missing context. Inventing an exact parameter from general model knowledge does not count as success, and a retrieval score is only a ranking signal.
Sources and further reading
Pilot steps in this guide are proposed engineering workflows. Examples do not constitute measured field performance.
Questions on this topic
Keep reading.
Full library
RAG
Making LLMs Reliable for Industrial Q&A: Evidence-First Design
General chatbots hallucinate; industrial answers must be verifiable. This article describes the evidence-first design pattern: curated corpora, hybrid retrieval, constrained synthesis, and inspectable output.
RAG
Word Embeddings and Semantic Search: How Modern Retrieval Works
Semantic search matches meaning, not spelling. This article explains embeddings, vector similarity, and why hybrid keyword-plus-vector retrieval is the robust production pattern.