RAG
Industrial RAG: how to measure answer quality
Evaluate answers from engineering manuals with source relevance, device context and appropriate abstention.
How do you reduce LLM hallucinations?
Short answer
A hallucination is an answer the model produces that has no support in its sources but reads as fluent and certain. To reduce them, ground answers in retrieved documents, require citations, and make the system decline when no evidence is found. Then measure the result on a fixed question set. No method removes errors completely.
Why hallucinations happen
A language model predicts the next likely token. By default it does not look up a database, so when information is missing it can still produce a plausible answer.
Three situations are especially risky. First, the question is ambiguous: which machine, which firmware version? Second, the information is missing from the training data or is out of date. Third, the model cannot check its own answer against a source.
In industry this can look like a part number that does not exist, an invented alarm description, or a wrong temperature limit. Hallucinations may never be eliminated fully, but addressing these three causes helps lower the risk.
Example: an ambiguous alarm question
Suppose an operator asks, 'What does alarm 4 on the pump mean?' (hypothetical). The question names no model or firmware version, so the model may guess. The fix has two steps: first the system asks for the model and version, then the answer is drawn only from the matching manual. If no passage supports an answer, the reply is 'This information is not in the selected documents.'
Measured result (hypothetical): a fixed test set of 50 questions has 40 answerable and 10 unanswerable ones. On the 10 unanswerable questions, the system said the information was missing 8 times and gave an invented answer 2 times: 80% correct abstention and 20% hallucination. Of the 40 answerable questions, 36 were answered correctly, which is 90%.
Limits and common mistakes
A citation alone is not enough. A model can attach a real paragraph that does not support the sentence next to it, so check that each claim is supported by the cited text.
The opposite error also occurs: if the refusal threshold is too strict, the system says 'not found' too often and users stop trusting it. A test set made only of easy questions gives a flattering score; include real shift questions and ambiguous ones.
If you tune the system on the test set and then report the score on that same set, the result is inflated. After any change to the model or the documents, run the test again from the start.
Follow-up questions
Does lowering the temperature setting prevent hallucinations?
No. A lower temperature makes the output more repeatable, but it does not make it true. A wrong statement stays wrong at low temperature. Grounding in sources and measured testing are still needed.
Can citation checks be automated?
Partly. Code can confirm that each cited passage was actually retrieved, and that numbers, units and part numbers in the answer appear in the cited text. Whether a passage really supports a claim needs human review or a separate evaluation step.
Sources
Example numbers are hypothetical, not measured field results.