KutupASP Dijital
TROpen NQ
ML

Alert thresholds, precision and recall

Why a single accuracy percentage can hide failure on rare events.

September 13, 20262 minKutup · ASP Dijital
In short
  • If only ten of a thousand observations are faults, always predicting normal reaches 99% accuracy while missing every fault.
  • Precision measures how many alerts are correct, while recall measures how many actual events were detected.
  • Thirty consecutive alerts around one failure are not thirty independent successes, so warnings should be grouped into event windows.

Expose the denominator

If only ten of a thousand observations are faults, always predicting normal produces 99% accuracy while missing every fault. This is an illustrative calculation, not a field result. Include true positives, false positives, false negatives and true negatives in the acceptance table.

Separate two questions

Precision asks how many alerts are correct. Recall asks how many actual positives were detected. A zero denominator makes a metric undefined; display that state instead of an arbitrary percentage. F1 combines precision and recall but does not automatically encode operational cost.

Select thresholds on validation data

A lower threshold typically produces more warnings. Evaluate the benefit and cost together. For a pilot, write down maintenance capacity and the impact of a missed event. Freeze the threshold before the final test. An uncalibrated model score must not be described as a failure probability.

Report event-level performance

Thirty consecutive alerts around one failure are not thirty independent successes. Group warnings into event windows and document repeat-notification rules. False alarms per shift and first-warning lead time are useful operational measures. Retain point-level metrics too, since they answer a different question.

Frequently asked

How do I choose an alert threshold for precision and recall?

Select the threshold on validation data and freeze it before the final test. A lower threshold typically produces more warnings, so evaluate benefit and cost together. Write down maintenance capacity and the impact of a missed event, and do not describe an uncalibrated model score as a failure probability.

Why is accuracy misleading for rare failure events?

When faults are rare, always predicting normal can score high accuracy while catching no fault at all. Include true positives, false positives, false negatives and true negatives in the acceptance table. Report event-level results too, because thirty consecutive alerts around one failure are not thirty independent successes.

Sources and further reading

Pilot steps in this guide are proposed engineering workflows. Examples do not constitute measured field performance.

Questions on this topic

What is alarm flooding, and how can it be prevented?CUSUM, EWMA or Shewhart: which should I use, and when?What is anomaly detection in sensor data?How do I reduce false alarms in an anomaly detection model?How do I split time series data into training and test sets?How do precision and recall apply to alarm thresholds?

Keep reading.

Full library