Evaluate a classifier when classes are imbalanced

A classifier can achieve high accuracy by predicting only the common class. If a synthetic dataset contains 990 ordinary messages and 10 important alerts, a classifier that always predicts ordinary is 99 percent accurate while finding no alerts.

Start evaluation with the confusion matrix and class support. For alert detection, define alert as the positive class. Record true positives, false positives, false negatives and true negatives so the operational consequences remain visible.

Consider another synthetic prediction set with six correctly detected alerts, four false alerts, four missed alerts and 986 correctly ignored ordinary messages.

tp, fp, fn, tn = 6, 4, 4, 986
precision = tp / (tp + fp)
recall = tp / (tp + fn)
accuracy = (tp + tn) / (tp + fp + fn + tn)
assert precision == 0.6
assert recall == 0.6
assert accuracy == 0.992

Those values are arithmetic for an invented fixture. They are not results from a project or trained model. The scikit-learn evaluation guide documents precision, recall and related metrics.

True and predicted labels determine the confusion matrix used to calculate class-specific metrics.

Connect the errors to the task

Precision asks how many predicted alerts were actually alerts. Recall asks how many actual alerts were found. Choose which errors matter from the product's use: four false alerts may waste review time, while four missed alerts may leave important work unseen.

For an all-negative prediction, the precision denominator is zero. Report how the implementation handles undefined metrics instead of silently comparing its default value with a meaningful measured proportion. The recall is zero when the positive class has support but none are detected.

Keep class support beside every aggregate score. A macro average gives classes equal weight, while a support-weighted average can conceal weak performance on a rare class. Inspect per-class results before deciding an aggregate meets the requirement.

Review thresholds and dataset boundaries

A model that emits scores needs a decision threshold. Changing it trades false positives against false negatives, so choose and evaluate it using a validation process that avoids tuning on the final test set.

Ensure related examples do not leak across splits. Repeated messages, records from one user or observations close in time can make evaluation easier than the intended deployment task. Choose the split around the source of dependence.

A tiny synthetic fixture verifies metric calculations but cannot estimate production performance. Use representative labeled data, inspect uncertain labels and report the evaluation period. If the underlying class frequency changes, precision can change even when the classifier's conditional behavior appears similar.