Classification Metrics

The classification metrics are the family of metrics for categorical targets. They read each head’s columns as scores over that head’s classes and its single target column as a class index, which is the layout the multiclass policy produces.

How it works

Heads are ragged so the family walks them one at a time rather than working across the packed row, pairing a head’s scores with its labels and rejecting a target block that is not one column per head.

The metrics here range from a confusion matrix to a binned score histogram, and each declares its own state. What they share is how the model output is read, and that splits them in two:

  • Arg-max metrics look only at which class scored highest. They are invariant to whether the model emits logits, log-probabilities or probabilities, and take no configuration for it.

  • Distribution metrics read the predicted probabilities themselves. These take a from_logits option, true by default.

Confusion-matrix metrics

ConfusionMetric pins the accumulator to one [K, K] matrix per head, indexed [true, predicted]. Counting over two batches and adding is counting over their union, so the monoid is trivial and folding across batches or merging across processes is exact.

  • Macro-F1: F1 averaged with equal weight per class.

  • Macro-Recall: recall averaged with equal weight per class.

  • Balanced Accuracy: accuracy as if every class were equally represented, optionally adjusted so chance scores 0. In its default form this is the same quantity as macro-recall.

  • Per-Class Recall: recall reported one value per class rather than one per head.

  • Cohen’s Kappa: agreement against what the class marginals would reach by chance.

A class the evaluation set never contains has an undefined score. These metrics report it as nan per class and leave it out of any macro average, rather than counting it as a zero that would drag the average down.

Other metrics

  • Top-K Accuracy: fraction of samples whose true class is within the top k scores. Arg-max style, but a rank count rather than a confusion matrix, and it runs across the whole packed row at once.

  • Cross-Entropy: mean negative log-probability of the true class.

  • Expected Calibration Error: gap between stated confidence and observed accuracy, over binned confidences.

  • AUPRC: macro-averaged area under the precision-recall curve, over binned scores.

Registering a new classification metric

A metric that is a pure function of the confusion matrix subclasses ConfusionMetric and implements only reduce, the function from one head’s matrix to that head’s 1-D value. recall, which returns per-class recall alongside the support to mask undefined classes with, is available on the base.

# plugin.py
from typing import ClassVar

from torch import Tensor

from icegraph.engine.services.metrics.metric import MetricFactory
from icegraph.engine.services.metrics.metric.variants.classification.confusion import ConfusionMetric
from icegraph.engine.services.metrics.metric.variants.classification.config import Config

class Specificity(ConfusionMetric[Config]):
    name: ClassVar[str] = "specificity"
    version: ClassVar[int] = 1

    @property
    def optimum(self) -> float:
        return 1.0

    def repr(self) -> str:
        return "specificity"

    def reduce(self, confusion: Tensor) -> Tensor:
        ...  # 1-D value for this head, from its [K, K] matrix

MetricFactory.register(Specificity)

Anything else subclasses ClassificationMetric directly, declaring its own accumulator type and the full monoid over it. The family still supplies the shared readers:

heads(out, target)

Walk the heads, yielding (head, scores [B, K], labels [B]), with the one-target-column-per-head invariant checked.

probabilities(scores, *, from_logits)

Class probabilities for one head’s scores.

macro(values, mask)

Mean over the selected entries as a 1-element tensor, dropping the rest rather than counting them as zero.

A metric that takes no options inherits the family’s empty Config. One that does declares its own config model and overrides validate_config alongside its binding of the config type.