Statistics

The statistics subsystem computes the dataset-level summaries that the rest of the framework relies on. A statistic is a streaming, per-column accumulator: it is folded over the data, merges with other accumulators, and resolves to a value on demand. The stats processor uses these to summarize a dataset during processing, and the engine’s normalizer reads the results back to build its per-column scale and offset.

Usage

A StatisticService is constructed with the set of statistics to track (by name), folded over one or more arrays, and then queried for base or derived values. Each column of a 2D array is accumulated independently.

import numpy as np

from icegraph.statistics import StatisticService
from icegraph.common.transforms import TransformSpace

# track a set of statistics over the data
stats = StatisticService(["mean", "m2", "min", "max", "finite_count"])

# fold an array in (rows are samples, columns are tracked independently)
stats.compute_from_array(np.asarray(data))

# read a base statistic, optionally in a transform space or in a specific base
mean = stats.get("mean")
mean_log = stats.get("mean", space=TransformSpace.LOG)
mean_log_base4 = stats.get("mean", space=TransformSpace.LOG, base=4)

# read a derived quantity, computed from the tracked bases
std = stats.std()
value_range = stats.range()

Derived quantities require their underlying bases to be tracked.

Services computed separately (for example per shard or per process) combine exactly, and serialize for storage and reload:

combined = StatisticService.merge([stats_a, stats_b])   # or: sum([stats_a, stats_b])

struct = stats.to_struct()
restored = StatisticService.from_struct(struct)

How it works

A statistic service holds a bundle of statistics and accumulates them together. Each statistic is tracked simultaneously in several transform spaces (linear, log, and inverse-hyperbolic-sine), so downstream code can request, say, the mean in log space without recomputation. Accumulators are mergeable, which lets statistics be computed in parallel over shards and combined exactly, and they serialize into the dataset so they travel with it.

Statistics fall into two groups. Base statistics are accumulated directly from the data (the counts, the running mean, the Welford second moment, the minimum, and the maximum). Derived quantities are computed from those bases on request; the service exposes the standard deviation, variance, range, geometric mean, standard error, coefficient of variation, RMS, and signal-to-noise ratio. Because some bases depend on others when merging (the mean and second moment use the count statistics), selecting a derived quantity generally means selecting the counts it builds on as well.

Variants

Running values:

  • mean: per-column mean.

  • m2: Welford second moment, the basis for variance and standard deviation.

  • min: per-column minimum.

  • max: per-column maximum.

Counts:

Registering a new statistic

A statistic is a subclass of Statistic that declares a name, the transform spaces it is tracked in, and its degree under linear rescaling, and implements two operations: folding an array into the accumulator and merging two accumulators. Register it with StatisticFactory. Statistics take no configuration.

from typing import ClassVar

from icegraph.typing.common import ArrayF
from icegraph.common.transforms import TransformSpace
from icegraph.statistics.statistic import Statistic
from icegraph.statistics.factory import StatisticFactory

class MyStatistic(Statistic):
    name: ClassVar[str] = "my-statistic"
    spaces: ClassVar[tuple[TransformSpace, ...]] = TransformSpace.all()
    degree = 1

    def _compute(self, array: ArrayF) -> ArrayF:
        ...  # reduce one array to a per-column value

    @classmethod
    def _merge(cls, a, b, space) -> ArrayF:
        ...  # combine two accumulators in the given space

StatisticFactory.register(MyStatistic)