Inference

The inference engine applies a trained model to new data to produce predictions.

Note

Currently only batched inference has been implemented. Realtime streaming inference is planned for a future update.

Usage

A batch inference engine is built from a configuration and run with execute; see the usage guide for a complete script.

from icegraph.inference import BatchInference

with BatchInference.from_yaml(config_path) as inference:
    inference.execute()

Configuration

The inference configuration restores its model from a checkpoint rather than configuring it from scratch:

Option	Description	Type	Default
outdir	Directory where predictions and outputs are written.	path	required
model_path	Path to the trained model checkpoint to run.	path	required

It uses the state, record, decode, and data services, and restores the model, normalizer, and transformer components from the checkpoint. No policy is required.

How it Works

The engine loads the checkpoint, reconstructing the model and its associated components from the stored weights and buffers, then streams the dataset through the model and produces the predictions.

Warning

Running inference in distributed mode may currently drop chunks so that the sample count matches across ranks, which can lose data. Run inference on a single process until this is resolved.

• Inference Callbacks
  • Registering
  • Writing a callback