Data

The data service provides the data loaders that feed the training, validation, and test loops. It turns the dataset into shuffled, batched graphs delivered to the model, and centralizes the knobs that govern batching, shuffling, and worker parallelism.

Usage

Configured under services.data.

services:
  data:
    batch_size: 2048
    buffer_size: 16384
    chunk_size: 4096
    num_workers: 8
    prefetch_factor: 8
    mp_context: fork
    persistent_workers: true

How it works

Samples are read in contiguous chunks and passed through a shuffle buffer so that batches can mix across chunk boundaries without holding the whole dataset in memory. A pool of worker processes reads and assembles batches ahead of the model to reduce I/O latency. The service builds and caches one loader per requested split.

Configuration

Option	Description	Type	Default
batch_size	Number of samples per batch.	int	required
chunk_size	Number of samples read as one contiguous block.	int	required
buffer_size	Size of the shuffle buffer, in samples. Chunks are mixed within it; aim for a buffer-to-chunk ratio above 10:1.	int	required
shuffle_chunks	Whether to shuffle whole chunks as blocks.	bool	false
num_workers	Number of worker processes loading data.	int	required
prefetch_factor	Number of batches each worker prefetches.	int	required
mp_context	Multiprocessing start method.	fork | spawn | forkserver	required
persistent_workers	Keep worker processes alive between epochs.	bool	required