Collector

The collector sits between the extractor and the processors of every pipeline. It packs the envelopes of whole input files into blocks. Each block is passed to the processors as one envelope and written as one shard, so shards can be much larger than the files they come from.

How it Works

Files are collected per group (see group_by). A block closes once its extracted tables reach min_bytes and the next file of its group holds events, which starts the next block. A file without events never starts a block, it always joins one and is counted in it. Once all input files have been extracted, any remaining partially filled block is emitted.

A file is never split across blocks. When files are merged:

their tables are stacked per key, in the order the files arrived.
their event ids must not repeat, since the block is processed as one file. A repeated event raises.
their global attributes must be identical.
each file’s local attributes are kept under attrs.LOCAL.sources, keyed by the origin file path:

sources:
  /path/to/a.i3.zst: { dropped: { ... } }
  /path/to/b.i3.zst: {}  # held no events

Every shard records each input file in it, with events or without, so file-count sensitive operations (like weighting, see i3-simulation) count every file that was processed. Shards are numbered in the order their blocks close, which is used to name the output shard (see Writer). Which files share a shard depends on the order they finish extraction, so it can differ between runs.

The collector runs in one process, in addition to nproc.

Note

A group whose files hold no events at all cannot be written and raises.

Configuration

Configured under the top-level collector key. It can be left out to use the defaults.

Option	Description	Type	Default
`min_bytes`	Size of extracted tables at which a block closes. `0` gives one file with events per block, along with any files without events that follow it.	int	`0`
`group_by`	Which files may share a block. `parent`: files in the same directory. `all`: every file, for when every directory holds the same set.	str	`parent`

extractor:
  ...
collector:
  min_bytes: 100_000_000
  group_by: parent
processors:
  ...

A block’s working frames are held in memory while it is processed, and each processor thread can work on a different block, so choose min_bytes with available memory in mind.