DATA · INGESTION FRAMEWORK
rypipe
Reusable ingestion framework. Rust core, Python bindings. Add a new format by implementing two traits.
Extracted from the original crxml execution engine and generalized into a reusable data-processing framework.
What is rypipe
rypipe is a reusable ingestion framework, not a single-format parser. It separates format-specific parsing from format-agnostic execution. Add a new format by implementing two small traits: Splitter and RecordParser.
rypipe itself does not ship parsers. Those live in separate adapter packages. Install the engine plus the adapters you need. The framework handles parallel scheduling, memory-bounded execution, query pushdown, and a chainable pipeline API.
Problem
Parsing row-oriented byte streams into columnar formats (Apache Arrow) usually requires writing a full engine per format. Each format gets its own parallel scheduler, memory manager, and execution infrastructure. The result is duplicated effort and inconsistent performance characteristics across formats.
Why it's hard
Each new data format usually requires writing a full engine: parallel scheduler, memory manager, execution infrastructure. rypipe's difficulty is separating format-specific parsing (two small traits) from format-agnostic execution (parallel scheduling, bounded memory, query pushdown, zero-copy Arrow export) while maintaining throughput at memory bandwidth. The adapter contract test suite verifies that any adapter produces identical output across single-thread, parallel, and bounded execution modes.
Constraints
- ·Bounded memory: cannot load entire files into RAM
- ·Parallel execution must scale linearly with cores
- ·Pipeline stages must fuse into the parse loop (no Python round-trip)
- ·Adapter API must be minimal: two traits, not a full engine
Key decisions
Two-trait adapter model
Adding a new format requires implementing Splitter and RecordParser. The engine handles everything else.
Predicate-first evaluation
Filters push down into the Rust parse loop. Rows that fail are rejected before any Python object is created.
Layout prediction via memcmp
Predicts row structure from byte patterns, skipping column discovery when schema is known.
Zero-copy Arrow export
Data moves from parse buffers directly into Arrow arrays without intermediate copies.
Origin
rypipe was originally developed as the ingestion engine for crxml, a Crystal Reports XML adapter. The engine's design, performance characteristics, and API were shaped by real-world production use with crxml. crxml is the reference adapter that proved this model: complex nested schemas, large files, parallel processing, and advanced filtering.
Tradeoffs
- ·Adapters must be written per format (not a universal parser)
- ·Rust core adds build complexity vs. pure Python
- ·Schema must be known upfront for maximum throughput