xmlstreamer-ng
Fault-isolating XML feed streamer built on rypipe, adapted from carlosplanchon/xmlstreamer. Bounded memory, conservative recovery, explicit partial failures.
A rypipe-based adaptation of carlosplanchon/xmlstreamer that streams huge or damaged XML feeds into Python records without hiding corrupted items.
Loading latest commits…
Parallel throughput
15.8M rows/s
Throughput
2.12 GB/s
Bounded parallel
10.2M rows/s
Peak RSS (bounded)
202 MB
Problem
Large XML feeds from remote systems are often malformed, mis-encoded, or interrupted in transit. A strict parser loses the whole document to one broken record; an aggressive recovery parser can silently rewrite damaged markup into records that never existed.
xmlstreamer-ng is built for the space between those two behaviours: deliver every intact item before and after localized corruption, never invent records, and report what was dropped instead of presenting incomplete processing as success.
Origin
xmlstreamer-ng is based on carlosplanchon/xmlstreamer by Carlos A. Planchón (MIT), and keeps its streaming, item-based approach to large or malformed XML feeds. The parsing engine is built on rypipe (rypipe-core), which provides the byte tokenizer, splitter, per-item parsing and Arrow execution.
Why it's hard
Recovery has to be conservative without becoming useless. Separator tags must be recognized inside comments, CDATA, processing instructions and DOCTYPE while never being invented by them. Encoding detection must win over a lying XML declaration without corrupting valid UTF-8. Bounded memory must hold while items, markup sections and descriptors can each be larger than the read buffer.
The columnar path adds a second contract: the same Rust engine has to produce Arrow batches under projection, filtering and parallel execution, with malformed-item drop semantics identical to the streaming path.
Performance
Latest benchmarks on a Ryzen 7 5800X over a 128 MiB feed with 951,898 items: the parallel XmlSource (8 threads) reaches 15.8M rows/s and 2.12 GB/s; bounded parallel streaming reaches 10.2M rows/s and 1.36 GB/s at 202 MB peak RSS.
Benchmark results
| Engine | Mode | items/s | MB/s | Peak RSS |
|---|---|---|---|---|
| xmlstreamer-ng | XmlSource parallel (8 threads) | 15.8M | 2122.6 | 283 MB |
| xmlstreamer-ng | Bounded parallel (8 threads) | 10.2M | 1364.7 | 202 MB |
| xmlstreamer-ng | XmlSource columnar | 4.6M | 621.6 | 297 MB |
| quick-xml (native) | parse + count only | 3.2M | 433.0 | 14 MB |
| lxml iterparse | per-item dicts | 304k | 40.8 | 26 MB |
| xmlstreamer (original) | per-item dicts | 103k | 13.8 | 34 MB |
128 MiB synthetic feed, 951,898 items, median of 5 runs on a Ryzen 7 5800X. Native rows parse and count only; they do not build records.
Key decisions
Two APIs, one engine
StreamInterpreter keeps the per-item recovery contract; XmlSource exposes Arrow projection, filtering and parallel batches from the same Rust parser.
Explicit failure accounting
Malformed items are logged and counted, never silently repaired. The closing stats distinguish feed damage from items removed by the caller's filter.
Conservative recovery
Corruption contained inside an item drops that item only. Sections that never close are a documented choice via Sections(on_limit=...), announced with a warning.
Guarantees
Tradeoffs
- Separator tag must not nest inside itself
- Items are built from element text, not XML attributes
- Untrusted, well-formed input may be better served by lxml.iterparse
- Per-item API is slower than the columnar/parallel adapter