Stream 50 million rows without OOM: Lazy chunked Query Streaming.
Day 09 of the WClickHouse Open-Source Engineering Series.
You don't need 64GB of RAM to process millions of ClickHouse records in Python. WClickHouse query_stream() keeps your memory footprint below 80MB.
The Pain Points We Faced
- Out-Of-Memory (OOM) killer crashing Python containers when fetching large results
- Cursor fetchall() loading 10GB of query data into Python RAM at once
- Slow pipeline startup waiting for entire query results before processing first row
The Implementation
db = WClickHouse(SensorReading, db_config)
# Stream 50 million rows in 50,000-row chunks: RAM never exceeds 80MB!
for chunk in db.query_stream("SELECT * FROM sensorreading", chunk_size=50000):
process_batch(chunk)
print(f"Processed chunk of {len(chunk)} rows cleanly.")
Why This Architecture Wins
- Lazy Generator: query_stream() yields chunks (e.g., 50,000 rows) on demand.
- Constant RAM Footprint: Memory stays under 80MB whether scanning 10k or 50M rows.
- Immediate First Row: Start processing pipeline logic immediately as first chunk arrives.
Verification & Status
Tested and verified against live ClickHouse server instances with 95%+ test coverage. Built for Python 3.9 through 3.14 with Apache Arrow and Pydantic v2.
Top comments (1)
When benchmarking
stream_query()against massive ClickHouse datasets (>50M rows), the memory footprint is primarily governed by the TCP socket buffer and chunk serialization overhead. In our load tests, chunk sizes between 50k and 100k rows achieved peak network utilization while keeping RSS memory pinned strictly below 120MB.What chunk sizing strategies do your teams employ when streaming OLAP query results into memory-constrained pipeline workers?