Day 04 of the WClickHouse Open-Source Engineering Series.
Row serialization is the biggest bottleneck in Python analytics. WClickHouse insert_arrow() moves contiguous columnar memory blocks directly to ClickHouse.
The Pain Points We Faced
- Python object conversion overhead serializing millions of numbers into JSON/tuples
- Garbage collection pauses freezing data engineering pipelines during ingestion
- High RAM consumption caused by duplicating analytical datasets in Python heap
The Implementation
import pyarrow as pa
from wclickhouse import WClickHouse
# Insert native Arrow Table directly without Python row iteration
arrow_table = pa.table({"id": [1, 2, 3], "metric": [99.4, 98.1, 99.9]})
db = WClickHouse(MetricModel, db_config)
db.insert_arrow(arrow_table)
# Query directly into Arrow
result_arrow = db.query_arrow("SELECT * FROM metricmodel WHERE metric > 99.0")
Why This Architecture Wins
- Zero-Copy Memory: Apache Arrow columnar format transfers directly over socket.
- insert_arrow(): Stream millions of rows from Parquet or Arrow tables in seconds.
- query_arrow(): Extract analytical query results directly into pyarrow.Table.
Verification & Status
Tested and verified against live ClickHouse server instances with 95%+ test coverage. Built for Python 3.9 through 3.14 with Apache Arrow and Pydantic v2.
Top comments (0)