Zero-copy select and return for pyarrow tables

[danthegoodman1]

Being able to select pyarrow tables without copying, as well as accessing results as pyarrow tables without copying would be massively beneficial to building low-latency ETL pipelines and other data processing pipelines.

Specifically having streaming support would be massive too: https://duckdb.org/2021/12/03/duck-arrow.html#streaming-data-fromto-arrow as this would greatly reduce the required memory usage for queries and post-processing of data

[auxten]

Faster path of query on ArrowTable is done on v2.0.0b1
Example:
https://github.com/chdb-io/chdb/blob/main/tests/test_query_py.py#L94

[auxten]

@auxten Has there been any updates on this?

We currently run a C++ application that uses duckdb in memory to store a bunch of data and runs user queries on it.

The ingestion into the DB is handled via DuckDB's arrow interface C API:

https://duckdb.org/docs/stable/clients/c/api.html#duckdb_arrow_array_scan

This allows us to convert our inmemory state to an Arrow Array (with defined schema) and DuckDB reads it into a Table.

Having something like this with chDB will be great as it'll allow us to do bulk inserts efficiently.

Originally posted by @dufferzafar in #187