Use datafusion's re-exported arrow and parquet

[trueleo]

Description

Datafusion re exports arrow and parquet. Use these for version compatiblity with datafusion.

Changes

• Remove direct dependency on arrow and parquet, use datafusion::arrow and datafusion::parquet.
• Remove other unused dependencies.

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.