Move to Arrow 42 and Datafusion 27

[trueleo]

Description

• Update datafusion dependencies along with related arrow dependencies.
• Dependencies list is reorganized

User facing changes

This PR removes AWS profile support as it is removed in object-store. We will reintroduce this feature later by using aws-config crate directly as a credential provider for object store

---

This PR has:

• been tested to ensure log ingestion and log query works.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.