Skip to content

Add support for Azure data lake storage authentication and google cloud storage to flowfile core #100

@Edwardvaneechoud

Description

@Edwardvaneechoud

At the moment there is support for s3 connections. We would like to extend this to Azure data lake storage and Google cloud storage.

Modifications that have to happen:

  • flowfile_core/flowfile_core/flowfile/flow_data_engine/cloud_storage_reader.py: The parsing of configs is handled here
  • flowfile_core/flowfile_core/flowfile/flow_data_engine/flow_data_engine.py: Depending on if the current set-up handles also scenario's for adls and gcs there might be some refactoring needed to it.
  • In the CI test template add Azurite and GCS emulator for testing purposes.

Metadata

Metadata

Assignees

Labels

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions