Feature/unify execution methods #110
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request refactors how execution location and worker offloading are handled throughout the codebase. It removes the "auto" execution location, clarifies the distinction between local and remote execution, and introduces utility functions for determining and validating execution locations based on global settings. Additionally, it updates related logic in data sampling, node execution, and flow graph management to align with these changes.
Execution Location and Offloading Refactor
ExecutionLocationsLiteral
, restricting execution locations to "local" or "remote", and added utility functions (get_global_execution_location
,is_valid_execution_location_in_current_global_settings
,get_prio_execution_location
) to determine and validate execution location based on global settings.settings.py
to useMutableBool
forSINGLE_FILE_MODE
andOFFLOAD_TO_WORKER
, and decoupled their logic for clearer control over worker offloading.Flow Graph and Node Execution Logic
FlowGraph
to encapsulateflow_settings
with getter/setter, ensuring the graph resets when execution location or mode changes, and updated the handling of execution location in methods and initialization. [1] [2] [3] [4]FlowNode
logic to default to "worker" execution, removed checks forSINGLE_FILE_MODE
in execution decisions, and improved error handling for remote worker connection issues. [1] [2] [3] [4]Data Engine and Sampling
FlowDataEngine.get_sample
to accept an explicitexecution_location
parameter, use the new global execution location logic, and simplify how the number of records is determined for sampling.get_number_of_records_in_process
method to clearly distinguish between local and worker-based record counting, and updated related usages for clarity and correctness. [1] [2] [3]Backward Compatibility and Defaults
Miscellaneous