Skip to content

Conversation

Edwardvaneechoud
Copy link
Owner

This pull request refactors the location of the calculate_fuzzy_match_schema and pre_calculate_pivot_schema functions, updates the pivot schema logic, and improves test coverage for pivot-related functionality. The main changes involve moving schema-related logic to a more appropriate module, refining how pivot output columns are named, and adding new tests to ensure correct schema calculation.

Refactoring and code organization:

  • Moved calculate_fuzzy_match_schema and pre_calculate_pivot_schema from flow_data_engine/fuzzy_matching/settings_validator.py to a new schema_callbacks.py module for better separation of concerns and maintainability.

Pivot schema logic improvements:

  • Updated the logic in pre_calculate_pivot_schema to generate simpler column names (e.g., '0', '1', etc.) when only one aggregation is used, instead of appending the aggregation name (e.g., '0_sum').
  • Adjusted test assertions to match the new column naming convention for pivot outputs.

Testing enhancements:

  • Added new tests test_pivot_schema_callback and test_schema_callback_in_graph to verify correct schema calculation for pivot operations and ensure the new schema logic is functioning as intended.
  • Updated imports in the test suite to reflect the new location of schema callback functions.

@Edwardvaneechoud Edwardvaneechoud merged commit 3da4a59 into main Aug 15, 2025
12 checks passed
@Edwardvaneechoud Edwardvaneechoud deleted the fix/alignment_schema_callable_pivot branch August 15, 2025 11:15
Bennylave pushed a commit to Bennylave/Flowfile that referenced this pull request Aug 26, 2025
* Fixing handling of pivot

* Fixing test for pivot table
Bennylave pushed a commit to Bennylave/Flowfile that referenced this pull request Aug 26, 2025
* Fixing handling of pivot

* Fixing test for pivot table
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant