Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This pull request addresses two critical issues in the
Non-Sensitivitymetric related to thefeatures_in_stepparameter and the internal logic of pixel perturbations.Logical inconsistency in perturbation evaluation
When
features_in_step > 1, multiple pixels (both feature and non-feature) are perturbed simultaneously within the same step.As a result, the computed difference between the perturbed and original predictions (
y_pred) corresponds to mixed pixel groups, making it impossible to determine whether individual pixels preserve the model’s insensitivity property.Shape mismatch causing
ValueErrorWhen running the metric with
features_in_step != 1, Quantus raised:This occurred at:
where:
non_features→ shape(batch_size, n_features)preds_differences→ shape(batch_size, n_perturbations)Since
n_perturbations != n_featuresfor multi-step perturbations, the XOR (^) operation failed due to incompatible dimensions.These issues caused both logical misinterpretation of sensitivity violations and runtime failures during evaluation.
Link to the issue
https://github.com/understandable-machine-intelligence-lab/Quantus/issues/367Implemented changes
Minimum acceptance criteria
tests/metrics/test_non_sensitivity_metric.pyand related evaluation modules pass successfully across supported environments (py310–py311).features_in_stepconfigurations without shape or logic errors.