Optim wip - Move & restructure loss objectives #527

ProGamerGov · 2020-11-13T22:20:58Z

I moved all loss objectives from captum.optim._core.objectives to captum.optim._core.loss as per: #500 (comment)

I changed the loss functions to classes and made it so that they were building off an abstract loss class. The new classes work with the existing objective optimization code, so no changes to that were needed.
The tutorials were also updated to reflect the loss changes.
The loss changes should be compatible with / not be in conflict with: Optim-WIP: Preliminary transform unit tests & more #521
Added a ton of missing type hints.
I added the 3 Direction objectives that we were missing. They've been tested and work with the current objective system.
Added Channel Interpolation objective, and made it possible to use without selecting specific channels.
Added layer objective.
Added alignment objective.
Added weight objective (all 3 weight objectives merged into a single objective class).
I got LaplacianImage working. It seems to work pretty good for feature visualization!
I added an init parameter to FFTImage, LaplacianImage, and NaturalImage.
FFTImage, NaturalImage, PixelImage, LaplacianImage, InputOptimization, and ToRGB all support batch sizes now!
Added decorrelate_init & squash_func parameters to NaturalImage.
Added size property / function to ImageTensor, as size is more PyTorch-like than shape I think.
Fixed major bug where errors / interrupts to the InputOptimization class' optimize function resulted in activations being None in subsequent runs.
Custom input images now also work correctly!

Move loss objectives to their own file

ProGamerGov · 2020-11-13T22:27:43Z

@NarineK I left the loss objective functionality the exact same, so that @greentfrapp and I can change them as we tackle issues like: #496 (comment), and allowing for non mean & penalty losses.

This PR is ready for review as I will likely not be making any major changes to it for the moment.

pytorch#527

* https://pytorch.org/docs/stable/generated/torch.nn.ParameterList.html Previously PyTorch had issues accessing the parameters inside LaplacianImage.

Fix LaplacianImage & add init option to NaturalImage

Handle input images

* Add missing type hints. * Add decorrelate_init & squash_func parameters to NaturalImage.

* Because why not

NarineK

Thank you for moving the losses into a separate file and extending from the Loss ABC, @ProGamerGov! It looks like, there are a couple of losses that assume NCHW input shape - we could group them together or leave as is. It's probably fine to leave them as is right now.

In general I think that it would be great if we could document and explain the losses a bit more in detail and point out to the related papers.
I also added a couple inline comments. Let me know what you think.

captum/optim/_core/loss.py

NarineK · 2020-11-22T01:56:14Z

captum/optim/_core/loss.py

+        pass
+
+    def get_neuron_pos(
+        self, H: int, W: int, x: Optional[int] = None, y: Optional[int] = None


Isn't this a specific logic for images ? Perhaps we can keep this function in _utils package ?

It's specific logic meant to be used along with activations in the form of NCHW, and I've moved it to a new _utils/images.py file.

NarineK · 2020-11-22T02:35:24Z

captum/optim/_core/loss.py

+        H_vec, W_vec = self.direction.size(2), self.direction.size(3)
+        H_activ, W_activ = activations.size(2), activations.size(3)
+
+        H = (H_activ - W_vec) // 2


nit: H_vec ?

I had it as H_vec because of the input variable name vec, but I've changed it to be H_direction now as that seems more readable.

NarineK · 2020-11-22T02:38:00Z

captum/optim/_core/loss.py

+
+class TensorDirection(Loss):
+    """
+    Visualize a tensor direction.


nit: Do you mind adding some description here about this loss w.r.t. cosine similarity, etc. ?

I've added a citation and link to the distil paper that the direction objectives come from.

NarineK · 2020-11-22T02:40:05Z

captum/optim/_core/loss.py

+        self.direction = vec.reshape((1, -1, 1, 1))
+
+    def __call__(self, targets_to_values: ModuleOutputMapping) -> torch.Tensor:
+        activations = targets_to_values[self.target]


nit: I think that it would be good to assert the sizes of self.direction and activations before computing cosine similarity on them.

I added an assert for the channel size of self.direction & activations. @greentfrapp may be able to add more assertion tests in his upcoming PR.

NarineK · 2020-11-23T00:02:51Z

captum/optim/_core/loss.py

+        activations = targets_to_values[self.target]
+        x_diff = activations[..., 1:, :] - activations[..., :-1, :]
+        y_diff = activations[..., :, 1:] - activations[..., :, :-1]
+        return torch.sum(torch.abs(x_diff)) + torch.sum(torch.abs(y_diff))


is this L1-norm w.r.t. total variation of height and width ?

It's the sum of the absolute differences for neighboring values in the activations or image. TensorFlow's version (that Lucid uses) links to this Wikipedia page: https://en.wikipedia.org/wiki/Total_variation_denoising

My neural-style-pt project uses basically the same algorithm, but it's origins trace back to this research article: https://arxiv.org/abs/1412.0035

Chris also says that TensorFlow's total variation algorithm comes from the Understanding Deep Image Representations by Inverting Them paper: https://arxiv.org/abs/1412.0035

NarineK · 2020-11-23T01:44:11Z

captum/optim/_core/loss.py

+
+class ActivationInterpolation(Loss):
+    """
+    Interpolate between two different layers & channels


Is there a paper related to this loss. It would be good to document it.

I've added a link and citiation for Activation Interpolation!

NarineK · 2020-11-23T03:08:53Z

captum/optim/_core/loss.py

+                    (activ_a - activ_b) ** 2
+                ).mean() / self.decay_ratio ** float(d)
+
+        return sum_tensor


nit: because we are optimizing negative loss do we want to return sum_tensor ? I'm asking that because the losses are returning the opposite sign here for other losses as well:
https://github.com/greentfrapp/lucent/blob/master/lucent/optvis/objectives.py

Currently the InputOptimization class doesn't support losses like Alignment. So, I've left them as they are for @greentfrapp to change, as he's currently working on the optimization system.

captum/optim/_param/image/images.py

captum/optim/_param/image/transform.py

ProGamerGov · 2020-11-24T17:17:33Z

@NarineK I'll have to think about the ordering of the objectives, but for the moment we can leave them in their current order.

As for the losses, I'll work on writing more in-depth descriptions with @greentfrapp, but in the meantime we can leave the descriptions as they are so that @greentfrapp can continue his work on the optimization stuff.

NarineK

Thank you for working on this PR, @ProGamerGov !

ProGamerGov added 7 commits November 13, 2020 14:55

Remove loss objectives

2b644af

Loss objectives moved to their own file

2fe480c

Fix some class names

fd19f3d

Update tutorials for loss changes

1471942

Add loss path

ed40559

Oops

7c28ba1

Merge pull request #11 from ProGamerGov/loss-class

f9af2bf

Move loss objectives to their own file

facebook-github-bot added the cla signed label Nov 13, 2020

ProGamerGov added a commit to ProGamerGov/captum that referenced this pull request Nov 13, 2020

Prevent Torchvision tutorial conflict with (#527)

e2971cd

pytorch#527

ProGamerGov added 18 commits November 14, 2020 07:55

Add function type hints

df979a1

Add init param to NaturalImage & other changes

dbeb667

Remove cuda call from LaplacianImage

4f54d5a

Non abstract init

abdf2ba

Non abstract __init__ func

e25b61e

Size var fixes

de92824

Fix error

5edd341

Hopefully fix LaplacianImage

6a36378

* https://pytorch.org/docs/stable/generated/torch.nn.ParameterList.html Previously PyTorch had issues accessing the parameters inside LaplacianImage.

Merge pull request #12 from ProGamerGov/init-param

4e8f079

Fix LaplacianImage & add init option to NaturalImage

Linting

8012d7b

Get user int images working

1041392

Update images.py

540d3a1

Add init torch.inverse() func

c99bdd6

Remove old decorrelate init func

864ca58

Fix func call

cbe6545

Re-add H & W names

804609c

Fix decorrelate init

2de31a7

Add type hints

21fa5bf

ProGamerGov referenced this pull request in ProGamerGov/captum Nov 15, 2020

Ensure there are no conflicts with (#527)

f2a445e

Merge pull request #13 from ProGamerGov/init-image

4428fd3

Handle input images

ProGamerGov closed this Nov 20, 2020

ProGamerGov reopened this Nov 20, 2020

ProGamerGov added 17 commits November 19, 2020 18:12

Add missing type hint to batch setup func

4b0df62

Add self to size values

1af18d9

Remove redundant float()

c246313

Bug fix & ToRGB improvements

78319dc

Full batch support for LaplacianImage

66e5742

Properly handle LaplacianImage init batch

759a27b

Type hints & new NaturalImage params

67a948d

* Add missing type hints. * Add decorrelate_init & squash_func parameters to NaturalImage.

Add type hints to init & forward functions

184f21a

Lint fix

de0baeb

More type hints

db5a04e

Add more type hints

159cf9f

Merge conflict fixes

807ea81

Fix mistake

116bce1

Add missing type hint

d2e35f8

Add missing type hints

74b4b01

Add size func to ImageTensor

b3cffe1

Add type hints to __torch_function__ functions

d49757b

* Because why not

ProGamerGov force-pushed the optim-wip branch from 096a138 to d49757b Compare November 23, 2020 18:50

NarineK reviewed Nov 24, 2020

View reviewed changes

ProGamerGov added 2 commits November 24, 2020 10:03

Change things based on feedback

958c8b3

Lint fix

e8eaf0e

Ensure that direction and activation channels are the same

36a67e7

ProGamerGov requested a review from NarineK November 24, 2020 19:38

Fix TotalVariation citation

95dbca3

NarineK approved these changes Nov 25, 2020

View reviewed changes

NarineK merged commit 3cf1675 into pytorch:optim-wip Nov 25, 2020

ProGamerGov mentioned this pull request Apr 6, 2022

Optim-wip: Composable loss improvements #828

Open

Optim wip - Move & restructure loss objectives #527

Optim wip - Move & restructure loss objectives #527

Uh oh!

Conversation

ProGamerGov commented Nov 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ProGamerGov commented Nov 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NarineK left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProGamerGov Nov 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ProGamerGov Nov 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ProGamerGov commented Nov 24, 2020

Uh oh!

NarineK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ProGamerGov commented Nov 13, 2020 •

edited

Loading

ProGamerGov commented Nov 13, 2020 •

edited

Loading

NarineK left a comment •

edited

Loading

ProGamerGov Nov 24, 2020 •

edited

Loading

ProGamerGov Nov 24, 2020 •

edited

Loading