Thresholded Linear Unit

[AakashKumarNain]

This activation unit is introduced in Filter Response Normalization

[Squadrick]

@AakashKumarNain Any reason to put this under layers as opposed to activations? Also, could you update the README.md?

[AakashKumarNain]

@Squadrick we have two learnable parameters for this activation, similar to PReLU. It makes more sense to build a layer for this and if we want to include it in the activations package, then the activation function should initalize an instance of this layer first and then call the activation function on the inputs. Lemme know what you think

[Squadrick]

@AakashKumarNain Yeah, you're right. I'd forgotten that tf.keras expects activations to be stateless. Let's not bother with creating anything under activations. There's a precedent with core tf.keras making PReLU a layer.

[Squadrick]

@AakashKumarNain Also test for serialization.

[AakashKumarNain]

@AakashKumarNain Also test for serialization.

Can you elaborate on this? Any example?

[Squadrick]

@AakashKumarNain Also test for serialization.

Can you elaborate on this? Any example?

addons/tensorflow_addons/layers/wrappers_test.py

Line 99 in 2bcf8c8

def test_serialization(self, base_layer, rnn):

[AakashKumarNain]

Will this suffice?

def test_serialization(self):
    tlu = TLU(affine=True, alpha_initializer='ones', tau_initializer='ones')
    serialized_tlu = tf.keras.layers.serialize(tlu)
    new_layer = tf.keras.layers.deserialize(serialized_tlu, custom_objects={'TLU':TLU})
    self.assertEqual(tlu.get_config(), new_layer.get_config())

[Squadrick]

@AakashKumarNain Yeah, that works. It'll work without the custom_objects as well.

[AakashKumarNain]

@seanpmorgan @Squadrick I think we can merge it now

[Squadrick]

LGTM

[AakashKumarNain]

Thanks

[AakashKumarNain]

I forgot to update the README. Will do it later.