Activation api style

[fsx950223]

Describe the feature and the current behavior/state.
Let the activation api style follows tensorflow.nn api style.

Advantage:

1. Debugging friendly(name support)
2. More tensor types support

Relevant information

• Are you willing to contribute it (yes/no): yes
• Are you willing to maintain it going forward? (yes/no): yes
• Is there a relevant academic paper? (if so, where):
• Is there already an implementation in another framework? (if so, where): tf.nn.leaky_relu
• Was it part of tf.contrib? (if so, where):

Which API type would this fall under (layer, metric, optimizer, etc.)
activation
Who will benefit with this feature?
developers
Any other info.
Reference #493

[WindQAQ]

cc @seanpmorgan and @facaiy.

[seanpmorgan]

Hi @fsx950223 thanks for bringing this up. So as mentioned we've had this discussion in #426.

I'm just going to summarize my understanding and let me know if I'm missing anything

• tf.nn is a grab bag for many ops -- including many that are not activations.
• tfa.activations does not have the best constrained API. However, they are all used for the same general purpose (~non-linear activation functions)
• tfa.keras.layers uses tf.nn functions (in examples) when specifing activation functions that do not have a string alias

Is there any "API style" that we would like to change or simply make it so our activations exist in the same subpackage as they would inside TF core?

IMO it's a better user experience for all activation to co-exist without all of the other things that may fall under nn. I think PyTorch uses a nice middle ground here with a single module:
https://github.com/pytorch/pytorch/blob/10c4b98ade8349d841518d22f19a653a939e260c/torch/nn/modules/activation.py

My personal vote is to keep the activations subpackage. But I'm happy to be out voted here if others think it's more important to match the subpackage name from TF.

[WindQAQ]

@seanpmorgan Sean, I think what @fsx950223 means is to

1. Add name argument: def hardtanh(x, name=None) like what tf.nn.leaky_relu does.
   https://github.com/tensorflow/tensorflow/blob/f7fd9b4fcebdd5b6b36f5604831b9b2a14de65ca/tensorflow/python/ops/nn_ops.py#L2801
2. Support integer type. This is also done in tf.nn.leaky_relu.
   https://github.com/tensorflow/tensorflow/blob/f7fd9b4fcebdd5b6b36f5604831b9b2a14de65ca/tensorflow/python/ops/nn_ops.py#L2818-L2819

I'm fine with the first one. Indeed, we documented this in our README.md but I literally forgot about it in my previous two PRs for gelu and tanhshrink. Apologize for my mistake.

As for the second one, I do not think we really need to support integer type. I suppose tf.nn.leaky_relu does it because of its linearity, while tf.nn.elu and tf.nn.selu do not support integer type. I think that the better choice is to let users cast the integer to float explicitly. Note that this is not only my opinion, but also tf.nn.elu, tf.nn.selu and PyTorch's solutions.

Thanks.

[seanpmorgan]

@seanpmorgan Sean, I think what @fsx950223 means is to

1. Add name argument: def hardtanh(x, name=None) like what tf.nn.leaky_relu does.
   https://github.com/tensorflow/tensorflow/blob/f7fd9b4fcebdd5b6b36f5604831b9b2a14de65ca/tensorflow/python/ops/nn_ops.py#L2801
2. Support integer type. This is also done in tf.nn.leaky_relu.
   https://github.com/tensorflow/tensorflow/blob/f7fd9b4fcebdd5b6b36f5604831b9b2a14de65ca/tensorflow/python/ops/nn_ops.py#L2818-L2819

I'm fine with the first one. Indeed, we documented this in our README.md but I literally forgot about it in my previous two PRs for gelu and tanhshrink. Apologize for my mistake.

As for the second one, I do not think we really need to support integer type. I suppose tf.nn.leaky_relu does it because of its linearity, while tf.nn.elu and tf.nn.selu do not support integer type. I think that the better choice is to let users cast the integer to float explicitly. Note that this is not only my opinion, but also tf.nn.elu, tf.nn.selu and PyTorch's solutions.

Thanks.

Ah understood, I apologize I didn't go through the referenced PR.

1. Seems like the right decision to add a name parameter and match the other activations.
2. I would vote that we don't cast dtypes without the end user knowing (If we do -- we should at least pass a warning)

[facaiy]

Thanks for bringing this into our attention :-)

1. Add name argument

I have no objection, however I prefer to keep consistency with tf.keras.activations, and we can add it later. If I'm not wrong, activations are small ops in most cases. name attribute make it hard to maintain argument backward compatibility, take transpose for example, note how weird name and conjugate are:

tf.transpose(
    a,
    perm=None,
    name='transpose',
    conjugate=False
)

2. Support integer type.

My first thought is not all activations should be forced to support integer type.

the better choice is to let users cast the integer to float explicitly.

+1, sounds good to me.

[fsx950223]

Do we need name scope?

[WindQAQ]

To conclude

• Activations are not necessary to have name argument
• Integer type supports might be needed when activations are linear (like relu)

Feel free to re-open the issue if there is anything unclear. Thanks!