Support for new metrics under tf.metric

[SSaishruthi]

Describe the feature and the current behavior/state.

Reference issue: tensorflow/tensorflow#28972 (comment)
Reviewer asked me to create a PR in addons.

Support for following metrics

• Cohen's Kappa
• R-Square
• F1 (micro/macro/weighted)
• F-Beta
• Multilabel confusion matrix.
• Hamming score
• Jaccard score
• Positive likelihood
• Negative likelihood
• Multiclass AUC
• Adjusted R-Square

[facaiy]

Thank you, SSaishruthi.

Yeah, both @AakashKumarNain and @shashvatshahi1998 have made a request for F1 score #232 . It looks necessary for our community.

related issue:

• F1_score with multilabel classfiers(micro,macro) should be added #232
• Multiclass AUC #209
• Add new metrics for tf.keras #126

[SSaishruthi]

Thanks @facaiy
I will put a hold on F1-Beta to see if anyone is already working on that.
I will start working on other metrics. I will be working on F1-micro/macro along with others mentioned so that it does not block others work.

[facaiy]

Great! Feel free to reach us if any help is needed.

[seanpmorgan]

Thanks for creating this! Just a note we do already have Focal Loss (also not really a metric... could we move the losses to a separate issue?):
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/losses/focal_loss.py

[SSaishruthi]

@seanpmorgan Sure.
I have a question before starting. Should this be created under tf.keras or tf.metric?
I was thinking about adding under tf.metric. Some of the linked current works are under tf.keras though.
@facaiy

[WindQAQ]

Hi, @SSaishruthi. Welcome to addons community!

I have a question before starting. Should this be created under tf.keras or tf.metric?
I was thinking about adding under tf.metric. Some of the linked current works are under tf.keras though.

Do you mean if we should inherit from either tf.metrics.Metric or tf.keras.metrics.Metric? If so, they are just the alias in TF2.0. But for code style unification, let's simply follow #267.

https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/metrics

[SSaishruthi]

Thank you @WindQAQ

[SSaishruthi]

@WindQAQ @facaiy @seanpmorgan
I have R-square first version ready
https://colab.research.google.com/drive/1qteFJCiJlN3ougAoZM-QtnrZeeHlXHB3

Also, I have F1 - Micro/Macro/ weighted first version too if you are fine with that.

[WindQAQ]

@SSaishruthi, well done! As the same question in #267 (review), is there any reason that we could only compute the metric on a single batch instead of accumulating states and computing overall R-square?

[SSaishruthi]

Hi @WindQAQ
That's a great point. I was looking at that as well.

1. One option for user is to use Callback
   https://keras.io/callbacks/#create-a-callback?

2. We need to save values in a list as we progress and do the calculation at the end.

[WindQAQ]

Hi @SSaishruthi, I have to clarify my example:

y_true = np.array([0, 1, 0, 1, 0])
y_pred = np.array([0, 1, 0, 1, 1])
m = tf.keras.metrics.BinaryAccuracy()
m.update_state(y_true, y_pred) # 1st acc: 0.8
print(m.result().numpy()) # 0.8

y_true = np.array([0, 1, 0, 1])
y_pred = np.array([0, 1, 0, 1])
m.update_state(y_true, y_pred) # 2nd acc: 1.0
print(m.result().numpy()) # 0.9 = (0.8 + 1) / 2

m.reset_states() # reset if needed

Metrics inherited from tf.keras.metrics.Mean or MeanMetricWrapper take the average over batches instead of over samples. So the the second result above is computed by (0.8 + 1) / 2 rather than (4 + 4) / (5 + 4). In this scenario, I think it is much easier to accumulate states (maintain two variables total and count and result is equal to total / count, or simply inherit from tf.keras.metrics.Mean).

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/metrics.py#L525

@facaiy @seanpmorgan, hi Facai and Sean, what do you feel about this? Is it acceptable to inherit from tf.keras.metrics.Mean?

[facaiy]

@WindQAQ Which metric to inherit from Mean, Tzu-Wei?

[WindQAQ]

Not so sure about that... Some metrics in tf.keras.metrics like BinaryAccuracy average over each batches' measurement, while others like Precision compute the metric per sample and return an overall number. You may understand what I say in this notebook. So if metrics follow BinaryAccuracy's pattern, one could easily implement it when inheriting from Mean. But I am quite confused if metrics should follow this pattern or not...