Add kappa

[AakashKumarNain]

Added Cohens Kappa as a new metric.

[WindQAQ]

Hi @AakashKumarNain, thanks for the contribution! I wonder if I misunderstand something here:

Shouldn't update_state aggregate the previous statistics like what other metrics in tf.keras.metrics do? For example EDIT:

y_true = np.array([0, 1, 0, 1, 0])
y_pred = np.array([0, 1, 0, 1, 1])
m = tf.keras.metrics.BinaryAccuracy()
m.update_state(y_true, y_pred) # 1st acc: 0.8
print(m.result().numpy()) # 0.8

y_true = np.array([0, 1, 0, 1, 0])
y_pred = np.array([0, 1, 0, 1, 0])
m.update_state(y_true, y_pred) # 2nd acc: 1.0
print(m.result().numpy()) # 0.9 = (0.8 + 1) / 2

m.reset_states() # reset if needed

Seems that the implementation now only takes current input into account. Is there any reason that we have to only compute the current input's Cohen's Kappa instead of accumulating states and computing overall Cohen's Kappa? Please correct me if I misunderstand. cc @facaiy.

EDIT: #265 (comment)

[AakashKumarNain]

Thanks @WindQAQ . I am aware that the state needs to be accumulated but couldn't figure out how to do that. @facaiy can you elaborate a bit on how to achieve the same?

[facaiy]

@AakashKumarNain I think tf.kears.metric.AUC is a good example. We should save/update intermediate variables (say, confusion matrix), rather than the result (kappa score).

[facaiy]

Or, please refer to https://github.com/tensorflow/tensorflow/blob/25c197e02393bd44f50079945409009dd4d434f8/tensorflow/contrib/metrics/python/ops/metric_ops.py#L3824

[AakashKumarNain]

Thank you @facaiy for the information.

@WindQAQ @facaiy I think I have found a very elegant solution to the problem. I cannot think of anything better than this. Please take a look at this notebook. I will make the changes in PR once you are okay with the solution, though the errors from graph mode are up again in test cases.

https://colab.research.google.com/drive/10CNyrnq10RUTHssUcfSdtIyYdGnC5bGD

[facaiy]

@AakashKumarNain Aakash, the new solution looks good :-)

[AakashKumarNain]

@facaiy Thanks Yan. Can you please help me with the errors in the test case? I have shown it in the same notebook

[facaiy]

Could you remove self.kappa_score variable, Aakash? It seems that we don't need it any more.

[AakashKumarNain]

It didn't help. Now the error is raised due to theconf_mtx variable that is dependent on the self.conf_mtx variable defined in the constructor.

[facaiy]

I'm not so sure. Seems that we have to initialize variables by ourselves or use metric in a keras model, please refer to metric's test cases :-)

https://github.com/tensorflow/tensorflow/blob/r2.0/tensorflow/python/keras/metrics_test.py

[AakashKumarNain]

I checked that and I am doing the same thing. This line self.conf_mtx.assign_add(new_conf_mtx) is causing the problem. The AssignAdd operation is throwing errors.

[WindQAQ]

Hi, @AakashKumarNain, you have to initialize variables within the metric object:

self.evaluate(tf.compat.v1.variables_initializer(kp_obj1.variables))

Here is the revised notebook.

[AakashKumarNain]

Thank you @WindQAQ for looking into it. Can you provide me the access to the notebook?

[WindQAQ]

Sorry about that... Already shared it. Please click the link above again.

[AakashKumarNain]

Got it. Thanks.

[Squadrick]

@AakashKumarNain I'm trying to figure that out. Could you tell your TF 2.x version using: tf.__version__

[AakashKumarNain]

@facaiy @WindQAQ @Squadrick I am on 2.0.0-beta0 . The funny thing is that now it is failing on tf.initializers, which shouldn't be the case anyway.

Another funny thing is that everything works with Py3, all test cases pass but something is always off with Py2. Py2 support is going to end in December, maybe it is the sign 😆

Either the version in Py2 is very different than it is with Py3 or something is seriously broken

[seanpmorgan]

@facaiy @WindQAQ @Squadrick I am on 2.0.0-beta0 . The funny thing is that now it is failing on tf.initializers, which shouldn't be the case anyway.

Another funny thing is that everything works with Py3, all test cases pass but something is always off with Py2. Py2 support is going to end in December, maybe it is the sign

Either the version in Py2 is very different than it is with Py3 or something is seriously broken

Sorry I haven't been tracking this too closely. If you're refering to why py3 tests are passing in our CI it's probably because there is no py34 tf2-nightly available so it's running on an old version. See #279

[AakashKumarNain]

But @seanpmorgan no matter what version, tf.initializers should be there. It is like tf.nn, an important part of the API, right?

[seanpmorgan]

See #278 #273 I asked but didn't look into when this API enforcement occurred. (So tf.keras.initalizers should work). The py34 test is passing on an outdated nightly I believe

[seanpmorgan]

I checked on py36 and there is no alias for tf.initializers anymore

[Squadrick]

Can confirm, tf.initializers is not found in the latest TF2.x nightly for Python3.6, but tf.keras.initializers works.

[AakashKumarNain]

@Squadrick I fixed it. Can you trigger the test again, please?

Outdated review.

[AakashKumarNain]

Thanks @Squadrick Finally! This has taken an enormous amount of time. Thanks @facaiy @WindQAQ @seanpmorgan for all the guidance and efforts.

PS: Is there a list where we are keeping track of all the removed APIs?

[Squadrick]

@AakashKumarNain Thanks again for the contribution.

[seanpmorgan]

Thanks @Squadrick Finally! This has taken an enormous amount of time. Thanks @facaiy @WindQAQ @seanpmorgan for all the guidance and efforts.

PS: Is there a list where we are keeping track of all the removed APIs?

Not that I know of, but may be worth asking. The best I would look at is https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/compatibility/all_renames_v2.py

But that mostly uses compat.v1 as replacement. And thanks very much to you for putting in the time to get it all working! It sets a great example for stateful metrics in Addons!

[llu025]

Hello,

I am using tfa.metrics.CohenKappa and something seems to be really wrong. While tf.keras.metrics.Accuracy is working fine, the CohenKappa gets values above 1 (sometimes above 2..).

I have a dictionary of metrics:

self.change_map_metrics = {
            "ACC": tf.keras.metrics.Accuracy(),
            "cohens kappa": tfa.metrics.CohenKappa(num_classes=2),
        }

and then after each training epoch I call this:

def _compute_metrics(self, y_true, y_pred, metrics):
        """
            Compute the metrics specified in metrics.
            Write results to self.tb_writer
            Input:
                y_true - tensor (n, )
                y_pred - tensor (n, )
                metrics - dict {name: tf.metrics class instance}
            Output:
                None
        """
        y_true, y_pred = tf.reshape(y_true, [-1]), tf.reshape(y_pred, [-1])
        for name, metric in metrics.items():
            metric.update_state(y_true, y_pred)
            with self.tb_writer.as_default():
                tf.summary.scalar(name, metric.result())
            metric.reset_states()

As you can see on Tensorboard, it does not really make any sense.

I am using TF 2.0 and the docker image tensorflow/tensorflow:2.0.0-gpu.
Please let me know if you need further details.

[WindQAQ]

@llu025 Hi, is it possible to dump y_true and y_pred in your program? Thank you.

[llu025]

@llu025 Hi, is it possible to dump y_true and y_pred in your program? Thank you.

I am not really sure what you mean with "dump". If you are interested in what they contain, they are boolean, so either 0s or 1s.

[WindQAQ]

@llu025 Hi, is it possible to dump y_true and y_pred in your program? Thank you.

I am not really sure what you mean with "dump". If you are interested in what they contain, they are boolean, so either 0s or 1s.

I mean the exact values of them, which could make kappa score > 1.

[AakashKumarNain]

Yeah, it would be really helpful to debug if you provide the values that produced this score

[llu025]

I printed 20 consecutive values from a random index for both y_pred and y_true, this is an example of one epoch:

I thought the problem might have been the use of boolean labels instead of integers, so I added this in my _compute_metrics function:

if y_pred.dtype == tf.bool:
            y_true, y_pred = tf.cast(y_true, tf.uint8), tf.cast(y_pred, tf.uint8)

so I dump them again

but I still get kappa above 1

[WindQAQ]

Thank you!
@AakashKumarNain I find some abnormal test case that could make it > 1:
https://colab.research.google.com/drive/1lA0mc0prr0XB4buKOdf44WvKi_xMXtIt
I guess it's due to overflow of int32.

[AakashKumarNain]

@WindQAQ Thanks for the test cases. Yeah, it seems that we need to change the dtype to int64

[WindQAQ]

This should be fix in #675 :D

[AakashKumarNain]

This should be fix in #675 :D

Thank you @WindQAQ . I checked the same on my end. Seems that dtype was the issue.
https://colab.research.google.com/drive/1vPGX11biU3CepL3wHp0tZocihu-d9fgK