Skip to content

Implement precision and recall metrics for classification evaluation #49671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Dec 19, 2019
Merged

Implement precision and recall metrics for classification evaluation #49671

merged 13 commits into from
Dec 19, 2019

Conversation

@przemekwitek
Copy link
Contributor

@przemekwitek przemekwitek commented Nov 28, 2019 •
edited
Loading

This PR implements precision and recall metrics for classification evaluation.

Additionally, it:

  • registers evaluation metrics in XContent and Writeable registries under qualified names in order to avoid name clashes
  • changes the interface of EvaluationMetric interface by allowing pipeline aggregations to be requested by aggs method
  • moves actualIsTrueQuery method from interface to implementation as this method is implementation-specific

Relates #48759

@cbuescher cbuescher added the :ml Machine learning label Nov 28, 2019
@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 28, 2019

Pinging @elastic/ml-core (:ml)

@przemekwitek przemekwitek force-pushed the precision_and_recall branch 12 times, most recently from 91d83e0 to 3edeb6b Compare December 3, 2019 14:45
@przemekwitek przemekwitek removed the WIP label Dec 3, 2019
@przemekwitek przemekwitek marked this pull request as ready for review December 3, 2019 15:00
@przemekwitek
Copy link
Contributor Author

przemekwitek commented Dec 4, 2019

run elasticsearch-ci/1
run elasticsearch-ci/packaging-sample-matrix

@przemekwitek przemekwitek force-pushed the precision_and_recall branch 4 times, most recently from 2b97424 to 93fc929 Compare December 11, 2019 09:41
@przemekwitek przemekwitek mentioned this pull request Dec 11, 2019
Closed
3 tasks
dimitris-athanasiou
dimitris-athanasiou reviewed Dec 12, 2019
View reviewed changes
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving a few comments. I'll revisit when recall has been adjusted.

dimitris-athanasiou
dimitris-athanasiou reviewed Dec 17, 2019
View reviewed changes
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments as I go along the way.

...main/java/org/elasticsearch/xpack/core/ml/dataframe/evaluation/classification/Precision.java Outdated
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need a size parameter in all these like in the multiclass confusion matrix?

Copy link
Contributor Author

@przemekwitek przemekwitek Dec 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just added size parameter to Precision and Recall metrics.
Also, I've added other_class_count parameter to Precision.Result and Recall.Result so that the user can tell if the result is complete.

Copy link
Contributor Author

@przemekwitek przemekwitek Dec 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just added size parameter to Precision and Recall metrics.
Also, I've added other_class_count parameter to Precision.Result and Recall.Result so that the user can tell if the result is complete.

I reverted this change and implemented max cardinality enforcement as discussed. PTAL

...main/java/org/elasticsearch/xpack/core/ml/dataframe/evaluation/classification/Precision.java Outdated
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be checking this? If for some reason precision is not finite, we'll end up reporting zero instead of NaN or infinity.

Copy link
Contributor Author

@przemekwitek przemekwitek Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will not report zero but rather we will not report this particular class.
My reasoning behind this condition is that if we have an actual class that is never predicted (e.g: "cat"), precision for "cats" cannot be calculated so there is no point in reporting a precision entry with NaN value.

@przemekwitek przemekwitek force-pushed the precision_and_recall branch 2 times, most recently from 6c6406d to 6f254d8 Compare December 18, 2019 13:30
@przemekwitek
Copy link
Contributor Author

przemekwitek commented Dec 19, 2019

run elasticsearch-ci/default-distro
run elasticsearch-ci/bwc

@przemekwitek
Copy link
Contributor Author

przemekwitek commented Dec 19, 2019

run elasticsearch-ci/bwc

@przemekwitek
Implement precision and recall metrics for classification evaluation
97bea88
@przemekwitek
Fix compile errors
1863ef5
@przemekwitek
Remove markup interface classes (SoftClassificationMetric, Classifica…
8365ad3 
…tionMetric, RegressionMetric)
@przemekwitek
Apply review comments
ccf5c30
@przemekwitek
Revert changes related to accuracy
26bb9bf
@przemekwitek
Add Precision and Recall metrics to evaluation verification in Classi…
935a26c 
…ficationIT
@przemekwitek
Add size parameter to Precision and Recall metrics
fc080be
@przemekwitek
Fix MlClientDocumentationIT
8376bb6
@przemekwitek
Relax assertEvaluation method in ClassificationIT
28be815
@przemekwitek
Revert "Add size parameter to Precision and Recall metrics"
db9c4f6 
This reverts commit 81be647d6465e62971ac763605be4a080161cbdf.
@przemekwitek
Enforce max cardinality of actual_class field for precision and recal…
452dfc8 
…l metrics
@przemekwitek
Fix MlClientDocumentationIT
7f571ae
@przemekwitek
Simplify handling actualField in process() method
dd74939
dimitris-athanasiou
dimitris-athanasiou reviewed Dec 19, 2019
View reviewed changes
Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@przemekwitek przemekwitek merged commit 786ead6 into elastic:master Dec 19, 2019
@przemekwitek przemekwitek deleted the precision_and_recall branch December 19, 2019 15:07
@przemekwitek przemekwitek mentioned this pull request Dec 19, 2019
Merged
przemekwitek added a commit that referenced this pull request Dec 19, 2019
@przemekwitek
[7.x] Implement precision and recall metrics for classification e…
cc4bc79 
…valuation (#49671) (#50378)
@przemekwitek przemekwitek mentioned this pull request Dec 20, 2019
Merged
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this pull request Jan 23, 2020
@przemekwitek @SivagurunathanV
Implement precision and recall metrics for classification evaluat…
d1b4000 
…ion (elastic#49671)
This was referenced Feb 3, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dimitris-athanasiou dimitris-athanasiou dimitris-athanasiou left review comments

Assignees

No one assigned

Labels

>feature :ml Machine learning v7.6.0 v8.0.0-alpha1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@przemekwitek @elasticmachine @dimitris-athanasiou @jakelandis @cbuescher