We currently have model evaluators that produce metrics on the predicted label. For practical use of machine learning, it is necessary to have a sense for any biases the model may propagate and any fairness issues the mode has. In this way, it would be great to have a standard evaluators for bias and fairness metrics.
Related to #511