[Dev only] MOO ensemble can fail to return pareto front if automl instance consutrcted with single metric

[eddiebergman]

If constructing the automl instance with only a single metric but then using multiple metrics during ensemble, querying for the pareto front will fail with:

File "/home/skantify/code/automlbenchmark-fairness-private/frameworks/autosklearn_moo/lib/auto-sklearn/autosklearn/automl.py", line 1656, in _load_pareto_set
    raise ValueError("Pareto set is only available for two or more metrics.")

[eddiebergman]

I guess this is an odd case and doesn't seem easy to fix. fit_ensemble() doesn't take different metrics and the only source of metrics is the init. I'll close but if this is an issue in the future, I guess we can re-open

[mfeurer]

Couldn't we fix this case by checking if the loaded ensemble is a multi-objective ensemble instead of checking if we have multiple metrics?

[eddiebergman]

That's an easy fix for sure, but the issue also lies in the fact that the point of entry for metrics is at __init__ time.

If you wish to have single objective SMAC runs, you pass in a single objective. Any Ensembler is then lmited to this one metric and it makes no sense to have a multiobjective one. I guess the easiest workaround is allow fit_ensemble to take metrics?

How does this sound?

# Optimize with respect to one metric
automl = AutoSklearnCalssifier(metric=acc, ensemble=None)
automl.fit(X, y)

# Fit ensemble with respect to many
# Need to add this `metrics` arg to API
automl.fit_ensemble(X, y, ensemble=MooEnsemble, metrics=[acc, auc])

[mfeurer]

That sounds good. I somehow assumed fit_ensemble accepts metric, but apparently, it doesn't.