Documentation request: test data + model order

[torronen]

Is your feature request related to a problem? Please describe.
I am concerned about how to evaluate the performance of the models as the documentation does not clearly explain how the test data is split. I also would like to estimate better how long to run iterations if I have already experienced with a related data set.

Describe the solution you'd like
I suggest adding to documentation:

• how is test data split. E.g. is it from the end, randomly or based on auto-selection of columns?
• how are the iterations generated. It would seem they might be in fixed order, or my data is fairly similar and leads to it. For example, if iteration 201 performed best, then when I re-train I could stop after iteration 201 next time, if I know this. If they are not in fixed order, then I know to let my next dataset run a bit longer.

[LittleLittleCloud]

@torronen thanks for the feedback

how is test data split. E.g. is it from the end, randomly or based on auto-selection of columns?

Depends on the size of your dataset. either 5-fold cv or hold-out (10% for test), all randomly. Considering the size of dataset you have, it will be hold-out

how are the iterations generated.

It's based on the performance of each trainer. If one trainer has a higher improving speed, it will be assigned with more time budget. The improving speed is evaluated via ECI, you can refer to the following paper for detailed info

https://export.arxiv.org/pdf/1911.04706

@briacht @luisquintanilla Maybe we can add those to the doc in the next release?

[briacht]

@luisquintanilla Can we please add a work item to Docs to talk about how Model Builder splits up the data?

Thanks for the suggestion @torronen !

[luisquintanilla]

Thanks for this feedback @torronen . I've added an issue to our backlog dotnet/docs#25091 to track this work.

[briacht]

This is being tracked in Docs now so I will close this issue here.