LightGBM should use default seed if not set explicitly

[torronen]

System Information (please complete the following information):

• OS & Version: Windows 11
• ML.NET Version: ML.NT 1.6.0
• .NET Version: .NET 5.0

Describe the bug
LightGBM must get random seed when called, or it will set a set a random seed from MLContext.
However, LightGBM has a default value for seeds. If we set this random_seed, then other seeds will get overridden. ML.NET interface does not provide option to set the other seeds explicitly.

Links:
https://lightgbm.readthedocs.io/en/latest/Parameters.html#seed

machinelearning/src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

Line 230 in b3a3c66

/// If not specified, <see cref="MLContext"/> will generate a random seed to be used.

If some features which use randomity are used then this may cause issues:
-not being able to reproduce results after running Python.
-not able to reproduce results in ML.NET due to MLContex random generator may have had a different number of calls already.

Expected behavior
Seed should not be set for LightGBM.
If not possible, then option to set the specific seeds ( data_random_seed, feature_fraction_seed ...) would be useful.

*Possible fix
Do not set res{"seed"] here if Seed == null
res["seed"] = (Seed.HasValue) ? Seed : host.Rand.Next();

Additional context
Related to this discussion about not being able to reproduce LightGBM results in ML.NET from Python
microsoft/FLAML#409 (comment)

[michaelgsharp]

@torronen since you closed the PR, do you think this is still an issue that we should take a look at?

Also, I wonder if the reason you are seeing the performance differences between this and python is due to the lightGBM version. We are still on version 2 while the latest version is version 3. We are planning on updating the version (it has some breaking changes so its not as easy as just changing the version) so it would be interesting to check again after that.

[torronen]

@michaelgsharp I am not sure, maybe this one is, what do you think? Seed affects the results if the feature_fraction is set. Although would not help with getting similar results as in Python or LightGBM CLI, it might help keep ML.NET results consistent. In PR, changing the other defaults might not make sense because it still does not help with equal results of Python but might affect someone who is relaying on current defaults. For random seed, I think it could possible be set because it should not affect existing users.

I have installed LightGBM 2.3.1. on Python testing. I do not see any major differrence between LightGBM 2.3.1 vs 3 in Python script. LittleLittleCloud helped me get approximately same results in Model Builder. I think I will have a look at the hyperparameters Model builder is giving and comparing with Python, maybe it help find the issue.

[torronen]

@michaelgsharp It is likely about how LightGBM and the pipeline deals with categorization microsoft/FLAML#409 (reply in thread)

If anyone could add some explanation for this LightGBMTrainerBase.cs: GetCategoricalMetaData() and ConstructCategoricalFeatureMetaData that would be interesting, although might not help the issue. I was not able to follow what they are doing.

I see the method gets all categorical columns at the beginning of the method, but at the end it gives only some column numbers as parameters. LightGBM binary expects column numbers, but maybe these method somehow translate the columns to the indices on the data that gets sent to the binary. Not sure if this has anything to do with the issue, though.

machinelearning/src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

Line 494 in b3a3c66

private static List<string> ConstructCategoricalFeatureMetaData(int[] categoricalFeatures, int rawNumCol, ref CategoricalMetaData catMetaData)

machinelearning/src/Microsoft.ML.LightGbm/LightGbmTrainerBase.cs

Line 531 in b3a3c66

private CategoricalMetaData GetCategoricalMetaData(IChannel ch, RoleMappedData trainData, int numRow)

[michaelgsharp]

I will see if I can get you a better explanation of what those things do. Maybe tomorrow if I can, if not next week sometime.

[michaelgsharp]

I decided to put my comment here instead of PR #6068 because I think it will be easier to discuss here where you have all the details posted. So I'll just leave that PR for now while we discuss.

After thinking about this I can think of 2 issues with it that would be good to discuss.

1 - The idea of the MLContext is that since its the central part of ML.NET, that anything you set there gets propagated everywhere unless you specifically overwrite it. Doing your suggestion to remove setting the seed using ML.NET's random state goes against this. With this change, even if you manually set the seed in the MLContext you would also have to manually set the seed in LightGBM. This is something that I think most people wouldn't expect (and probably many would consider a bug), and I can see this causing accidental issues, especially with reproducibility.

2 - The other issues is mostly just with reproducibility. All, or at least many, of the tests that we have using LightGBM would have to be updated to have the seed manually set and then probably all the tests results would need to be updated. This isn't a blocker, obviously, but another issue to call out.

So I think the main issue is with the first point.

You said that you wanted to be able to compare results to those you obtained in python. Could you manually set the seed for Light GBM in ML.NET with the options class , and then set that same seed in python with res["seed"]? You said it will overwrite the other seeds, but if they are all overwritten in both python and ML.NET would you be able to see the same results? Or is there another issue with that?

Also, as we need to update to the latest version of LightGBM which does require some rework due to breaking changes, we could look at exposing the other seeds you were mentioning. Do you think setting all the seeds differently is something that is used? Or at least used enough to make it worth it to expose?

[torronen]

ok, makes sense., I agree. Changing seed in MLContext should take priority, I did not think about that.

Reproducing results between Python and .NET requires setting random_seed and also some other changes to defaults in .NET so this update alone would not be adequate for reproducability. Also, if .NET would have option to import LightGBM .txt-format then users would not need to re-train through ML.NET to use the models from LightGBM CLI or Python. Users can anyway explicitly set random_seed on Python and then put same on .NET side like you said.

The issue left is that if we want to tune feature_fraction, then we must set seed (or create new MLContext for every iteration), or the changing seed may affect more than value of feature_fraction. Although, this issue might happen only in datasets with high column count. For this purpose I am now "tuning" the seeds as part of the tuning process. I suppose tuning random_seed should be enough, but maybe, for the absolute optimum we might want to optimize data_random_seed and feature_fraction_seed separately. I am not sure if it enough reason to expose the seeds, but even if needed it it fairly easy to do as user can just copy&paste res["seed"] and which ever seeds they need.

[torronen]

This is probably not very important issue, but two options. How about these:

1. Set seed of MLContext as seed for LightGBM. E.g. new MLContext(123) => LightGBM.Seed would be by default 123, not random.Next()

2. Set seed from random.Next only once per LightGBM trainer to a static shared variable. I think the current implementation sets a different seed for each time training is ran. It might solve the above issues if it would be only set, let's say, once in a static class constructor?

I think exposing the more specific seeds might not hurt in the updated version of LightGBM, especially if IntelliSense comment instructs that they will be generated based on Seed, if not set. It might also help follow LightGBM's own documentation. Considering LightGBM's own documentation, maybe, next version could consider renaming the parameters to match. On other hand, might be an unnecessary breaking change.

[torronen]

FastTree might use random seed from MLContext, but I did not yet follow this further, just a note.

machinelearning/src/Microsoft.ML.FastTree/FastTreeArguments.cs

Line 373 in 6d0a9f3

public int Seed = 123;

[michaelgsharp]

@torronen Those are good thoughts, we are going to be having an internal meeting to discuss some of them.

Are you able to build from source to test things? If you are, we could have it manually load your text file instead of the mL.NET trained model. Let me know if you are interested in trying that.

[michaelgsharp]

So doing number 1 would be hard as LightGBM is included as an extra nuget package. So we can't just assume its going to be there to set directly with the MLContext.

I can see the benefits for number 2, but only if multiple models were done in the same MLContext. This seems to be more of an edge case than the standard case though.

I guess though I'm curious about your overall workflow with this. Are you testing multiple light gbm trainers with a single context? If so, what is your overall flow?