diff --git a/samples/csharp/getting-started/Ranking_Web/README.md b/samples/csharp/getting-started/Ranking_Web/README.md new file mode 100644 index 000000000..d77128962 --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/README.md @@ -0,0 +1,155 @@ +# Rank search engine results + +| ML.NET version | API type | Status | App Type | Data type | Scenario | ML Task | Algorithms | +|----------------|-------------------|-------------------------------|-------------|-----------|---------------------|---------------------------|-----------------------------| +| v1.1.0 | Dynamic API | Up-to-date | Console app | .csv file | Ranking search engine results | Ranking | LightGBM | + +This introductory sample shows how to use ML.NET to predict the the best order to display search engine results. In the world of machine learning, this type of prediction is known as ranking. + +## Problem +The ability to perform ranking is a common problem faced by search engines since users expect query results to be ranked\sorted according to their relevance. This problem extends beyond the needs of search engines to include a variety of business scenarios where personalized sorting is key to the user experience. Here are a few specific examples: +* Travel Agency - Provide a list of hotels with those that are most likely to be purchased\booked by the user positioned highest in the list. +* Shopping - Display items from a product catalog in an order that aligns with a user's shopping preferences. +* Recruiting - Retrieve job applications ranked according to the candidates that are most qualified for a new job opening. + +Ranking is useful to any scenario where it is important to list items in an order that increases the likelihood of a click, purchase, reservation, etc. + +In this sample, we show how to apply ranking to search engine results. To perform ranking, there are two algorithms currently available - FastTree Boosting (FastRank) and Light Gradient Boosting Machine (LightGBM). We use the LightGBM's LambdaRank implementation in this sample to automatically build an ML model to predict ranking. + +## Dataset +The training and testing data used by this sample is based on a public [dataset provided by Microsoft](https://www.microsoft.com/en-us/research/project/mslr/) originally provided Microsoft Bing. The dataset is released under a [CC-by 4.0](https://creativecommons.org/licenses/by/4.0/) license. + +@article{DBLP:journals/corr/QinL13, + author = {Tao Qin and + Tie{-}Yan Liu}, + title = {Introducing {LETOR} 4.0 Datasets}, + journal = {CoRR}, + volume = {abs/1306.2597}, + year = {2013}, + url = {https://arxiv.org/abs/1306.2597}, + timestamp = {Mon, 01 Jul 2013 20:31:25 +0200}, + biburl = {https://dblp.uni-trier.de/rec/bib/journals/corr/QinL13}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} + +The following description is provided for this dataset: + + The datasets are machine learning data, in which queries and urls are represented by IDs. The datasets consist of feature vectors extracted from query-url pairs along with relevance judgment labels: + + * The relevance judgments are obtained from a retired labeling set of a commercial web search engine (Microsoft Bing), which take 5 values from 0 (irrelevant) to 4 (perfectly relevant). + + * The features are basically extracted by us (e.g. Microsoft), and are those widely used in the research community. + + In the data files, each row corresponds to a query-url pair. The first column is relevance label of the pair, the second column is query id, and the following columns are features. The larger value the relevance label has, the more relevant the query-url pair is. A query-url pair is represented by a 136-dimensional feature vector. + +## ML Task - Ranking +As previously mentioned, this sample uses the LightGBM LambdaRank algorithm which is applied using a supervised learning technique known as [**Learning to Rank**](https://en.wikipedia.org/wiki/Learning_to_rank). This technique requires that train\test datasets contain groups of data instances that are each labeled with their relevance score (e.g. relevance judgment label). The label is a numerical\ordinal value, such as {0, 1, 2, 3, 4}. The process for labeling these data instances with their relevance scores can be done manually by subject matter experts. Or, the labels can be determined using other metrics, such as the number of clicks on a given search result. + +It is expected that the dataset will have many more "Bad" relevance scores than "Perfect". This helps to avoid converting a ranked list directly into equally sized bins of {0, 1, 2, 3, 4}. The relevance scores are also reused so that you will have many items **per group** that are labeled 0, which means the result is "Bad". And, only one or a few labeled 4, which means that the result is "Perfect". + +Once the train\test datasets are labeled with relevance scores, the model (e.g. ranker) can then be trained and tested using this data. Through the model training process, the ranker learns how to score each data instance within a group based on their label value. The resulting score of an individual data instance by itself isn't important -- instead, the scores should be compared against one another to determine the relative ordering of a group's data instances. The higher the score a data instance has, the more relevant and more highly ranked it is within its group. + +## Solution +Since this sample's dataset already is already labeled with relevance scores, we can immediately start with training the model. In cases where you start with a dataset that isn't labeled, you will need to go through this process first by having subject matter experts provide relevance scores or by using some other metrics to determine relevance. + +This sample performs the following high-level steps to rank the search engine results: +1. The model is **trained** using the train dataset with LightGBM LambdaRank. +2. The model is **tested** using the test dataset. This results in a **prediction** that includes a **score** for each search engine result. The score is used to determine the ranking relative to other results within the same query (e.g. group). The predictions are then **evaluated** by examining metrics; specifically the [Normalized Discounted Cumulative Gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain)(NDCG). +3. The final step is to **consume** the model to perform ranking predictions for new incoming searches. + +### 1. Train Model +This sample trains the model using the LightGbmRankingTrainer which relies on LightGBM LambdaRank. The model requires the following input columns: + +* Group Id - Column that contains the group id for each data instance. Data instances are contained in logical groupings representing all candidate results in a single query and each group has an identifier known as the group id. In the case of the search engine dataset, search results are grouped by their corresponding query where the group id corresponds to the query id. The input group id data type must be [key type](https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.data.keydataviewtype). +* Label - Column that contains the relevance label of each data instance where higher values indicate higher relevance. The input label data type must be [key type](https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.data.keydataviewtype) or [Single](https://docs.microsoft.com/en-us/dotnet/api/system.single). +* Features - The columns that are influential in determining the relevance\rank of a data instance. The input feature data must be a fixed size vector of type [Single](https://docs.microsoft.com/en-us/dotnet/api/system.single). + +When the trainer is set, **custom gains** (or relevance gains) can also be used to apply weights to each of the labeled relevance scores. This helps to ensure that the model places more emphasis on ranking results higher that have a higher weight. For the purposes of this sample, we use the default provided weights. + +The following code is used to train the model: + +```CSharp +const string FeaturesVectorName = "Features"; + +// Load the training dataset. +IDataView trainData = mlContext.Data.LoadFromTextFile(trainDatasetPath, separatorChar: '\t', hasHeader: true); + +// Specify the columns to include in the feature input data. +var featureCols = trainData.Schema.AsQueryable() + .Select(s => s.Name) + .Where(c => + c != nameof(SearchResultData.Label) && + c != nameof(SearchResultData.GroupId)) + .ToArray(); + +// Create an Estimator and transform the data: +// 1. Concatenate the feature columns into a single Features vector. +// 2. Create a key type for the label input data by using the value to key transform. +// 3. Create a key type for the group input data by using a hash transform. +IEstimator dataPipeline = mlContext.Transforms.Concatenate(FeaturesVectorName, featureCols) + .Append(mlContext.Transforms.Conversion.MapValueToKey(nameof(SearchResultData.Label))) + .Append(mlContext.Transforms.Conversion.Hash(nameof(SearchResultData.GroupId), nameof(SearchResultData.GroupId), numberOfBits: 20)); + +// Set the LightGBM LambdaRank trainer. +IEstimator trainer = mlContext.Ranking.Trainers.LightGbm(labelColumnName: nameof(SearchResultData.Label), featureColumnName: FeaturesVectorName, rowGroupColumnName: nameof(SearchResultData.GroupId)); ; +IEstimator trainerPipeline = dataPipeline.Append(trainer); + +// Training the model is a process of running the chosen algorithm on the given data. To perform training you need to call the Fit() method. +ITransformer model = trainerPipeline.Fit(trainData); +IDataView transformedTrainData = model.Transform(trainData); + +// Save the model +mlContext.Model.Save(model, null, modelPath); +````` + +### 2. Test and Evaluate Model +We need this step to determine how effective our model is at ranking. To do so, the model from the previous step is run against another dataset that was not used in training (e.g. the test dataset). + +`Evaluate()` compares the predicted values for the test dataset and produces various metrics you can explore. Specifically, we can gauge the quality of our model using Discounted Cumulative Gain (DCG) and Normalized Discounted Cumulative Gain (NDCG) which are included in the `RankingMetrics` returned by `Evaluate()`. + +When evaluating the `RankingMetrics` for this sample's model, you'll notice that the following metrics are reported for DCG and NDCG (the values that you see when running the sample will be similar to these): +* DCG - @1:11.9058, @2:17.4132, @3:21.2908, @4:24.5243, @5:27.3235, @6:29.6794, @7:31.9928, @8:34.0955, @9:36.0850, @10:37.9679 + +* NDCG - @1:0.5012, @2:0.4945, @3:0.4986, @4:0.5055, @5:0.5131, @6:0.5182, @7:0.5251, @8:0.5308, @9:0.5365, @10:0.5417 + +The NDCG values are most useful to examine since this allows us to compare our model's ranking ability across different datasets. The potential value of NDCG ranges from **0.0** to **1.0**, with 1.0 being a perfect model that exactly matches the ideal ranking. + +With this in mind, let's look at our model's values for NDCG. In particular, let's look at the value for **NDCG@10** which is **0.5417**. This is the average NDCG for a query returning the top **10** search engine results. To increase the model's ranking ability, we would need to experiment with feature engineering and model hyperparameters to continue to improve our model. + +Refer to the following code used to test and evaluate the model: + +```CSharp +// Load the test data and use the model to perform predictions on the test data. +IDataView testData = mlContext.Data.LoadFromTextFile(testDatasetPath, separatorChar: '\t', hasHeader: false); +IDataView predictions = model.Transform(testData); + +[...] + +// Evaluate the metrics for the data using NDCG; by default, metrics for the up to 3 search results in the query are reported (e.g. NDCG@3). +RankingMetrics metrics = mlContext.Ranking.Evaluate(predictions); +````` + +### 3. Consume Model + +After the model is built and trained, we can use the `Predict()` API to predict the ranking of search engine results for a user query. + +```CSharp +// Load test data and use the model to perform predictions on it. +IDataView testData = mlContext.Data.LoadFromTextFile(testDatasetPath, separatorChar: '\t', hasHeader: false); + +// Load the model. +DataViewSchema predictionPipelineSchema; +ITransformer predictionPipeline = mlContext.Model.Load(modelPath, out predictionPipelineSchema); + +// Predict rankings. +IDataView predictions = predictionPipeline.Transform(testData); + +// In the predictions, get the scores of the search results included in the first query (e.g. group). +IEnumerable searchQueries = mlContext.Data.CreateEnumerable(predictions, reuseRowObject: false); +var firstGroupId = searchQueries.First().GroupId; +IEnumerable firstGroupPredictions = searchQueries.Take(100).Where(p => p.GroupId == firstGroupId).OrderByDescending(p => p.Score).ToList(); + +// The individual scores themselves are NOT a useful measure of result quality; instead, they are only useful as a relative measure to other scores in the group. +// The scores are used to determine the ranking where a higher score indicates a higher ranking versus another candidate result. +ConsoleHelper.PrintScores(firstGroupPredictions); +````` diff --git a/samples/csharp/getting-started/Ranking_Web/WebRanking.sln b/samples/csharp/getting-started/Ranking_Web/WebRanking.sln new file mode 100644 index 000000000..367beec77 --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/WebRanking.sln @@ -0,0 +1,25 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio 15 +VisualStudioVersion = 15.0.28307.705 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "WebRanking", "WebRanking\WebRanking.csproj", "{D502394E-930B-401A-812F-2A996751B80A}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {D502394E-930B-401A-812F-2A996751B80A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {D502394E-930B-401A-812F-2A996751B80A}.Debug|Any CPU.Build.0 = Debug|Any CPU + {D502394E-930B-401A-812F-2A996751B80A}.Release|Any CPU.ActiveCfg = Release|Any CPU + {D502394E-930B-401A-812F-2A996751B80A}.Release|Any CPU.Build.0 = Release|Any CPU + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection + GlobalSection(ExtensibilityGlobals) = postSolution + SolutionGuid = {92FA42B0-28BF-4531-B744-F3125DAAC91A} + EndGlobalSection +EndGlobal diff --git a/samples/csharp/getting-started/Ranking_Web/WebRanking/Common/ConsoleHelper.cs b/samples/csharp/getting-started/Ranking_Web/WebRanking/Common/ConsoleHelper.cs new file mode 100644 index 000000000..a262e10be --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/WebRanking/Common/ConsoleHelper.cs @@ -0,0 +1,62 @@ +using Microsoft.ML; +using Microsoft.ML.Data; +using WebRanking.DataStructures; +using System; +using System.Collections.Generic; +using System.Linq; + +namespace WebRanking.Common +{ + public class ConsoleHelper + { + // To evaluate the accuracy of the model's predicted rankings, prints out the Discounted Cumulative Gain and Normalized Discounted Cumulative Gain for search queries. + public static void EvaluateMetrics(MLContext mlContext, IDataView predictions) + { + // Evaluate the metrics for the data using NDCG; by default, metrics for the up to 3 search results in the query are reported (e.g. NDCG@3). + RankingMetrics metrics = mlContext.Ranking.Evaluate(predictions); + + Console.WriteLine($"DCG: {string.Join(", ", metrics.DiscountedCumulativeGains.Select((d, i) => $"@{i + 1}:{d:F4}").ToArray())}"); + + Console.WriteLine($"NDCG: {string.Join(", ", metrics.NormalizedDiscountedCumulativeGains.Select((d, i) => $"@{i + 1}:{d:F4}").ToArray())}\n"); + } + + // Performs evaluation with the truncation level set up to 10 search results within a query. + // This is a temporary workaround for this issue: https://github.com/dotnet/machinelearning/issues/2728. + public static void EvaluateMetrics(MLContext mlContext, IDataView scoredData, int truncationLevel) + { + if (truncationLevel < 1 || truncationLevel > 10) + { + throw new InvalidOperationException("Currently metrics are only supported for 1 to 10 truncation levels."); + } + + // Uses reflection to set the truncation level before calling evaluate. + var mlAssembly = AppDomain.CurrentDomain.GetAssemblies().Where(a => a.FullName.Contains("Microsoft.ML.Data")).First(); + var rankEvalType = mlAssembly.DefinedTypes.Where(t => t.Name.Contains("RankingEvaluator")).First(); + + var evalArgsType = rankEvalType.GetNestedType("Arguments"); + var evalArgs = Activator.CreateInstance(rankEvalType.GetNestedType("Arguments")); + + var dcgLevel = evalArgsType.GetField("DcgTruncationLevel"); + dcgLevel.SetValue(evalArgs, truncationLevel); + + var ctor = rankEvalType.GetConstructors().First(); + var evaluator = ctor.Invoke(new object[] { mlContext, evalArgs }); + + var evaluateMethod = rankEvalType.GetMethod("Evaluate"); + RankingMetrics metrics = (RankingMetrics)evaluateMethod.Invoke(evaluator, new object[] { scoredData, "Label", "GroupId", "Score" }); + + Console.WriteLine($"DCG: {string.Join(", ", metrics.DiscountedCumulativeGains.Select((d, i) => $"@{i + 1}:{d:F4}").ToArray())}"); + + Console.WriteLine($"NDCG: {string.Join(", ", metrics.NormalizedDiscountedCumulativeGains.Select((d, i) => $"@{i + 1}:{d:F4}").ToArray())}\n"); + } + + // Prints out the the individual scores used to determine the relative ranking. + public static void PrintScores(IEnumerable predictions) + { + foreach (var prediction in predictions) + { + Console.WriteLine($"GroupId: {prediction.GroupId}, Score: {prediction.Score}"); + } + } + } +} diff --git a/samples/csharp/getting-started/Ranking_Web/WebRanking/DataStructures/SearchResultData.cs b/samples/csharp/getting-started/Ranking_Web/WebRanking/DataStructures/SearchResultData.cs new file mode 100644 index 000000000..177e77a38 --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/WebRanking/DataStructures/SearchResultData.cs @@ -0,0 +1,558 @@ +using Microsoft.ML.Data; + +namespace WebRanking.DataStructures +{ + public class SearchResultData + { + [ColumnName("Label"), LoadColumn(0)] + public uint Label { get; set; } + + + [ColumnName("GroupId"), LoadColumn(1)] + public uint GroupId { get; set; } + + + [ColumnName("CoveredQueryTermNumberAnchor"), LoadColumn(2)] + public float CoveredQueryTermNumberAnchor { get; set; } + + + [ColumnName("CoveredQueryTermNumberTitle"), LoadColumn(3)] + public float CoveredQueryTermNumberTitle { get; set; } + + + [ColumnName("CoveredQueryTermNumberUrl"), LoadColumn(4)] + public float CoveredQueryTermNumberUrl { get; set; } + + + [ColumnName("CoveredQueryTermNumberWholeDocument"), LoadColumn(5)] + public float CoveredQueryTermNumberWholeDocument { get; set; } + + + [ColumnName("CoveredQueryTermNumberBody"), LoadColumn(6)] + public float CoveredQueryTermNumberBody { get; set; } + + + [ColumnName("CoveredQueryTermRatioAnchor"), LoadColumn(7)] + public float CoveredQueryTermRatioAnchor { get; set; } + + + [ColumnName("CoveredQueryTermRatioTitle"), LoadColumn(8)] + public float CoveredQueryTermRatioTitle { get; set; } + + + [ColumnName("CoveredQueryTermRatioUrl"), LoadColumn(9)] + public float CoveredQueryTermRatioUrl { get; set; } + + + [ColumnName("CoveredQueryTermRatioWholeDocument"), LoadColumn(10)] + public float CoveredQueryTermRatioWholeDocument { get; set; } + + + [ColumnName("CoveredQueryTermRatioBody"), LoadColumn(11)] + public float CoveredQueryTermRatioBody { get; set; } + + + [ColumnName("StreamLengthAnchor"), LoadColumn(12)] + public float StreamLengthAnchor { get; set; } + + + [ColumnName("StreamLengthTitle"), LoadColumn(13)] + public float StreamLengthTitle { get; set; } + + + [ColumnName("StreamLengthUrl"), LoadColumn(14)] + public float StreamLengthUrl { get; set; } + + + [ColumnName("StreamLengthWholeDocument"), LoadColumn(15)] + public float StreamLengthWholeDocument { get; set; } + + + [ColumnName("StreamLengthBody"), LoadColumn(16)] + public float StreamLengthBody { get; set; } + + + [ColumnName("IdfAnchor"), LoadColumn(17)] + public float IdfAnchor { get; set; } + + + [ColumnName("IdfTitle"), LoadColumn(18)] + public float IdfTitle { get; set; } + + + [ColumnName("IdfUrl"), LoadColumn(19)] + public float IdfUrl { get; set; } + + + [ColumnName("IdfWholeDocument"), LoadColumn(20)] + public float IdfWholeDocument { get; set; } + + + [ColumnName("IdfBody"), LoadColumn(21)] + public float IdfBody { get; set; } + + + [ColumnName("SumTfAnchor"), LoadColumn(22)] + public float SumTfAnchor { get; set; } + + + [ColumnName("SumTfTitle"), LoadColumn(23)] + public float SumTfTitle { get; set; } + + + [ColumnName("SumTfUrl"), LoadColumn(24)] + public float SumTfUrl { get; set; } + + + [ColumnName("SumTfWholeDocument"), LoadColumn(25)] + public float SumTfWholeDocument { get; set; } + + + [ColumnName("SumTfBody"), LoadColumn(26)] + public float SumTfBody { get; set; } + + + [ColumnName("MinTfAnchor"), LoadColumn(27)] + public float MinTfAnchor { get; set; } + + + [ColumnName("MinTfTitle"), LoadColumn(28)] + public float MinTfTitle { get; set; } + + + [ColumnName("MinTfUrl"), LoadColumn(29)] + public float MinTfUrl { get; set; } + + + [ColumnName("MinTfWholeDocument"), LoadColumn(30)] + public float MinTfWholeDocument { get; set; } + + + [ColumnName("MinTfBody"), LoadColumn(31)] + public float MinTfBody { get; set; } + + + [ColumnName("MaxTfAnchor"), LoadColumn(32)] + public float MaxTfAnchor { get; set; } + + + [ColumnName("MaxTfTitle"), LoadColumn(33)] + public float MaxTfTitle { get; set; } + + + [ColumnName("MaxTfUrl"), LoadColumn(34)] + public float MaxTfUrl { get; set; } + + + [ColumnName("MaxTfWholeDocument"), LoadColumn(35)] + public float MaxTfWholeDocument { get; set; } + + + [ColumnName("MaxTfBody"), LoadColumn(36)] + public float MaxTfBody { get; set; } + + + [ColumnName("MeanTfAnchor"), LoadColumn(37)] + public float MeanTfAnchor { get; set; } + + + [ColumnName("MeanTfTitle"), LoadColumn(38)] + public float MeanTfTitle { get; set; } + + + [ColumnName("MeanTfUrl"), LoadColumn(39)] + public float MeanTfUrl { get; set; } + + + [ColumnName("MeanTfWholeDocument"), LoadColumn(40)] + public float MeanTfWholeDocument { get; set; } + + + [ColumnName("MeanTfBody"), LoadColumn(41)] + public float MeanTfBody { get; set; } + + + [ColumnName("VarianceTfAnchor"), LoadColumn(42)] + public float VarianceTfAnchor { get; set; } + + + [ColumnName("VarianceTfTitle"), LoadColumn(43)] + public float VarianceTfTitle { get; set; } + + + [ColumnName("VarianceTfUrl"), LoadColumn(44)] + public float VarianceTfUrl { get; set; } + + + [ColumnName("VarianceTfWholeDocument"), LoadColumn(45)] + public float VarianceTfWholeDocument { get; set; } + + + [ColumnName("VarianceTfBody"), LoadColumn(46)] + public float VarianceTfBody { get; set; } + + + [ColumnName("SumStreamLengthNormalizedTfAnchor"), LoadColumn(47)] + public float SumStreamLengthNormalizedTfAnchor { get; set; } + + + [ColumnName("SumStreamLengthNormalizedTfTitle"), LoadColumn(48)] + public float SumStreamLengthNormalizedTfTitle { get; set; } + + + [ColumnName("SumStreamLengthNormalizedTfUrl"), LoadColumn(49)] + public float SumStreamLengthNormalizedTfUrl { get; set; } + + + [ColumnName("SumStreamLengthNormalizedTfWholeDocument"), LoadColumn(50)] + public float SumStreamLengthNormalizedTfWholeDocument { get; set; } + + + [ColumnName("SumStreamLengthNormalizedTfBody"), LoadColumn(51)] + public float SumStreamLengthNormalizedTfBody { get; set; } + + + [ColumnName("MinStreamLengthNormalizedTfAnchor"), LoadColumn(52)] + public float MinStreamLengthNormalizedTfAnchor { get; set; } + + + [ColumnName("MinStreamLengthNormalizedTfTitle"), LoadColumn(53)] + public float MinStreamLengthNormalizedTfTitle { get; set; } + + + [ColumnName("MinStreamLengthNormalizedTfUrl"), LoadColumn(54)] + public float MinStreamLengthNormalizedTfUrl { get; set; } + + + [ColumnName("MinStreamLengthNormalizedTfWholeDocument"), LoadColumn(55)] + public float MinStreamLengthNormalizedTfWholeDocument { get; set; } + + + [ColumnName("MinStreamLengthNormalizedTfBody"), LoadColumn(56)] + public float MinStreamLengthNormalizedTfBody { get; set; } + + + [ColumnName("MaxStreamLengthNormalizedTfAnchor"), LoadColumn(57)] + public float MaxStreamLengthNormalizedTfAnchor { get; set; } + + + [ColumnName("MaxStreamLengthNormalizedTfTitle"), LoadColumn(58)] + public float MaxStreamLengthNormalizedTfTitle { get; set; } + + + [ColumnName("MaxStreamLengthNormalizedTfUrl"), LoadColumn(59)] + public float MaxStreamLengthNormalizedTfUrl { get; set; } + + + [ColumnName("MaxStreamLengthNormalizedTfWholeDocument"), LoadColumn(60)] + public float MaxStreamLengthNormalizedTfWholeDocument { get; set; } + + + [ColumnName("MaxStreamLengthNormalizedTfBody"), LoadColumn(61)] + public float MaxStreamLengthNormalizedTfBody { get; set; } + + + [ColumnName("MeanStreamLengthNormalizedTfAnchor"), LoadColumn(62)] + public float MeanStreamLengthNormalizedTfAnchor { get; set; } + + + [ColumnName("MeanStreamLengthNormalizedTfTitle"), LoadColumn(63)] + public float MeanStreamLengthNormalizedTfTitle { get; set; } + + + [ColumnName("MeanStreamLengthNormalizedTfUrl"), LoadColumn(64)] + public float MeanStreamLengthNormalizedTfUrl { get; set; } + + + [ColumnName("MeanStreamLengthNormalizedTfWholeDocument"), LoadColumn(65)] + public float MeanStreamLengthNormalizedTfWholeDocument { get; set; } + + + [ColumnName("MeanStreamLengthNormalizedTfBody"), LoadColumn(66)] + public float MeanStreamLengthNormalizedTfBody { get; set; } + + + [ColumnName("VarianceStreamLengthNormalizedTfAnchor"), LoadColumn(67)] + public float VarianceStreamLengthNormalizedTfAnchor { get; set; } + + + [ColumnName("VarianceStreamLengthNormalizedTfTitle"), LoadColumn(68)] + public float VarianceStreamLengthNormalizedTfTitle { get; set; } + + + [ColumnName("VarianceStreamLengthNormalizedTfUrl"), LoadColumn(69)] + public float VarianceStreamLengthNormalizedTfUrl { get; set; } + + + [ColumnName("VarianceStreamLengthNormalizedTfWholeDocument"), LoadColumn(70)] + public float VarianceStreamLengthNormalizedTfWholeDocument { get; set; } + + + [ColumnName("VarianceStreamLengthNormalizedTfBody"), LoadColumn(71)] + public float VarianceStreamLengthNormalizedTfBody { get; set; } + + + [ColumnName("SumTfidfAnchor"), LoadColumn(72)] + public float SumTfidfAnchor { get; set; } + + + [ColumnName("SumTfidfTitle"), LoadColumn(73)] + public float SumTfidfTitle { get; set; } + + + [ColumnName("SumTfidfUrl"), LoadColumn(74)] + public float SumTfidfUrl { get; set; } + + + [ColumnName("SumTfidfWholeDocument"), LoadColumn(75)] + public float SumTfidfWholeDocument { get; set; } + + + [ColumnName("SumTfidfBody"), LoadColumn(76)] + public float SumTfidfBody { get; set; } + + + [ColumnName("MinTfidfAnchor"), LoadColumn(77)] + public float MinTfidfAnchor { get; set; } + + + [ColumnName("MinTfidfTitle"), LoadColumn(78)] + public float MinTfidfTitle { get; set; } + + + [ColumnName("MinTfidfUrl"), LoadColumn(79)] + public float MinTfidfUrl { get; set; } + + + [ColumnName("MinTfidfWholeDocument"), LoadColumn(80)] + public float MinTfidfWholeDocument { get; set; } + + + [ColumnName("MinTfidfBody"), LoadColumn(81)] + public float MinTfidfBody { get; set; } + + + [ColumnName("MaxTfidfAnchor"), LoadColumn(82)] + public float MaxTfidfAnchor { get; set; } + + + [ColumnName("MaxTfidfTitle"), LoadColumn(83)] + public float MaxTfidfTitle { get; set; } + + + [ColumnName("MaxTfidfUrl"), LoadColumn(84)] + public float MaxTfidfUrl { get; set; } + + + [ColumnName("MaxTfidfWholeDocument"), LoadColumn(85)] + public float MaxTfidfWholeDocument { get; set; } + + + [ColumnName("MaxTfidfBody"), LoadColumn(86)] + public float MaxTfidfBody { get; set; } + + + [ColumnName("MeanTfidfAnchor"), LoadColumn(87)] + public float MeanTfidfAnchor { get; set; } + + + [ColumnName("MeanTfidfTitle"), LoadColumn(88)] + public float MeanTfidfTitle { get; set; } + + + [ColumnName("MeanTfidfUrl"), LoadColumn(89)] + public float MeanTfidfUrl { get; set; } + + + [ColumnName("MeanTfidfWholeDocument"), LoadColumn(90)] + public float MeanTfidfWholeDocument { get; set; } + + + [ColumnName("MeanTfidfBody"), LoadColumn(91)] + public float MeanTfidfBody { get; set; } + + + [ColumnName("VarianceTfidfAnchor"), LoadColumn(92)] + public float VarianceTfidfAnchor { get; set; } + + + [ColumnName("VarianceTfidfTitle"), LoadColumn(93)] + public float VarianceTfidfTitle { get; set; } + + + [ColumnName("VarianceTfidfUrl"), LoadColumn(94)] + public float VarianceTfidfUrl { get; set; } + + + [ColumnName("VarianceTfidfWholeDocument"), LoadColumn(95)] + public float VarianceTfidfWholeDocument { get; set; } + + + [ColumnName("VarianceTfidfBody"), LoadColumn(96)] + public float VarianceTfidfBody { get; set; } + + + [ColumnName("BooleanModelAnchor"), LoadColumn(97)] + public float BooleanModelAnchor { get; set; } + + + [ColumnName("BooleanModelTitle"), LoadColumn(98)] + public float BooleanModelTitle { get; set; } + + + [ColumnName("BooleanModelUrl"), LoadColumn(99)] + public float BooleanModelUrl { get; set; } + + + [ColumnName("BooleanModelWholeDocument"), LoadColumn(100)] + public float BooleanModelWholeDocument { get; set; } + + + [ColumnName("BooleanModelBody"), LoadColumn(101)] + public float BooleanModelBody { get; set; } + + + [ColumnName("VectorSpaceModelAnchor"), LoadColumn(102)] + public float VectorSpaceModelAnchor { get; set; } + + + [ColumnName("VectorSpaceModelTitle"), LoadColumn(103)] + public float VectorSpaceModelTitle { get; set; } + + + [ColumnName("VectorSpaceModelUrl"), LoadColumn(104)] + public float VectorSpaceModelUrl { get; set; } + + + [ColumnName("VectorSpaceModelWholeDocument"), LoadColumn(105)] + public float VectorSpaceModelWholeDocument { get; set; } + + + [ColumnName("VectorSpaceModelBody"), LoadColumn(106)] + public float VectorSpaceModelBody { get; set; } + + + [ColumnName("Bm25Anchor"), LoadColumn(107)] + public float Bm25Anchor { get; set; } + + + [ColumnName("Bm25Title"), LoadColumn(108)] + public float Bm25Title { get; set; } + + + [ColumnName("Bm25Url"), LoadColumn(109)] + public float Bm25Url { get; set; } + + + [ColumnName("Bm25WholeDocument"), LoadColumn(110)] + public float Bm25WholeDocument { get; set; } + + + [ColumnName("Bm25Body"), LoadColumn(111)] + public float Bm25Body { get; set; } + + + [ColumnName("LmirAbsAnchor"), LoadColumn(112)] + public float LmirAbsAnchor { get; set; } + + + [ColumnName("LmirAbsTitle"), LoadColumn(113)] + public float LmirAbsTitle { get; set; } + + + [ColumnName("LmirAbsUrl"), LoadColumn(114)] + public float LmirAbsUrl { get; set; } + + + [ColumnName("LmirAbsWholeDocument"), LoadColumn(115)] + public float LmirAbsWholeDocument { get; set; } + + + [ColumnName("LmirAbsBody"), LoadColumn(116)] + public float LmirAbsBody { get; set; } + + + [ColumnName("LmirDirAnchor"), LoadColumn(117)] + public float LmirDirAnchor { get; set; } + + + [ColumnName("LmirDirTitle"), LoadColumn(118)] + public float LmirDirTitle { get; set; } + + + [ColumnName("LmirDirUrl"), LoadColumn(119)] + public float LmirDirUrl { get; set; } + + + [ColumnName("LmirDirWholeDocument"), LoadColumn(120)] + public float LmirDirWholeDocument { get; set; } + + + [ColumnName("LmirDirBody"), LoadColumn(121)] + public float LmirDirBody { get; set; } + + + [ColumnName("LmirJmAnchor"), LoadColumn(122)] + public float LmirJmAnchor { get; set; } + + + [ColumnName("LmirJmTitle"), LoadColumn(123)] + public float LmirJmTitle { get; set; } + + + [ColumnName("LmirJmUrl"), LoadColumn(124)] + public float LmirJmUrl { get; set; } + + + [ColumnName("LmirJmWholeDocument"), LoadColumn(125)] + public float LmirJmWholeDocument { get; set; } + + + [ColumnName("LmirJm"), LoadColumn(126)] + public float LmirJm { get; set; } + + + [ColumnName("NumberSlashInUrl"), LoadColumn(127)] + public float NumberSlashInUrl { get; set; } + + + [ColumnName("LengthUrl"), LoadColumn(128)] + public float LengthUrl { get; set; } + + + [ColumnName("InlinkNumber"), LoadColumn(129)] + public float InlinkNumber { get; set; } + + + [ColumnName("OutlinkNumber"), LoadColumn(130)] + public float OutlinkNumber { get; set; } + + + [ColumnName("PageRank"), LoadColumn(131)] + public float PageRank { get; set; } + + + [ColumnName("SiteRank"), LoadColumn(132)] + public float SiteRank { get; set; } + + + [ColumnName("QualityScore"), LoadColumn(133)] + public float QualityScore { get; set; } + + + [ColumnName("QualityScore2"), LoadColumn(134)] + public float QualityScore2 { get; set; } + + + [ColumnName("QueryUrlClickCount"), LoadColumn(135)] + public float QueryUrlClickCount { get; set; } + + + [ColumnName("UrlClickCount"), LoadColumn(136)] + public float UrlClickCount { get; set; } + + + [ColumnName("UrlDwellTime"), LoadColumn(137)] + public float UrlDwellTime { get; set; } + } +} diff --git a/samples/csharp/getting-started/Ranking_Web/WebRanking/DataStructures/SearchResultPrediction.cs b/samples/csharp/getting-started/Ranking_Web/WebRanking/DataStructures/SearchResultPrediction.cs new file mode 100644 index 000000000..9b6db3933 --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/WebRanking/DataStructures/SearchResultPrediction.cs @@ -0,0 +1,17 @@ + +namespace WebRanking.DataStructures +{ + // Representation of the prediction made by the model (e.g. ranker). + public class SearchResultPrediction + { + public uint GroupId { get; set; } + + public uint Label { get; set; } + + // Prediction made by the model that is used to indicate the relative ranking of the candidate search results. + public float Score { get; set; } + + // Values that are influential in determining the relevance of a data instance. This is a vector that contains concatenated columns from the underlying dataset. + public float[] Features { get; set; } + } +} diff --git a/samples/csharp/getting-started/Ranking_Web/WebRanking/Program.cs b/samples/csharp/getting-started/Ranking_Web/WebRanking/Program.cs new file mode 100644 index 000000000..1a2eaf1cb --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/WebRanking/Program.cs @@ -0,0 +1,183 @@ +using Microsoft.ML; +using WebRanking.Common; +using WebRanking.DataStructures; +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Net; + +namespace WebRanking +{ + class Program + { + const string AssetsPath = @"../../../Assets"; + const string TrainDatasetUrl = "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KTrain720kRows.tsv"; + const string ValidationDatasetUrl = "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KValidate240kRows.tsv"; + const string TestDatasetUrl = "https://aka.ms/mlnet-resources/benchmarks/MSLRWeb10KTest240kRows.tsv"; + + readonly static string InputPath = Path.Combine(AssetsPath, "Input"); + readonly static string OutputPath = Path.Combine(AssetsPath, "Output"); + readonly static string TrainDatasetPath = Path.Combine(InputPath, "MSLRWeb10KTrain720kRows.tsv"); + readonly static string ValidationDatasetPath = Path.Combine(InputPath, "MSLRWeb10KValidate240kRows.tsv"); + readonly static string TestDatasetPath = Path.Combine(InputPath, "MSLRWeb10KTest240kRows.tsv"); + readonly static string ModelPath = Path.Combine(OutputPath, "RankingModel.zip"); + + static void Main(string[] args) + { + // Create a common ML.NET context. + // Seed set to any number so you have a deterministic environment for repeateable results. + MLContext mlContext = new MLContext(seed: 0); + + try + { + PrepareData(InputPath, OutputPath, TrainDatasetPath, TrainDatasetUrl, TestDatasetUrl, TestDatasetPath, ValidationDatasetUrl, ValidationDatasetPath); + + var model = TrainModel(mlContext, TrainDatasetPath, ModelPath); + + EvaluateModel(mlContext, model, TestDatasetPath); + + ConsumeModel(mlContext, ModelPath, TestDatasetPath); + } + catch (Exception e) + { + Console.WriteLine(e.Message); + } + + Console.ReadLine(); + } + + static void PrepareData(string inputPath, string outputPath, string trainDatasetPath, string trainDatasetUrl, + string testDatasetUrl, string testDatasetPath, string validationDatasetUrl, string validationDatasetPath) + { + Console.WriteLine("===== Prepare data =====\n"); + + if (!Directory.Exists(outputPath)) + { + Directory.CreateDirectory(outputPath); + } + + if (!Directory.Exists(inputPath)) + { + Directory.CreateDirectory(inputPath); + } + + if (!File.Exists(trainDatasetPath)) + { + Console.WriteLine("===== Download the train dataset - this may take several minutes =====\n"); + using (var client = new WebClient()) + { + client.DownloadFile(trainDatasetUrl, TrainDatasetPath); + } + } + + if (!File.Exists(validationDatasetPath)) + { + Console.WriteLine("===== Download the validation dataset - this may take several minutes =====\n"); + using (var client = new WebClient()) + { + client.DownloadFile(validationDatasetUrl, validationDatasetPath); + } + } + + if (!File.Exists(testDatasetPath)) + { + Console.WriteLine("===== Download the test dataset - this may take several minutes =====\n"); + using (var client = new WebClient()) + { + client.DownloadFile(testDatasetUrl, testDatasetPath); + } + } + + Console.WriteLine("===== Download is finished =====\n"); + } + + static ITransformer TrainModel(MLContext mlContext, string trainDatasetPath, string modelPath) + { + const string FeaturesVectorName = "Features"; + + Console.WriteLine("===== Load the training dataset =====\n"); + + // Load the training dataset. + IDataView trainData = mlContext.Data.LoadFromTextFile(trainDatasetPath, separatorChar: '\t', hasHeader: true); + + Console.WriteLine("===== Set up the trainer =====\n"); + + // Specify the columns to include in the feature input data. + var featureCols = trainData.Schema.AsQueryable() + .Select(s => s.Name) + .Where(c => + c != nameof(SearchResultData.Label) && + c != nameof(SearchResultData.GroupId)) + .ToArray(); + + // Create an Estimator and transform the data: + // 1. Concatenate the feature columns into a single Features vector. + // 2. Create a key type for the label input data by using the value to key transform. + // 3. Create a key type for the group input data by using a hash transform. + IEstimator dataPipeline = mlContext.Transforms.Concatenate(FeaturesVectorName, featureCols) + .Append(mlContext.Transforms.Conversion.MapValueToKey(nameof(SearchResultData.Label))) + .Append(mlContext.Transforms.Conversion.Hash(nameof(SearchResultData.GroupId), nameof(SearchResultData.GroupId), numberOfBits: 20)); + + // Set the LightGBM LambdaRank trainer. + IEstimator trainer = mlContext.Ranking.Trainers.LightGbm(labelColumnName: nameof(SearchResultData.Label), featureColumnName: FeaturesVectorName, rowGroupColumnName: nameof(SearchResultData.GroupId)); + IEstimator trainerPipeline = dataPipeline.Append(trainer); + + Console.WriteLine("===== Train the model =====\n"); + + // Training the model is a process of running the chosen algorithm on the given data. To perform training you need to call the Fit() method. + ITransformer model = trainerPipeline.Fit(trainData); + IDataView transformedTrainData = model.Transform(trainData); +; + Console.WriteLine("===== Save the model =====\n"); + + // Save the model + mlContext.Model.Save(model, null, modelPath); + + return model; + } + + static void EvaluateModel(MLContext mlContext, ITransformer model, string testDatasetPath) + { + Console.WriteLine("===== Evaluate the model's result quality with test data =====\n"); + + // Load the test data and use the model to perform predictions on the test data. + IDataView testData = mlContext.Data.LoadFromTextFile(testDatasetPath, separatorChar: '\t', hasHeader: false); + IDataView predictions = model.Transform(testData); + + Console.WriteLine("===== Use metrics for the data using NDCG@3 =====\n"); + + // Evaluate the metrics for the data using NDCG; by default, metrics for the up to 3 search results in the query are reported (e.g. NDCG@3). + ConsoleHelper.EvaluateMetrics(mlContext, predictions); + + Console.WriteLine("===== Use metrics for the data using NDCG@10 =====\n"); + + // Evaluate metrics for up to 10 search results (e.g. NDCG@10). + ConsoleHelper.EvaluateMetrics(mlContext, predictions, 10); + } + + public static void ConsumeModel(MLContext mlContext, string modelPath, string testDatasetPath) + { + Console.WriteLine("===== Consume the model =====\n"); + + // Load test data and use the model to perform predictions on it. + IDataView testData = mlContext.Data.LoadFromTextFile(testDatasetPath, separatorChar: '\t', hasHeader: false); + + // Load the model. + DataViewSchema predictionPipelineSchema; + ITransformer predictionPipeline = mlContext.Model.Load(modelPath, out predictionPipelineSchema); + + // Predict rankings. + IDataView predictions = predictionPipeline.Transform(testData); + + // In the predictions, get the scores of the search results included in the first query (e.g. group). + IEnumerable searchQueries = mlContext.Data.CreateEnumerable(predictions, reuseRowObject: false); + var firstGroupId = searchQueries.First().GroupId; + IEnumerable firstGroupPredictions = searchQueries.Take(100).Where(p => p.GroupId == firstGroupId).OrderByDescending(p => p.Score).ToList(); + + // The individual scores themselves are NOT a useful measure of result quality; instead, they are only useful as a relative measure to other scores in the group. + // The scores are used to determine the ranking where a higher score indicates a higher ranking versus another candidate result. + ConsoleHelper.PrintScores(firstGroupPredictions); + } + } +} diff --git a/samples/csharp/getting-started/Ranking_Web/WebRanking/WebRanking.csproj b/samples/csharp/getting-started/Ranking_Web/WebRanking/WebRanking.csproj new file mode 100644 index 000000000..fbca6af6c --- /dev/null +++ b/samples/csharp/getting-started/Ranking_Web/WebRanking/WebRanking.csproj @@ -0,0 +1,13 @@ + + + + Exe + netcoreapp2.2 + + + + + + + + diff --git a/samples/csharp/v1.0.0-All-Samples.sln b/samples/csharp/v1.0.0-All-Samples.sln index 7d466c25c..44487c525 100644 --- a/samples/csharp/v1.0.0-All-Samples.sln +++ b/samples/csharp/v1.0.0-All-Samples.sln @@ -1,7 +1,7 @@  Microsoft Visual Studio Solution File, Format Version 12.00 -# Visual Studio Version 16 -VisualStudioVersion = 16.0.28803.452 +# Visual Studio 15 +VisualStudioVersion = 15.0.28307.705 MinimumVisualStudioVersion = 10.0.40219.1 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "BikeSharingDemand.Solution", "BikeSharingDemand.Solution", "{820E8AF2-A47D-4AB8-A4AF-5CDFF97EBCDF}" EndProject @@ -129,6 +129,10 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "TFImageClassififcationE2E.S EndProject Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "TensorFlowImageClassification", "end-to-end-apps\DeepLearning_ImageClassification_TensorFlow\TensorFlowImageClassification\TensorFlowImageClassification.csproj", "{C5D5BEBF-DC10-4065-A27B-AB56E1ABCA47}" EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "WebRanking.Solution", "WebRanking.Solution", "{B76DD928-A78E-497C-BA7D-83C5864452F9}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "WebRanking", "getting-started\Ranking_Web\WebRanking\WebRanking.csproj", "{4EA790BB-76C7-471A-ADE4-6FBD183C461B}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU @@ -425,6 +429,14 @@ Global {C5D5BEBF-DC10-4065-A27B-AB56E1ABCA47}.Release|Any CPU.Build.0 = Release|Any CPU {C5D5BEBF-DC10-4065-A27B-AB56E1ABCA47}.Release|x64.ActiveCfg = Release|Any CPU {C5D5BEBF-DC10-4065-A27B-AB56E1ABCA47}.Release|x64.Build.0 = Release|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Debug|Any CPU.Build.0 = Debug|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Debug|x64.ActiveCfg = Debug|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Debug|x64.Build.0 = Debug|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Release|Any CPU.ActiveCfg = Release|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Release|Any CPU.Build.0 = Release|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Release|x64.ActiveCfg = Release|Any CPU + {4EA790BB-76C7-471A-ADE4-6FBD183C461B}.Release|x64.Build.0 = Release|Any CPU EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE @@ -466,6 +478,7 @@ Global {EA9E37C6-8C62-4370-A9CF-369D002B89B6} = {7C3A7DA5-CBEB-420F-B7AC-CDE34BE2D52E} {F2C0FCE9-9F76-4318-826E-892441E4A169} = {EF9F8695-25DE-4FE4-894A-6DE24E0BDD73} {C5D5BEBF-DC10-4065-A27B-AB56E1ABCA47} = {F59681C2-D829-4538-A41A-568F7A7D07FD} + {4EA790BB-76C7-471A-ADE4-6FBD183C461B} = {B76DD928-A78E-497C-BA7D-83C5864452F9} EndGlobalSection GlobalSection(ExtensibilityGlobals) = postSolution SolutionGuid = {98369941-33DD-450C-A410-B9A91C8CDE91}