Merge pull request #2 from dotnet/master

Update local fork

Author: zeahmed

Documentation/building/unix-instructions.md

@@ -7,37 +7,44 @@ Building ML.NET on Linux and macOS
 3. Navigate to the `machinelearning` directory
 4. Run the build script `./build.sh`
 
-Calling the script `build.sh` builds both the native and managed code.
+Calling the script `./build.sh` builds both the native and managed code.
 
-For more information about the different options when building, run `build.sh -?` and look at examples in the [developer-guide](../project-docs/developer-guide.md).
+For more information about the different options when building, run `./build.sh -?` and look at examples in the [developer-guide](../project-docs/developer-guide.md).
 
 ## Minimum Hardware Requirements
 - 2GB RAM
+- x64
 
-## Prerequisites (native build)
+## Prerequisites
 
 ### Linux
 
-First, the package lists might need to be updated
+The following components are needed:
 
-`sudo apt-get update`
+* git
+* clang-3.9
+* cmake 2.8.12
+* libunwind8
+* curl
+* All the requirements necessary to run .NET Core 2.0 applications: libssl1.0.0 (1.0.2 for Debian 9) and libicu5x (libicu52 for ubuntu 14.x, libicu55 for ubuntu 16.x, and libicu57 for ubuntu 17.x). For more information on prerequisites in different linux distributions click [here](https://docs.microsoft.com/en-us/dotnet/core/linux-prerequisites?tabs=netcore2x).
 
-On Linux, the following components are needed
+e.g. for Ubuntu 16.x:
 
-* CMake on the PATH
-* Clang 3.5+ (same requirements as coreclr/corefx)
-* All the requirements necessary to run .NET Core 2.0 applications
-* libunwind
-* curl
+```sh
+sudo apt-get update
+sudo apt-get install git clang-3.9 cmake libunwind8 curl
+sudo apt-get install libssl1.0.0 libicu55
+```
 
 ### macOS
 
 macOS 10.12 or higher is needed to build dotnet/machinelearning.
 
 On macOS a few components are needed which are not provided by a default developer setup:
-* CMake
+* cmake 3.10.3
+* All the requirements necessary to run .NET Core 2.0 applications. To view macOS prerequisites click [here](https://docs.microsoft.com/en-us/dotnet/core/macos-prerequisites?tabs=netcore2x).
 
 One way of obtaining CMake is via [Homebrew](http://brew.sh):
 ```sh
 $ brew install cmake
-```
+```

README.md

@@ -14,23 +14,25 @@ Along with these ML capabilities this first release of ML.NET also brings the fi
 
 ## Installation
 
+[![NuGet Status](https://img.shields.io/nuget/v/Microsoft.ML.svg?style=flat)](https://www.nuget.org/packages/Microsoft.ML/)
+
 ML.NET runs on Windows, Linux, and macOS - any platform where 64 bit [.NET Core](https://github.com/dotnet/core) or later is available.
 
-The current release is 0.1. Check out the [release notes](https://github.com/dotnet/machinelearning/blob/master/Documentation/release-notes/0.1/release-0.1.md).
+The current release is 0.1. Check out the [release notes](Documentation/release-notes/0.1/release-0.1.md).
 
 First ensure you have installed [.NET Core 2.0](https://www.microsoft.com/net/learn/get-started) or later. ML.NET also works on the .NET Framework. Note that ML.NET currently must run in a 64 bit process.
 
-Once you have an app, you can install ML.NET NuGet from the .NET Core CLI using:
+Once you have an app, you can install the ML.NET NuGet package from the .NET Core CLI using:
 ```
 dotnet add package Microsoft.ML
 ```
 
-or from the package manager:
+or from the NuGet package manager:
 ```
 Install-Package Microsoft.ML
 ```
 
-Or alternatively you can add the Microsoft.ML package from within Visual Studio's NuGet package manager.
+Or alternatively you can add the Microsoft.ML package from within Visual Studio's NuGet package manager or via [Paket](https://github.com/fsprojects/Paket).
 
 ## Building
 
@@ -55,7 +57,8 @@ For more information, see the [.NET Foundation Code of Conduct](https://dotnetfo
 
 ## Examples
 
-Here's an example of code to train a model to predict sentiment from text samples. (You can see the complete sample [here](https://github.com/dotnet/machinelearning/blob/master/test/Microsoft.ML.Tests/Scenarios/Scenario3_SentimentPrediction.cs)):
+Here's an example of code to train a model to predict sentiment from text samples. 
+(You can see the complete sample [here](test/Microsoft.ML.Tests/Scenarios/SentimentPredictionTests.cs)):
 
 ```C#
 var pipeline = new LearningPipeline();

build/BranchInfo.props

@@ -1,7 +1,7 @@
 <Project>
   <PropertyGroup>
     <MajorVersion>0</MajorVersion>
-    <MinorVersion>1</MinorVersion>
+    <MinorVersion>2</MinorVersion>
     <PatchVersion>0</PatchVersion>
     <PreReleaseLabel>preview</PreReleaseLabel>
   </PropertyGroup>

src/Microsoft.ML/LearningPipeline.cs

@@ -26,25 +26,113 @@ public ScorerPipelineStep(Var<IDataView> data, Var<ITransformModel> model)
         public Var<ITransformModel> Model { get; }
     }
 
+
+    /// <summary>
+    /// The <see cref="LearningPipeline"/> class is used to define the steps needed to perform a desired machine learning task.<para/>
+    /// The steps are defined by adding a data loader (e.g. <see cref="TextLoader"/>) followed by zero or more transforms (e.g. <see cref="Microsoft.ML.Transforms.TextFeaturizer"/>) 
+    /// and at most one trainer/learner (e.g. <see cref="Microsoft.ML.Trainers.FastTreeBinaryClassifier"/>) in the pipeline.
+    /// 
+    /// </summary>
+    /// <example>
+    /// <para/>
+    /// For example,<para/>
+    /// <code>
+    /// var pipeline = new LearningPipeline();
+    /// pipeline.Add(new TextLoader &lt;SentimentData&gt; (dataPath, separator: ","));
+    /// pipeline.Add(new TextFeaturizer("Features", "SentimentText"));
+    /// pipeline.Add(new FastTreeBinaryClassifier());
+    /// 
+    /// var model = pipeline.Train&lt;SentimentData, SentimentPrediction&gt;();
+    /// </code>
+    /// </example>
     [DebuggerTypeProxy(typeof(LearningPipelineDebugProxy))]
     public class LearningPipeline : ICollection<ILearningPipelineItem>
     {
         private List<ILearningPipelineItem> Items { get; } = new List<ILearningPipelineItem>();
 
+        /// <summary>
+        /// Construct an empty <see cref="LearningPipeline"/> object.
+        /// </summary>
         public LearningPipeline()
         {
         }
 
+        /// <summary>
+        /// Get the count of ML components in the <see cref="LearningPipeline"/> object
+        /// </summary>
         public int Count => Items.Count;
         public bool IsReadOnly => false;
+
+        /// <summary>
+        /// Add a data loader, transform or trainer into the pipeline. 
+        /// Possible data loader(s), transforms and trainers options are
+        /// <para>
+        /// Data Loader:
+        ///     <see cref="Microsoft.ML.TextLoader{TInput}" />
+        ///     etc.
+        /// </para>
+        /// <para>
+        /// Transforms:
+        ///     <see cref="Microsoft.ML.Transforms.Dictionarizer"/>,
+        ///     <see cref="Microsoft.ML.Transforms.CategoricalOneHotVectorizer"/>
+        ///     <see cref="Microsoft.ML.Transforms.MinMaxNormalizer"/>,
+        ///     <see cref="Microsoft.ML.Transforms.ColumnCopier"/>,
+        ///     <see cref="Microsoft.ML.Transforms.ColumnConcatenator"/>,
+        ///     <see cref="Microsoft.ML.Transforms.TextFeaturizer"/>,
+        ///     etc.
+        /// </para>
+        /// <para>
+        /// Trainers:
+        ///     <see cref="Microsoft.ML.Trainers.AveragedPerceptronBinaryClassifier"/>,
+        ///     <see cref="Microsoft.ML.Trainers.LogisticRegressor"/>,
+        ///     <see cref="Microsoft.ML.Trainers.StochasticDualCoordinateAscentClassifier"/>,
+        ///     <see cref="Microsoft.ML.Trainers.FastTreeRegressor"/>,
+        ///     etc.
+        /// </para>
+        /// For a complete list of transforms and trainers, please see "Microsoft.ML.Transforms" and "Microsoft.ML.Trainers" namespaces.
+        /// </summary>
+        /// <param name="item">Any ML component (data loader, transform or trainer) defined as <see cref="ILearningPipelineItem"/>.</param>
         public void Add(ILearningPipelineItem item) => Items.Add(item);
+
+        /// <summary>
+        /// Remove all the loaders/transforms/trainers from the pipeline.
+        /// </summary>
         public void Clear() => Items.Clear();
+
+        /// <summary>
+        /// Check if a specific loader/transform/trainer is in the pipeline?
+        /// </summary>
+        /// <param name="item">Any ML component (data loader, transform or trainer) defined as <see cref="ILearningPipelineItem"/>.</param>
+        /// <returns>true if item is found in the pipeline; otherwise, false.</returns>
         public bool Contains(ILearningPipelineItem item) => Items.Contains(item);
+
+        /// <summary>
+        /// Copy the pipeline items into an array.
+        /// </summary>
+        /// <param name="array">The one-dimensional Array that is the destination of the elements copied from.</param>
+        /// <param name="arrayIndex">The zero-based index in <paramref name="array" /> at which copying begins.</param>
         public void CopyTo(ILearningPipelineItem[] array, int arrayIndex) => Items.CopyTo(array, arrayIndex);
         public IEnumerator<ILearningPipelineItem> GetEnumerator() => Items.GetEnumerator();
+
+        /// <summary>
+        /// Remove an item from the pipeline.
+        /// </summary>
+        /// <param name="item"><see cref="ILearningPipelineItem"/> to remove.</param>
+        /// <returns>true if item was removed from the pipeline; otherwise, false.</returns>
         public bool Remove(ILearningPipelineItem item) => Items.Remove(item);
         IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
 
+        /// <summary>
+        /// Train the model using the ML components in the pipeline.
+        /// </summary>
+        /// <typeparam name="TInput">Type of data instances the model will be trained on. It's a custom type defined by the user according to the structure of data.
+        /// <para/>
+        /// Please see https://www.microsoft.com/net/learn/apps/machine-learning-and-ai/ml-dotnet/get-started/windows for more details on input type.
+        /// </typeparam>
+        /// <typeparam name="TOutput">Ouput type. The prediction will be return based on this type.
+        /// Please see https://www.microsoft.com/net/learn/apps/machine-learning-and-ai/ml-dotnet/get-started/windows for more details on output type.
+        /// </typeparam>
+        /// <returns>PredictionModel object. This is the model object used for prediction on new instances. </returns>
         public PredictionModel<TInput, TOutput> Train<TInput, TOutput>()
             where TInput : class
             where TOutput : class, new()

src/Microsoft.ML/TextLoader.cs

@@ -55,7 +55,7 @@ private void SetCustomStringFromType(bool useHeader, string separator,
             {
                 var mappingAttr = field.GetCustomAttribute<ColumnAttribute>();
                 if(mappingAttr == null)
-                    throw Contracts.ExceptParam(nameof(field.Name), " is missing ColumnAttribute");
+                    throw Contracts.ExceptParam(field.Name, $"{field.Name} is missing ColumnAttribute");
 
                 schemaBuilder.AppendFormat("col={0}:{1}:{2} ",
                     mappingAttr.Name ?? field.Name, 

src/Native/FastTreeNative/ExpandFloatType.cpp

@@ -16,7 +16,8 @@
 #include "SumupOneBit.h"
 
 // Ideally we should expand this using C++ templates. 
-// However, In order to exporting functions from DLL float and double versions need to have different names (cannot be overloaded on type parameters)// Expanding here with ugly pre-processor macros to get double and float versions (with fucntion mes suffixes _float and _double)
+// However, in order to export functions from DLL, float and double versions need to have different names (cannot be overloaded on type parameters)
+// Expanding here with ugly pre-processor macros to get double and float versions (with function name suffixes _float and _double)
 // --andrzejp, 2010-03-05
 
 #define FloatType float

src/Native/FastTreeNative/getderivatives.cpp

@@ -106,7 +106,7 @@ EXPORT_API(void) C_GetDerivatives(
 
             bool sameLabel = labelHigh == pLabels[low];
 
-            // calculate the lambdaP for this pair by looking it up in the lambdaTable (computed in LambdaMart.FillLambdaTable)
+            // calculate the lambdaP for this pair by looking it up in the sigmoidTable (e.g. computed in FastTreeRanking.FillSigmoidTable)
             double lambdaP;
             if (scoreHighMinusLow <= minScore) lambdaP = sigmoidTable[0];
             else if (scoreHighMinusLow >= maxScore) lambdaP = sigmoidTable[sigmoidTableLength - 1];

test/Microsoft.ML.Tests/Scenarios/Scenario_HousePricePrediction.cs renamed to test/Microsoft.ML.Tests/Scenarios/HousePricePredictionTests.cs

@@ -2,14 +2,17 @@
 // The .NET Foundation licenses this file to you under the MIT license.
 // See the LICENSE file in the project root for more information.
 
+using Microsoft.ML.Models;
 using Microsoft.ML.Runtime.Api;
 using Microsoft.ML.TestFramework;
+using Microsoft.ML.Trainers;
+using Microsoft.ML.Transforms;
 using Xunit;
 using Xunit.Abstractions;
 
 namespace Microsoft.ML.Scenarios
 {
-    public partial class Top5Scenarios : BaseTestClass
+    public partial class ScenariosTests : BaseTestClass
     {
         /*
          A real-estate firm Contoso wants to add a house price prediction to their ASP.NET/Xamarin application.
@@ -121,7 +124,7 @@ public class HousePricePrediction
             public float Price;
         }
 
-        public Top5Scenarios(ITestOutputHelper output) : base(output)
+        public ScenariosTests(ITestOutputHelper output) : base(output)
         {
         }
     }

test/Microsoft.ML.Tests/Scenarios/Scenario_TrainPredictionModel.cs renamed to test/Microsoft.ML.Tests/Scenarios/HousePriceTrainAndPredictionTests.cs

@@ -1,15 +1,18 @@
-// Licensed to the .NET Foundation under one or more agreements.
+// Licensed to the .NET Foundation under one or more agreements.
 // The .NET Foundation licenses this file to you under the MIT license.
 // See the LICENSE file in the project root for more information.
 
 using Microsoft.ML.Models;
+using Microsoft.ML.Runtime.Api;
+using Microsoft.ML.TestFramework;
 using Microsoft.ML.Trainers;
 using Microsoft.ML.Transforms;
 using Xunit;
+using Xunit.Abstractions;
 
 namespace Microsoft.ML.Scenarios
 {
-    public partial class Top5Scenarios
+    public partial class ScenariosTests
     {
         [Fact(Skip = "Missing data set. See https://github.com/dotnet/machinelearning/issues/3")]
         public void TrainAndPredictHousePriceModelTest()
@@ -70,4 +73,3 @@ public void TrainAndPredictHousePriceModelTest()
         }
     }
 }
-

test/Microsoft.ML.Tests/Scenarios/TrainAndPredictIrisModelTest.cs renamed to test/Microsoft.ML.Tests/Scenarios/IrisPlantClassificationTests.cs

@@ -10,7 +10,7 @@
 
 namespace Microsoft.ML.Scenarios
 {
-    public partial class Top5Scenarios
+    public partial class ScenariosTests
     {
         [Fact]
         public void TrainAndPredictIrisModelTest()