Skip to content

Scrubbing schema related transforms #2863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 6, 2019
Merged

Scrubbing schema related transforms #2863

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/Microsoft.ML.Data/Transforms/ColumnConcatenatingEstimator.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ public sealed class ColumnConcatenatingEstimator : IEstimator<ColumnConcatenatin
internal ColumnConcatenatingEstimator(IHostEnvironment env, string outputColumnName, params string[] inputColumnNames)
{
Contracts.CheckValue(env, nameof(env));
_host = env.Register("ColumnConcatenatingEstimator ");
_host = env.Register(nameof(ColumnConcatenatingEstimator));

_host.CheckNonEmpty(outputColumnName, nameof(outputColumnName));
_host.CheckValue(inputColumnNames, nameof(inputColumnNames));
Expand Down
4 changes: 1 addition & 3 deletions src/Microsoft.ML.Data/Transforms/ColumnConcatenatingTransformer.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,7 @@
using Microsoft.ML;
using Microsoft.ML.CommandLine;
using Microsoft.ML.Data;
using Microsoft.ML.EntryPoints;
using Microsoft.ML.Internal.Utilities;
using Microsoft.ML.Model;
using Microsoft.ML.Model.OnnxConverter;
using Microsoft.ML.Model.Pfa;
using Microsoft.ML.Transforms;
Expand Down Expand Up @@ -223,7 +221,7 @@ internal ColumnOptions(ModelLoadContext ctx)
/// <summary>
/// The names of the output and input column pairs for the transformation.
/// </summary>
public IReadOnlyCollection<(string outputColumnName, string[] inputColumnNames)> Columns
internal IReadOnlyCollection<(string outputColumnName, string[] inputColumnNames)> Columns
=> _columns.Select(col => (outputColumnName: col.Name, inputColumnNames: col.Sources.Select(source => source.name).ToArray())).ToArray().AsReadOnly();

/// <summary>
Expand Down
2 changes: 1 addition & 1 deletion src/Microsoft.ML.Data/Transforms/ColumnCopying.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ public sealed class ColumnCopyingTransformer : OneToOneTransformerBase
/// <summary>
/// Names of output and input column pairs on which the transformation is applied.
/// </summary>
public IReadOnlyCollection<(string outputColumnName, string inputColumnName)> Columns => ColumnPairs.AsReadOnly();
internal IReadOnlyCollection<(string outputColumnName, string inputColumnName)> Columns => ColumnPairs.AsReadOnly();

private static VersionInfo GetVersionInfo()
{
Expand Down
10 changes: 4 additions & 6 deletions src/Microsoft.ML.Data/Transforms/ColumnSelecting.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,7 @@
using Microsoft.ML;
using Microsoft.ML.CommandLine;
using Microsoft.ML.Data;
using Microsoft.ML.EntryPoints;
using Microsoft.ML.Internal.Utilities;
using Microsoft.ML.Model;
using Microsoft.ML.Transforms;

[assembly: LoadableClass(ColumnSelectingTransformer.Summary, typeof(IDataTransform), typeof(ColumnSelectingTransformer),
Expand Down Expand Up @@ -140,12 +138,12 @@ public sealed class ColumnSelectingTransformer : ITransformer

bool ITransformer.IsRowToRowMapper => true;

public IEnumerable<string> SelectColumns => _selectedColumns.AsReadOnly();
internal IEnumerable<string> SelectColumns => _selectedColumns.AsReadOnly();

public bool KeepColumns { get; }
internal bool KeepColumns { get; }

public bool KeepHidden { get; }
public bool IgnoreMissing { get; }
internal bool KeepHidden { get; }
internal bool IgnoreMissing { get; }
Copy link
Member

@wschin wschin Mar 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to hide them? They are some direct states of public options.

Copy link
Contributor Author

@Ivanidzo4ka Ivanidzo4ka Mar 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it just me been paranoid and I want to hide everything I can.
On other hand we need to provide this information for each transformer, like term dictionary or list of columns we been concatenating for concat, and we don't have conclusion how to do it yet.


private static VersionInfo GetVersionInfo()
{
Expand Down
16 changes: 8 additions & 8 deletions src/Microsoft.ML.Data/Transforms/ExtensionsCatalog.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,16 +97,16 @@ public static ColumnConcatenatingEstimator Concatenate(this TransformsCatalog ca
/// that you don't want to save, you can use <see cref="DropColumns"/> to remove them from the schema.
/// </remarks>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="columnsToDrop">The array of column names to drop.</param>
/// <param name="columnNames">The array of column names to drop.</param>
/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[Concat](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/DropColumns.cs)]
/// ]]>
/// </format>
/// </example>
public static ColumnSelectingEstimator DropColumns(this TransformsCatalog catalog, params string[] columnsToDrop)
=> ColumnSelectingEstimator.DropColumns(CatalogUtils.GetEnvironment(catalog), columnsToDrop);
public static ColumnSelectingEstimator DropColumns(this TransformsCatalog catalog, params string[] columnNames)
=> ColumnSelectingEstimator.DropColumns(CatalogUtils.GetEnvironment(catalog), columnNames);

/// <summary>
/// Select a list of columns to keep in a given <see cref="IDataView"/>.
Expand All @@ -121,7 +121,7 @@ public static ColumnSelectingEstimator DropColumns(this TransformsCatalog catalo
/// </format>
/// </remarks>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="keepColumns">The array of column names to keep.</param>
/// <param name="columnNames">The array of column names to keep.</param>
/// <param name="keepHidden">If <see langword="true"/> will keep hidden columns and <see langword="false"/> will remove hidden columns.</param>
/// <example>
/// <format type="text/markdown">
Expand All @@ -131,10 +131,10 @@ public static ColumnSelectingEstimator DropColumns(this TransformsCatalog catalo
/// </format>
/// </example>
public static ColumnSelectingEstimator SelectColumns(this TransformsCatalog catalog,
string[] keepColumns,
string[] columnNames,
bool keepHidden)
=> new ColumnSelectingEstimator(CatalogUtils.GetEnvironment(catalog),
keepColumns, null, keepHidden, ColumnSelectingEstimator.Defaults.IgnoreMissing);
columnNames, null, keepHidden, ColumnSelectingEstimator.Defaults.IgnoreMissing);

/// <summary>
/// Select a list of columns to keep in a given <see cref="IDataView"/>.
Expand All @@ -146,7 +146,7 @@ public static ColumnSelectingEstimator SelectColumns(this TransformsCatalog cata
/// ]]></format>
/// </remarks>
/// <param name="catalog">The transform's catalog.</param>
/// <param name="keepColumns">The array of column names to keep.</param>
/// <param name="columnNames">The array of column names to keep.</param>
/// <example>
/// <format type="text/markdown">
/// <![CDATA[
Expand All @@ -155,6 +155,6 @@ public static ColumnSelectingEstimator SelectColumns(this TransformsCatalog cata
/// </format>
/// </example>
public static ColumnSelectingEstimator SelectColumns(this TransformsCatalog catalog,
params string[] keepColumns) => catalog.SelectColumns(keepColumns, false);
params string[] columnNames) => catalog.SelectColumns(columnNames, false);
}
}