Column name indexing removed in .4?

[JamesAlexander42]

Why was indexing into a DataFrame removed in latest? Looking at the commit history too I can see the block was deleted. This makes very an awkward use experience now.

[pgovind]

Hello! Which API are you talking about? I still see this:

corefxlab/src/Microsoft.Data.Analysis/DataFrame.cs

Line 72 in f44a099

public object this[long rowIndex, int columnIndex]

[JamesAlexander42]

I referred above the change I'm referring to

[pgovind]

Ah I see. It still exists! We just moved that to the DataFrameColumnCollection class. See this for an example:

corefxlab/tests/Microsoft.Data.Analysis.Tests/DataFrameTests.cs

Line 648 in f44a099

df.Columns["Int3"] = df.Columns["Int1"] * 2 + df.Columns["Int2"];

[rhysparry]

I think @hamaze was specifically asking why the API was changed that way. Maybe you can point to the API review?

I know that I'd rather write:

df["Int3"] = df["Int1"] * 2 + df["Int2"];

As opposed to:

df.Columns["Int3"] = df.Columns["Int1"] * 2 + df.Columns["Int2"]; 

[JamesAlexander42]

Yeah, the former is more pandas-esque and comfortable IMO.

[MikaelUmaN]

Agreed.

I suspect the reason is for the row filter to work.

But you are more often interested in column selection than row selection so it's better to have the penalty there instead.

df.Rows[2 .. 12]

df["Int3"] = df["Int1"] * 2 + df["Int2"];

[pgovind]

Just out of curiosity, are you using DataFrame in a notebook? Reason I ask is that we've worked on a cool extension for DataFrame in notebooks that'll let you write df.Int1 * 2 + df.Int2. To be specific, with the new extension you can now refer to a column as a field of a DataFrame object. With intellisense enabled in notebooks, this will be very discoverable.

[JamesAlexander42]

I'm not using it in a notebook context for this exercise. Using it in an asp.net app.

[MikaelUmaN]

Haven't tested the new notebook support yet but will do.

I would say that even though notebooks are very useful, I much prefer the experience to be the same when doing normal software and when doing notebooks.

Usually I prototype in notebooks and then structure and copy stuff to some kind of software that is more production-like. So I would avoid using any extensions in notebook except for ones that are interactive such as plotting.

[zyzhu]

I concur with @MikaelUmaN.

For instance, I would expect code that runs in F# kernel notebook can be run in FSI under Visual Studio directly. I would also expect it to be compiled to be part of a bigger production system mixed with C# and F#. That's how I explore my problems in Ifsharp notebook and put them in production all the time.

However, if syntax involving dataframe relies on an extra notebook extension that only works in notebook, the beauty of production-ready scripts is no longer feasible.

cc @cartermp @dsyme to chime in.

[pgovind]

Just tagging @eerhardt for visibility here. This is great feedback! We're busy helping out with .NET 5 stuff this week, but I'll revisit this next week. There's enough support here to consider bringing back the column name indexer on DataFrame.

[eerhardt]

There's enough support here to consider bringing back the column name indexer on DataFrame.

I agree. Personally I like the ease of use of df["Int1"] as well, so I'm glad I'm not alone.

It should be pretty easy to add the API back as a wrapper over the .Columns[string] indexer, and a test or two. Anyone want to make a PR for that?

[dsyme]

Just out of curiosity, are you using DataFrame in a notebook? Reason I ask is that we've worked on a cool extension for DataFrame in notebooks that'll let you write df.Int1 * 2 + df.Int2. To be specific, with the new extension you can now refer to a column as a field of a DataFrame object. With intellisense enabled in notebooks, this will be very discoverable.

However, if syntax involving dataframe relies on an extra notebook extension that only works in notebook, the beauty of production-ready scripts is no longer feasible.

Yes, we need to be very careful about promoting non-standard extensions to the programming model for C# or F# which are only deployed only through select channels. Notebook programming should ideally not be using variations of these programming languages, though these things are subtle

This is a tricky area because there is a notable tendency to use the incremantal-dynamicity of notebook programming

@pgovind What APIs are you using to craft this language variation? Please discuss this with @MadsTorgersen, @jaredpar and myself. We can't have random variations on C# and F# floating around that fragment the overall programming experience.