ERR: dont' allow ambiguous usecols

[jreback]

#12551 and comment: #12512 (comment)

usecols=['a',1] should raise

[gfyoung]

@jreback : I'll tackle this one as a follow-up in terms of fixing documentation and enforcement. In the meantime, I'll remove the tests I added that had the ambiguous columns.

[jorisvandenbossche]

Shouldn't we also just raise on duplicates in usecols ? eg ['a', 'a']

[gfyoung]

@jorisvandenbossche : usecols is a set, so you won't ever have such a problem.

[jreback]

#11822 and #11823 converts it to a list.

though if we allow only non-duplicated and make it a set operation against names...

though current doc-string say its array-like

[jorisvandenbossche]

@gfyoung aha, I didn't know that, but then adding tests for the behaviour for duplicates values in usecols as we are doing in #11882 makes no sense?

[jreback]

what we could do is:

• don't allow mixed-integers (e.g. 'a', 1) as these are ambiguous
• still use a set for usecols, so duplicates are gone, but handle this as a set-selection operation against .names/columns (IOW if names is passed or a header is read in). If there are duplicates there its ok, this usecols just sub-selects. Duplicate handling will be solely there.

[gfyoung]

@jreback : What do you mean by "duplicate" handling? Is that what #11823 will be doing?

[jreback]

This is on master.

In [12]: pd.read_csv(StringIO("""1,2,3"""), engine='c', header=None, 
   ....:             names=['a', 'b', 'a'], usecols=['a','a'])
Out[12]: 
   a  a
0  1  1

In [13]: pd.read_csv(StringIO("""1,2,3"""), engine='python', header=None, 
            names=['a', 'b', 'a'], usecols=['a','a'])
Out[13]: 
   a
0  1

but I think that these should BOTH output

   a   a
0  1   3

IOW, the usecols is just a filter as a set (so its 'a' in this case).

Then names takes over and you get the 0th and 2nd columns (that are named 'a')

[jreback]

cc @sxwang

[gfyoung]

@jreback : AFAICT, such behaviour will be fixed in #11882 right?

[jorisvandenbossche]

@jreback I agree

@gfyoung Well the PR for that issue is #11882, and currently there it is not yet this behaviour (but still in reviewing phase)