[pandas/io/stata.py] Importing option for importing specified columns from stata .dta files

[sanguineturtle]

It would be great if a columns option could be added to the Stata Reader. Some dta files can be large and I would like to import a few specified columns. Current I am importing the entire file and returning a selection such as:

data = pd.read_stata(fn)[['year', 'exports']]

I would propose pd.read_stata(fn, columns=['year', 'exports'])

[jreback]

cc @bashtage

[jreback]

should be called names for compat with other readers

[bashtage]

This will be very difficult since dta files are stored in binary as rows and the reader sequentially reads each row, parses the binary and keeps the data. A names input would realistically only be able to drop columns once all data had been imported, and so would not be incredibly useful, since this can be trivially done from any DataFrame.

Rewriting the reader to skip columns would be a substantial undertaking, imo. This said, there may be one way to do it that wouldn't be too onerous, but I won't have any time for a while.

As a workaround I would recommend doing a once and for all conversion from a DTA to a more native pandas format (e.g. hdf) and then not worrying about it.

[bashtage]

@sanguineturtle

Recent changes to master allow 100,000 observations with 5 variables to be read in around 0.05 seconds. Hopefully this performance means reading select columns is not really a performance issue any more. See #8073 for benchmarks.

@jreback

This probably isn't remotely possible with the modifications I made to the reader since it uses np.frombuffer for performance. However, the performance improvements are large enough that this isn't an issue except possibly in memory constrained situations. Moreover, reading only a subset of columns to minimize memory impact would probably be so slow (relative to master) that it wouldn't be be useful in situations where the dataset was very big.

[jreback]

yep in fact in hdf you read entire rows with every column and then just reindex at the end
but good for now
thanks!

[sanguineturtle]

@bashtage This is great news. Thanks for letting me know. My dataset has ~27 million observations with 8 variables. The main reason I was looking for column selection is that 4 are redundant identifiers. As you have suggested I looked into HDF files and have written a conversion method (for the class) and moved the source to using HDF files (via pandas) with year keys which has proven to have very fast retrieval. I am still learning about HDF but it's in built compression delivers some pretty amazing results on file size etc.