Read SAS (sas7bdat)

[ywhcuhk]

Very excited to have this new feature in Pandas. I have a few comments to share:

1. pd.read_sas() doesn't read SAS date variable correctly (this is noted in the doc). Dates are read as numpy.float64. Note in SAS, dates are recorded as numbers relative to 1960-1-1. It would be helpful to allow some sort of arguments to parse the date variable correctly.
2. Moreover, SAS has some special missing variables such as .B or .R. I wonder how are these cases treated?
3. Not nearly as fast as read_csv(). To read a 700MB SAS data. The time is

CPU times: user 1min 47s, sys: 955 ms, total: 1min 48s
Wall time: 1min 48s

The time for the same CSV file (I covered the same file to CSV using SAS) is

CPU times: user 3.93 s, sys: 343 ms, total: 4.28 s
Wall time: 4.29 s

[benjello]

To improve performance, the path to follow might be the one indicated in #10517.

[jreback]

cc @kshedden

this is mostly in python ATM. You always write for correctness first, profile, then if necessary use things like cython in critical sections. pull-requests are welcome for an improved implm.

[kshedden]

Thanks for the feedback. A few comments:

• Performance is a major issue. The pandas version is currently not usable for my own use case (files with billions of rows). FWIW I have a golang version that is about 20x faster (https://github.com/kshedden/datareader) which I use for big files. Most of the slowness in pandas.read_sas is in process_byte_array_with_data (already in cython but probably not optimally set up). I have a local version with improvements that is about 30% faster, but I was looking for a much bigger improvement.
• The other open sourced SAS7BDAT readers that I know of , including the one from wizard, don't support compression properly or at all, e.g.: Support binary compression in sas7bdat WizardMac/ReadStat#21 I put a lot of time into getting the compression support to work.
• Date support needs some work, but this should be relatively easy to fix. There are several different date formats in SAS. We detect and autoconvert some but not all of them.
• I don't have documentation for missing value codes, if you can point me to it (i.e. which float values correspond to which codes) I should be able to add it.
• I just noticed that wizard supports sas7bcat (categorical data format codes), I will look into porting this over at some point.

[marks]

@kshedden any update on sas7bcat support by any chance? Thanks!!

[xappppp]

Is there any updated on sas7bdat? I have a 5GB sas data to read into python. But the performance is still a hassle. Thanks!

[ofajardo]

@marks @xappppp @kshedden I have released a wrapper around the ReadStat C library called pyreadstat. Because most of the code is C is faster than pandas. It can also read sas7bcat files. It handles value labels and column labels. Missing values tags will come soon (@kshedden it would be nice if you could provide a sample file with tagged missing values).
https://github.com/Roche/pyreadstat