ENH: Allow get_dummies to return SparseDataFrame

[artemyk]

For dataframes with a large number of unique values, get_dummies can use enormous amounts of memory. This provides a sparse flag to get_dummies which returns a much more memory-efficient structure.

For example:

import pandas as pd
df1 = pd.get_dummies(pd.DataFrame({'a':range(1000)}, dtype='category').a, sparse=False)
print 'dense:', df1.memory_usage().sum()
df2 = pd.get_dummies(pd.DataFrame({'a':range(1000)}, dtype='category').a, sparse=True)
print 'sparse:', df2.memory_usage().sum()
pd.util.testing.assert_frame_equal(df1, df2)

returns

dense: 8000000
sparse: 8000

Performance could probably be improved a lot.

Fails pandas/tests/test_reshape.py:TestGetDummiesSparse.test_include_na due to #8822 .

[jreback]

@artemyk can you rebase this. and you will need to compare vs Sparse (or not) in the tests.

[artemyk]

@jreback Rebased on the #8822 branch

[artemyk]

@jreback This should be ready once #8822 is merged.
Regarding tests for comparing sparse vs. not sparse --- there are a battery of tests that compare get_dummies results against 'expected' versions. Since both sparse and non-sparse versions of get_dummies get compared to the same 'expected' dataframes, doesn't this effectively test that they are returning the same things?

[artemyk]

@jreback Ready to merge?

[jreback]

put backticks around DataFrame.
say "the return DataFrame is sparse (e.g. SparseDataFrame)"

[jreback]

merged via 4673225

thanks!