fix issue with grouping with sort=True on an unordered Categorical

update categorical.rst docs

test unsortable when ordered=True

Author: jreback

doc/source/api.rst

@@ -585,6 +585,8 @@ following usable methods and properties (all available as ``Series.cat.<method_o
    Categorical.remove_categories
    Categorical.remove_unused_categories
    Categorical.set_categories
+   Categorical.as_ordered
+   Categorical.as_unordered
    Categorical.codes
 
 To create a Series of dtype ``category``, use ``cat = s.astype("category")``.

doc/source/categorical.rst

@@ -90,8 +90,6 @@ By using some special functions:
 See :ref:`documentation <reshaping.tile.cut>` for :func:`~pandas.cut`.
 
 By passing a :class:`pandas.Categorical` object to a `Series` or assigning it to a `DataFrame`.
-This is the only possibility to specify differently ordered categories (or no order at all) at
-creation time and the only reason to use :class:`pandas.Categorical` directly:
 
 .. ipython:: python
 
@@ -103,6 +101,14 @@ creation time and the only reason to use :class:`pandas.Categorical` directly:
     df["B"] = raw_cat
     df
 
+You can also specify differently ordered categories or make the resulting data ordered, by passing these arguments to ``astype()``:
+
+.. ipython:: python
+
+    s = Series(["a","b","c","a"])
+    s_cat = s.astype("category", categories=["b","c","d"], ordered=False)
+    s_cat
+
 Categorical data has a specific ``category`` :ref:`dtype <basics.dtypes>`:
 
 .. ipython:: python
@@ -176,10 +182,9 @@ It's also possible to pass in the categories in a specific order:
     s.cat.ordered
 
 .. note::
-    New categorical data is automatically ordered if the passed in values are sortable or a
-    `categories` argument is supplied. This is a difference to R's `factors`, which are unordered
-    unless explicitly told to be ordered (``ordered=TRUE``). You can of course overwrite that by
-    passing in an explicit ``ordered=False``.
+
+    New categorical data are NOT automatically ordered. You must explicity pass ``ordered=True`` to
+    indicate an ordered ``Categorical``.
 
 
 Renaming categories
@@ -270,6 +275,10 @@ Sorting and Order
 
 .. _categorical.sort:
 
+.. warning::
+
+   The default for construction has change in v0.16.0 to ``ordered=False``, from the prior implicit ``ordered=True``
+
 If categorical data is ordered (``s.cat.ordered == True``), then the order of the categories has a
 meaning and certain operations are possible. If the categorical is unordered, a `TypeError` is
 raised.
@@ -281,18 +290,26 @@ raised.
         s.sort()
     except TypeError as e:
         print("TypeError: " + str(e))
-    s = Series(["a","b","c","a"], dtype="category") # ordered per default!
+    s = Series(["a","b","c","a"]).astype('category',ordered=True)
     s.sort()
     s
     s.min(), s.max()
 
+You can set categorical data to be ordered by using ``as_ordered()`` or unordered by using ``as_unordered()``. These will by
+default return a *new* object.
+
+.. ipython:: python
+
+    s.cat.as_ordered()
+    s.cat.as_unordered()
+
 Sorting will use the order defined by categories, not any lexical order present on the data type.
 This is even true for strings and numeric data:
 
 .. ipython:: python
 
     s = Series([1,2,3,1], dtype="category")
-    s.cat.categories = [2,3,1]
+    s = s.cat.set_categories([2,3,1], ordered=True)
     s
     s.sort()
     s
@@ -310,7 +327,7 @@ necessarily make the sort order the same as the categories order.
 .. ipython:: python
 
     s = Series([1,2,3,1], dtype="category")
-    s = s.cat.reorder_categories([2,3,1])
+    s = s.cat.reorder_categories([2,3,1], ordered=True)
     s
     s.sort()
     s
@@ -339,7 +356,7 @@ The ordering of the categorical is determined by the ``categories`` of that colu
 
 .. ipython:: python
 
-   dfs = DataFrame({'A' : Categorical(list('bbeebbaa'),categories=['e','a','b']),
+   dfs = DataFrame({'A' : Categorical(list('bbeebbaa'),categories=['e','a','b'],ordered=True),
                     'B' : [1,2,1,2,2,1,2,1] })
    dfs.sort(['A','B'])
 
@@ -664,9 +681,6 @@ The following differences to R's factor functions can be observed:
 
 * R's `levels` are named `categories`
 * R's `levels` are always of type string, while `categories` in pandas can be of any dtype.
-* New categorical data is automatically ordered if the passed in values are sortable or a
-  `categories` argument is supplied. This is a difference to R's `factors`, which are unordered
-  unless explicitly told to be ordered (``ordered=TRUE``).
 * It's not possible to specify labels at creation time. Use ``s.cat.rename_categories(new_labels)``
   afterwards.
 * In contrast to R's `factor` function, using categorical data as the sole input to create a

doc/source/whatsnew/v0.16.0.txt

@@ -312,51 +312,54 @@ Categorical Changes
 
 .. _whatsnew_0160.api_breaking.categorical:
 
-In prior versions, Categoricals that with had an unspecified ordering (meaning no ``ordered`` keyword was passed) were defaulted to have a lexographic ordering of the values and designated as ``ordered`` Categoricals. Going forward, the ``ordered`` keyword in the ``Categorical`` constructor will default to ``False``, so ordering must now be explicit.
+In prior versions, ``Categoricals`` that had an unspecified ordering (meaning no ``ordered`` keyword was passed) were defaulted as ``ordered`` Categoricals. Going forward, the ``ordered`` keyword in the ``Categorical`` constructor will default to ``False``, ordering must now be explicit.
 
-Furthermore, previously you *could* change the ``ordered`` attribute of a Categorical by just setting the attribute, e.g. ``cat.ordered=True``; This is now deprecated and you should use ``cat.set_ordered(True)``. This will by default return a **new** object and not modify the existing object.
+Furthermore, previously you *could* change the ``ordered`` attribute of a Categorical by just setting the attribute, e.g. ``cat.ordered=True``; This is now deprecated and you should use ``cat.as_ordered()`` or ``cat.as_unordered()``. These will by default return a **new** object and not modify the existing object. (:issue:`9347`, :issue:`9190`)
 
 Previous Behavior
 
 .. code-block:: python
 
-   In [1]: cat = pd.Categorical([0,1,2])
+   In [3]: s = Series([0,1,2], dtype='category')
 
-   In [2]: cat
-   Out[2]:
-   [0, 1, 2]
+   In [4]: s
+   Out[4]:
+   0    0
+   1    1
+   2    2
+   dtype: category
    Categories (3, int64): [0 < 1 < 2]
 
-   In [3]: cat.ordered
-   Out[3]: True
+   In [5]: s.cat.ordered
+   Out[5]: True
 
-   In [4]: cat.ordered=False
+   In [6]: s.cat.ordered = False
 
-   In [5]: cat
-   Out[5]:
-   [0, 1, 2]
+   In [7]: s
+   Out[7]:
+   0    0
+   1    1
+   2    2
+   dtype: category
    Categories (3, int64): [0, 1, 2]
 
-
 New Behavior
 
 .. ipython:: python
 
-   cat = pd.Categorical([0,1,2])
-   cat
-   cat.ordered
-   cat.ordered=True
-   cat = cat.set_ordered(True)
-   cat
-   cat.ordered
+   s = Series([0,1,2], dtype='category')
+   s
+   s.cat.ordered
+   s = s.cat.as_ordered()
+   s
+   s.cat.ordered
 
-   # you can set in the construtor
-   cat = pd.Categorical([0,1,2],ordered=True)
-   cat
-   cat.ordered
+   # you can set in the constructor of the Categorical
+   s = Series(Categorical([0,1,2],ordered=True))
+   s
+   s.cat.ordered
 
-For ease of creation of series of ``Categoricals``, we have added the ability to pass keywords when calling ``.astype()``, these
-are passed directly to the constructor.
+For ease of creation of series of categorical data, we have added the ability to pass keywords when calling ``.astype()``, these are passed directly to the constructor.
 
 .. ipython:: python
 
@@ -365,6 +368,27 @@ are passed directly to the constructor.
    s = Series(["a","b","c","a"]).astype('category',categories=list('abcdef'),ordered=False)
    s
 
+.. warning::
+
+   This simple API change may have suprising effects if a user is relying on the previous defaulted behavior implicity. In particular,
+   sorting operations with a ``Categorical`` will now raise an error:
+
+   .. code-block:: python
+
+      In [1]: df = DataFrame({ 'A' : Series(list('aabc')).astype('category'), 'B' : np.arange(4) })
+
+      In [2]: df['A'].order()
+      TypeError: Categorical not ordered
+      you can use .as_ordered() to change the Categorical to an ordered one
+
+      In [3]: df.groupby('A').sum()
+      ValueError: cannot sort by an unordered Categorical in the grouper
+      you can set sort=False in the groupby expression or
+      make the categorical ordered by using .as_ordered()
+
+   The solution is to make 'A' orderable, e.g. ``df['A'] = df['A'].cat.as_ordered()``
+
+
 Indexing Changes
 ~~~~~~~~~~~~~~~~
 

pandas/core/categorical.py

@@ -139,7 +139,7 @@ class Categorical(PandasObject):
     categories : Index-like (unique), optional
         The unique categories for this categorical. If not given, the categories are assumed
         to be the unique values of values.
-    ordered : boolean, optional
+    ordered : boolean, (default False)
         Whether or not this categorical is treated as a ordered categorical. If not given,
         the resulting categorical will not be ordered.
     name : str, optional
@@ -259,12 +259,14 @@ def __init__(self, values, categories=None, ordered=False, name=None, fastpath=F
 
         if categories is None:
             try:
-                codes, categories = factorize(values, sort=ordered)
+                codes, categories = factorize(values, sort=True)
             except TypeError:
-                # raise, as we don't have a sortable data structure and so the user should
-                # give us one by specifying categories
-                raise TypeError("'values' is not factorizable, please pass "
-                                "categories order by passing in a categories argument.")
+                codes, categories = factorize(values, sort=False)
+                if ordered:
+                    # raise, as we don't have a sortable data structure and so the user should
+                    # give us one by specifying categories
+                    raise TypeError("'values' is not ordered, please explicitly specify the "
+                                    "categories order by passing in a categories argument.")
             except ValueError:
 
                 ### FIXME ####
@@ -290,7 +292,7 @@ def __init__(self, values, categories=None, ordered=False, name=None, fastpath=F
                 warn("None of the categories were found in values. Did you mean to use\n"
                      "'Categorical.from_codes(codes, categories)'?", RuntimeWarning)
 
-        self._ordered = ordered
+        self.set_ordered(ordered, inplace=True)
         self.categories = categories
         self.name = name
         self._codes = _coerce_indexer_dtype(codes, categories)
@@ -345,7 +347,7 @@ def from_codes(cls, codes, categories, ordered=False, name=None):
             An integer array, where each integer points to a category in categories or -1 for NaN
         categories : index-like
             The categories for the categorical. Items need to be unique.
-        ordered : boolean, optional
+        ordered : boolean, (default False)
             Whether or not this categorical is treated as a ordered categorical. If not given,
             the resulting categorical will be unordered.
         name : str, optional
@@ -470,6 +472,30 @@ def set_ordered(self, value, inplace=False):
         if not inplace:
             return cat
 
+    def as_ordered(self, inplace=False):
+        """
+        Sets the Categorical to be ordered
+
+        Parameters
+        ----------
+        inplace : boolean (default: False)
+           Whether or not to set the ordered attribute inplace or return a copy of this categorical
+           with ordered set to True
+        """
+        return self.set_ordered(True, inplace=inplace)
+
+    def as_unordered(self, inplace=False):
+        """
+        Sets the Categorical to be unordered
+
+        Parameters
+        ----------
+        inplace : boolean (default: False)
+           Whether or not to set the ordered attribute inplace or return a copy of this categorical
+           with ordered set to False
+        """
+        return self.set_ordered(False, inplace=inplace)
+
     def _get_ordered(self):
         """ Gets the ordered attribute """
         return self._ordered
@@ -853,7 +879,8 @@ def searchsorted(self, v, side='left', sorter=None):
         array([3, 5])       # eggs after donuts, after switching milk and donuts
         """
         if not self.ordered:
-            raise ValueError("searchsorted requires an ordered Categorical.")
+            raise ValueError("Categorical not ordered\n"
+                             "you can use .as_ordered() to change the Categorical to an ordered one\n")
 
         from pandas.core.series import Series
         values_as_codes = self.categories.values.searchsorted(Series(v).values, side)
@@ -981,7 +1008,8 @@ def argsort(self, ascending=True, **kwargs):
         argsorted : numpy array
         """
         if not self.ordered:
-            raise TypeError("Categorical not ordered")
+            raise TypeError("Categorical not ordered\n"
+                            "you can use .as_ordered() to change the Categorical to an ordered one\n")
         result = np.argsort(self._codes.copy(), **kwargs)
         if not ascending:
             result = result[::-1]
@@ -1013,7 +1041,8 @@ def order(self, inplace=False, ascending=True, na_position='last'):
         Category.sort
         """
         if not self.ordered:
-            raise TypeError("Categorical not ordered")
+            raise TypeError("Categorical not ordered\n"
+                            "you can use .as_ordered() to change the Categorical to an ordered one\n")
         if na_position not in ['last','first']:
             raise ValueError('invalid na_position: {!r}'.format(na_position))
 
@@ -1394,7 +1423,8 @@ def min(self, numeric_only=None, **kwargs):
         min : the minimum of this `Categorical`
         """
         if not self.ordered:
-            raise TypeError("Categorical not ordered")
+            raise TypeError("Categorical not ordered\n"
+                            "you can use .as_ordered() to change the Categorical to an ordered one\n")
         if numeric_only:
             good = self._codes != -1
             pointer = self._codes[good].min(**kwargs)
@@ -1421,7 +1451,8 @@ def max(self, numeric_only=None, **kwargs):
         max : the maximum of this `Categorical`
         """
         if not self.ordered:
-            raise TypeError("Categorical not ordered")
+            raise TypeError("Categorical not ordered\n"
+                            "you can use .as_ordered() to change the Categorical to an ordered one\n")
         if numeric_only:
             good = self._codes != -1
             pointer = self._codes[good].max(**kwargs)
@@ -1524,7 +1555,8 @@ class CategoricalAccessor(PandasDelegate):
     >>> s.cat.remove_categories(['d'])
     >>> s.cat.remove_unused_categories()
     >>> s.cat.set_categories(list('abcde'))
-    >>> s.cat.set_ordered(True)
+    >>> s.cat.as_ordered()
+    >>> s.cat.as_unordered()
 
     """
 
@@ -1561,7 +1593,8 @@ def _delegate_method(self, name, *args, **kwargs):
                                                        "remove_categories",
                                                        "remove_unused_categories",
                                                        "set_categories",
-                                                       "set_ordered"],
+                                                       "as_ordered",
+                                                       "as_unordered"],
                                             typ='method')
 
 ##### utility routines #####

pandas/core/groupby.py

@@ -1923,8 +1923,16 @@ def __init__(self, index, grouper=None, obj=None, name=None, level=None,
 
             # a passed Categorical
             elif isinstance(self.grouper, Categorical):
+
+                # must have an ordered categorical
+                if self.sort:
+                    if not self.grouper.ordered:
+                        raise ValueError("cannot sort by an unordered Categorical in the grouper\n"
+                                         "you can set sort=False in the groupby expression or\n"
+                                         "make the categorical ordered by using .set_ordered(True)\n")
+
                 # fix bug #GH8868 sort=False being ignored in categorical groupby
-                if not self.sort:
+                else:
                     self.grouper = self.grouper.reorder_categories(self.grouper.unique())
                 self._labels = self.grouper.codes
                 self._group_index = self.grouper.categories