Backport PR #35898 on branch 1.1.x: CI: docker 32-bit linux build

[simonjayhawkins]

Backport PR #35898

[simonjayhawkins]

32 bit failures

=========================== short test summary info ============================
FAILED pandas/tests/indexing/test_partial.py::TestPartialSetting::test_slice_irregular_datetime_index_with_nan
FAILED pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_string_index
FAILED pandas/tests/window/test_grouper.py::TestGrouperGrouping::test_groupby_rolling_no_sort
= 3 failed, 69851 passed, 4429 skipped, 14962 deselected, 1047 xfailed, 6 xpassed, 12 warnings in 758.55s (0:12:38) =

This matches https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=46211&view=logs&j=bb1c2637-64c6-57bd-9ea6-93823b2df951&t=9c62e7ff-391e-517d-ede1-af63bc6a25da with test_groupby_nat_exclude and test_info_int_columns fixed here.

will merge this and address these failures separately

test_groupby_rolling_string_index and test_groupby_rolling_no_sort only occuring on 1.1.

test_slice_irregular_datetime_index_with_nan is failing on master and only appeared on MacPython/pandas-wheels#106 after the last commit so is recent and not picked up #35898