Skip to content

CI: Don't run xdist if pytest.mark.single_cpu #46014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed

CI: Don't run xdist if pytest.mark.single_cpu #46014

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
22d91ac
CI: Don't run xdist if pytest.mark.single_cpu
mroeschke Feb 16, 2022
62b6d32
Merge remote-tracking branch 'upstream/main' into ci/single_followup
mroeschke Feb 16, 2022
331312e
Fix condidition
mroeschke Feb 16, 2022
84ace2d
Merge remote-tracking branch 'upstream/main' into ci/single_followup
mroeschke Feb 16, 2022
7e4ec61
Merge remote-tracking branch 'upstream/main' into ci/single_followup
mroeschke Feb 16, 2022
6e26bd3
Merge remote-tracking branch 'upstream/main' into ci/single_followup
mroeschke Feb 17, 2022
c010a0e
Turn into a debug PR
mroeschke Feb 18, 2022
540525a
Try seeing what pyarrow version we get if no mamba
mroeschke Feb 18, 2022
ddaee2a
Trigger CI
mroeschke Feb 18, 2022
b2fd1d0
reset to mamba to get v2, and remove setting cpu
mroeschke Feb 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
resource_class: arm.medium
environment:
ENV_FILE: ci/deps/circle-38-arm64.yaml
PYTEST_WORKERS: auto
TEST_ARGS: "--numprocesses auto --dist=loadfile"
PATTERN: "not slow and not network and not clipboard and not arm_slow"
PYTEST_TARGET: "pandas"
steps:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/datamanger.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
env:
PANDAS_DATA_MANAGER: array
PATTERN: "not network and not clipboard and not single_cpu"
PYTEST_WORKERS: "auto"
TEST_ARGS: "--numprocesses auto --dist=loadfile"
PYTEST_TARGET: pandas
run: |
source activate pandas-dev
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/posix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ jobs:
LANG: ${{ matrix.lang || '' }}
LC_ALL: ${{ matrix.lc_all || '' }}
PANDAS_TESTING_MODE: ${{ matrix.pandas_testing_mode || '' }}
TEST_ARGS: ${{ matrix.test_args || '' }}
PYTEST_WORKERS: ${{ contains(matrix.pattern, 'not single_cpu') && 'auto' || '1' }}
# Don't activate pytest-xdist (-n) if signle_cpu build
TEST_ARGS: ${{ format('{0} {1}', matrix.test_args || '', contains(matrix.pattern, 'not single_cpu') && '--numprocesses auto --dist=loadfile' || '') }}
PYTEST_TARGET: ${{ matrix.pytest_target || 'pandas' }}
IS_PYPY: ${{ contains(matrix.env_file, 'pypy') }}
# TODO: re-enable coverage on pypy, its slow
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python-dev.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ on:
- "doc/**"

env:
PYTEST_WORKERS: "auto"
TEST_ARGS: "--numprocesses auto --dist=loadfile"
PANDAS_CI: 1
PATTERN: "not slow and not network and not clipboard and not single_cpu"
COVERAGE: true
Expand Down
2 changes: 1 addition & 1 deletion azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ pr:
- 1.4.x

variables:
PYTEST_WORKERS: auto
TEST_ARGS: "--numprocesses auto --dist=loadfile"
PYTEST_TARGET: pandas
PATTERN: "not slow and not high_memory and not db and not network and not single_cpu"
PANDAS_CI: 1
Expand Down
4 changes: 2 additions & 2 deletions ci/run_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ if [[ $(uname) == "Linux" && -z $DISPLAY ]]; then
XVFB="xvfb-run "
fi

PYTEST_CMD="${XVFB}pytest -r fEs -n $PYTEST_WORKERS --dist=loadfile $TEST_ARGS $COVERAGE $PYTEST_TARGET"
PYTEST_CMD="${XVFB}pytest -v -r fEs $TEST_ARGS $COVERAGE $PYTEST_TARGET"

if [[ "$PATTERN" ]]; then
PYTEST_CMD="$PYTEST_CMD -m \"$PATTERN\""
Expand All @@ -39,7 +39,7 @@ sh -c "$PYTEST_CMD"

if [[ "$PANDAS_DATA_MANAGER" != "array" ]]; then
# The ArrayManager tests should have already been run by PYTEST_CMD if PANDAS_DATA_MANAGER was already set to array
PYTEST_AM_CMD="PANDAS_DATA_MANAGER=array pytest -n $PYTEST_WORKERS --dist=loadfile $TEST_ARGS $COVERAGE pandas"
PYTEST_AM_CMD="PANDAS_DATA_MANAGER=array pytest -r fEs $TEST_ARGS $COVERAGE $PYTEST_TARGET"

if [[ "$PATTERN" ]]; then
PYTEST_AM_CMD="$PYTEST_AM_CMD -m \"$PATTERN and arraymanager\""
Expand Down
3 changes: 0 additions & 3 deletions pandas/tests/io/parser/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,9 +110,6 @@ def all_parsers(request):
pytest.importorskip("pyarrow", VERSIONS["pyarrow"])
# Try finding a way to disable threads all together
# for more stable CI runs
import pyarrow

pyarrow.set_cpu_count(1)
return parser


Expand Down