Skip to content

Adds support for converting function tasks #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 78 commits into from
Mar 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
997ed92
changed generated module names to snake_case
tclose Nov 6, 2023
c93ab63
reworked ci-cd workflow template
tclose Nov 21, 2023
76f0732
touched up generation of fileformats packages
tclose Nov 21, 2023
e3b75b6
added package fileformats to nipype-conv
tclose Nov 21, 2023
847632a
fixed up handling of extensions starting with digits in package gener…
tclose Nov 22, 2023
e911037
fixed up issue with lists not being given lists in tests
tclose Nov 23, 2023
f516b38
fixed issue with command not being set
tclose Nov 23, 2023
d63ce04
fixed executable detection
tclose Nov 23, 2023
a8c34d9
gave fileset samples seeds
tclose Nov 23, 2023
0ee39df
catch attribute error on executable
tclose Nov 23, 2023
91c2899
detect directory input/output types in create_packages script
tclose Jan 10, 2024
4de5a82
fixed up parsing of doctest strings split over multiple paras
tclose Jan 10, 2024
df10533
dropped python based fsl commands from list to import
tclose Jan 11, 2024
6b20bd0
dropped pass_after_timeout in favour of "with-timeout" plugin
tclose Jan 11, 2024
c8d9fca
handle classified fileformats in task modules
tclose Jan 15, 2024
7ca8dae
added all venvs to gitignore
tclose Feb 5, 2024
b419cf6
Revert "dropped python based fsl commands from list to import"
tclose Feb 5, 2024
55a9b76
Revert "removed workflow from main branch"
tclose Nov 23, 2023
c8cc5e7
Revert "deleted workflow test"
tclose Nov 23, 2023
e89299f
tidied up workflows linting
tclose Nov 23, 2023
cd73529
removed debugging line
tclose Feb 6, 2024
2f30a35
updated extras module generation template
tclose Dec 20, 2023
313c6ea
updated smriprep template
tclose Dec 20, 2023
8161493
updated test_workflow
tclose Dec 20, 2023
910a79e
cleaned up workflow conversion and added example specs for *prep work…
tclose Jan 18, 2024
fc9b21d
added 'base-package' option to create packages
tclose Jan 25, 2024
4d51d41
added list of qsiprep interfaces
tclose Jan 25, 2024
f289147
implemented function task importer
tclose Jan 25, 2024
f4cfcd6
implemented passing of inputs to methods
tclose Jan 30, 2024
7558320
split task converter into shell-command and function converters
tclose Jan 30, 2024
6098051
finished splitting task converter into function and shell command tasks
tclose Jan 30, 2024
ec1d430
added find/replace for Multi(Input|Output) and prepend specs. module …
tclose Feb 2, 2024
b85c570
fixed up recursive inclusion of local functions in function task conv…
tclose Feb 4, 2024
dda62bd
added in constants defined within the file body
tclose Feb 5, 2024
109114e
function converter now works for mapmri_reconstruction
tclose Feb 5, 2024
bd7a5c1
renamed *_pdr to pydra_*
tclose Feb 5, 2024
bd478e2
cleaned up duplicate lines introduced in rebase
tclose Feb 6, 2024
3e97cc7
cleaning up rebase, reintroduced with-classifier handling
tclose Feb 6, 2024
f8de6fa
debugged function and shell command task conversion
tclose Feb 6, 2024
3c63562
stripped out workflow code
tclose Feb 6, 2024
958f5c1
cleaned up merge artefacts in gitignore
tclose Feb 6, 2024
b9e3ee9
added example specs for all nipype internal interfaces
tclose Feb 19, 2024
bde7ddc
copied in required fileformat packages until they are available on PyPI
tclose Feb 26, 2024
d6246ca
bug fixes
tclose Feb 28, 2024
3571f80
added mriqc yaml
tclose Feb 28, 2024
d468d04
updated generate script
tclose Feb 29, 2024
d2a834f
added in pass after time out worker
tclose Mar 1, 2024
8ef6b0e
trimmed down example specifications to core packages (e.g. afni, ants…
tclose Mar 5, 2024
d586e7a
fixed up output file types
tclose Mar 6, 2024
776e46e
implemented callables porting
tclose Mar 6, 2024
9de519c
explicitly mark output files as type path, generate more callables
tclose Mar 6, 2024
697a872
moved pkg-gen into new command
tclose Mar 7, 2024
86a90b7
brought pkg-gen command inside package instead of scripts dir
tclose Mar 7, 2024
f4f383c
debugged type guessing from docstring
tclose Mar 7, 2024
23aec59
sort example spec file fields for consistent order
tclose Mar 8, 2024
0147531
chasing down nested includes
tclose Mar 8, 2024
69711ea
sorted out issue with loggers being dropped
tclose Mar 8, 2024
c9503f6
beefed up split_parens_contents to handle nested functions and quotest
tclose Mar 11, 2024
44e73e9
regenerated example specs with the latest version of the package gene…
tclose Mar 11, 2024
14dbbac
debugging pkg gen
tclose Mar 12, 2024
9765d2a
all generated interfaces pass (except for the xfails)
tclose Mar 13, 2024
4a1e434
added fileformats requirements to tests
tclose Mar 13, 2024
5d07608
all tests pass after generation
tclose Mar 14, 2024
c517d6f
reordered how callables files are written
tclose Mar 14, 2024
c29d821
updated pyproject to only include selected fileformats packagse
tclose Mar 14, 2024
4f4cb70
explicitly use python3
tclose Mar 14, 2024
f6ca0a1
added python3-packaging to gh action
tclose Mar 14, 2024
fc1a10b
added packaging dependency
tclose Mar 14, 2024
0020696
fixed up unittests
tclose Mar 15, 2024
6dc4e9f
added packaging dependency to pyproject.toml
tclose Mar 15, 2024
741e54e
combined test and publish github actions
tclose Mar 15, 2024
daa5268
added setuptools and packaging to build-system requires
tclose Mar 15, 2024
b622398
separate install of dipy
tclose Mar 15, 2024
547cd95
dropped qsiprep from tests config
tclose Mar 15, 2024
fa09577
marked EPI de warp as xfail due to 'exfdw' being marked as a string r…
tclose Mar 16, 2024
1a703a3
added sanity check to test task
tclose Mar 16, 2024
b544247
added more xfails for combined tests
tclose Mar 16, 2024
6c76ea6
added another of the mixed up interfaces
tclose Mar 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
111 changes: 111 additions & 0 deletions .github/workflows/ci-cd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: CI/CD

on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main, develop ]
release:
types: [published]

defaults:
run:
shell: bash

jobs:
test:
strategy:
matrix:
os: [macos-latest, ubuntu-latest]
python-version: ["3.8", "3.11"]
fail-fast: false
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash -l {0}

steps:
- name: Checkout
uses: actions/checkout@v2

- name: Unset header
# checkout@v2 adds a header that makes branch protection report errors
# because the Github action bot is not a collaborator on the repo
run: git config --local --unset http.https://github.com/.extraheader

- name: Fetch tags
run: git fetch --prune --unshallow

- name: Disable etelemetry
run: echo "NO_ET=TRUE" >> $GITHUB_ENV

- name: Set up Python ${{ matrix.python-version }} on ${{ matrix.os }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}

- name: Update build tools
run: python3 -m pip install --upgrade pip setuptools wheel

- name: Install required file-formats packages
run: |
pushd required-fileformats
python3 -m pip install -r requirements.txt
popd

- name: Install Dipy separately as it was causing trouble
run: python3 -m pip install dipy

- name: Install Package
run: python3 -m pip install .[test]

- name: Pytest
run: pytest -vvs --cov nipype2pydra --cov-config .coveragerc --cov-report xml

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
with:
fail_ci_if_error: true
token: ${{ secrets.CODECOV_TOKEN }}

deploy:
needs: [test]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
submodules: recursive
fetch-depth: 0

- name: Unset header
# checkout@v2 adds a header that makes branch protection report errors
# because the Github action bot is not a collaborator on the repo
run: git config --local --unset http.https://github.com/.extraheader

- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'

- name: Install build tools
run: python3 -m pip install build twine

- name: Build source and wheel distributions
run: python3 -m build .

- name: Check distributions
run: twine check dist/*

- name: Check for PyPI token on tag
id: deployable
if: github.event_name == 'release'
env:
PYPI_API_TOKEN: "${{ secrets.PYPI_API_TOKEN }}"
run: if [ -n "$PYPI_API_TOKEN" ]; then echo "DEPLOY=true" >> $GITHUB_OUTPUT; fi

- name: Upload to PyPI
if: steps.deployable.outputs.DEPLOY
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_API_TOKEN }}
36 changes: 0 additions & 36 deletions .github/workflows/publish.yml

This file was deleted.

48 changes: 0 additions & 48 deletions .github/workflows/tests.yml

This file was deleted.

6 changes: 3 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.venv*
*.venv*
/.project
/.pydevproject
*.pyc
Expand All @@ -20,6 +20,6 @@ __pycache__
*.egg-info
~*
/outputs
/Test.venv
/test-data
/nipype2pydra/_version.py
/nipype2pydra/_version.py

20 changes: 13 additions & 7 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@
import tempfile
import pytest
from click.testing import CliRunner
from fileformats.generic import File


PKG_DIR = Path(__file__).parent
EXAMPLE_SPECS_DIR = PKG_DIR / "example-specs"
EXAMPLE_TASKS_DIR = EXAMPLE_SPECS_DIR / "task"
EXAMPLE_TASKS_DIR = EXAMPLE_SPECS_DIR / "task" / "nipype"
EXAMPLE_WORKFLOWS_DIR = EXAMPLE_SPECS_DIR / "workflow"


Expand All @@ -18,11 +17,6 @@ def gen_test_conftest():
return PKG_DIR / "scripts" / "pkg_gen" / "resources" / "conftest.py"


@pytest.fixture(params=[str(p.stem) for p in (EXAMPLE_TASKS_DIR).glob("*.yaml")])
def task_spec_file(request):
return (EXAMPLE_TASKS_DIR / request.param).with_suffix(".yaml")


@pytest.fixture(params=[str(p.stem) for p in EXAMPLE_WORKFLOWS_DIR.glob("*.yaml")])
def workflow_spec_file(request):
return (EXAMPLE_WORKFLOWS_DIR / request.param).with_suffix(".yaml")
Expand All @@ -34,6 +28,13 @@ def work_dir():
return Path(work_dir)


@pytest.fixture
def outputs_dir():
outputs_dir = PKG_DIR / "outputs" / "workflows"
outputs_dir.mkdir(parents=True, exist_ok=True)
return outputs_dir


@pytest.fixture
def cli_runner(catch_cli_exceptions):
def invoke(*args, catch_exceptions=catch_cli_exceptions, **kwargs):
Expand All @@ -56,6 +57,11 @@ def pytest_exception_interact(call):
def pytest_internalerror(excinfo):
raise excinfo.value

def pytest_configure(config):
config.option.capture = 'no' # allow print statements to show up in the console
config.option.log_cli = True # show log messages in the console
config.option.log_level = "INFO" # set the log level to INFO

CATCH_CLI_EXCEPTIONS = False
else:
CATCH_CLI_EXCEPTIONS = True
Expand Down
Loading