From f2095e28038f3929a9cd3b1cc7537f4464e58cdd Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" <1330696+mr-c@users.noreply.github.com> Date: Mon, 4 Oct 2021 17:30:23 +0200 Subject: [PATCH 1/8] Document how to write a new conformance test and better explain the format of that file and better how to invoke ./run-tests.sh --- CONFORMANCE_TESTS.md | 136 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 127 insertions(+), 9 deletions(-) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index 7920ed8b..a693a6e5 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -15,28 +15,75 @@ All tests passed ## Options -RUNNER=other-cwl-runner +`RUNNER=other-cwl-runner` The CWL implementation to be tested. --nN +### Test Selection -Run a single specific test number N. +`--tags` -EXTRA=--parallel +A comma separated list of [tags](#tags-for-conformance-tests); only tests with these tags will be tested. +`--tags shell_command` will run all tests with `shell_command` in their `tags` list. -Extra options to pass to the CWL runner +`-n{test_range}` -For example, to run conformance test 15 against the "cwltool" -reference implementation with `--parallel`: +Run only the specific test numbers. `{test_range}` is a comma separated list of +single test numbers and/or numeric ranges. +`-n5-7,15` == only runs the 5th, 6th, 7th, and 15th tests. + +`-N{test_range}` + +Like the lowercase `-n` option, except that the specified tests will not be run. +Can be mixed with the other test selectors: `-n5-7,15 -N6` == only runs the 5th, 7th, and 15th tests, skipping the 6th test. + +`-s{test_names}` + +Run the specific tests according to their `label`s. `{test_names}` is a comma separated list of +test labels. `-scl_optional_bindings_provided,stdout_redirect_docker,expression_any_null` +achieves the same effect as `-n5-7,15 -N6` and it will still work if the tests are re-ordered. + +`-S{test_names}` + +Excludes specific tests according to their `label`s. `{test_names}` is a comma separated list of +test labels. `--tags shell_command -Sstderr_redirect_shortcut` will run all test with `expression_tool` +in their `tags` list except the test with the label `stderr_redirect_shortcut`. + +### Misc + +`EXTRA="--parallel --singularity"` + +Extra options to pass to the CWL runner (check your runner for exact options) + +`-j=4` + +The number of different tests to run at the same time + +`--junit-xml=FILENAME` + +Store results in JUnit XML format using the given FILENAME. + +`--classname=CLASSNAME` + +In the JUnit XML, tag the results with the given CLASSNAME. +Can be useful when multiple test runs are combined to document the differences in `EXTRA=`. + +--- + +For example, to run conformance test 15,16,17, and 20 against the "cwltool" +reference implementation using the `podman` container engine +and parallel execution of workflow steps ```bash -$ ./run_test.sh RUNNER=cwltool -n15 EXTRA=--parallel +$ ./run_test.sh RUNNER=cwltool -n15-17,20 EXTRA="--parallel --podman" Test [15/49] +Test [16/49] +Test [17/49] +Test [20/49] All tests passed ``` -## Notes +## OS X / macOS Notes _NOTE_: For running on OSX systems, you'll need to install coreutils via brew. This will add to your system some needed GNU-like tools like `greadlink`. @@ -44,6 +91,77 @@ system some needed GNU-like tools like `greadlink`. 1. If you haven't already, install [brew](http://brew.sh/) package manager in your mac 2. Run `brew install coreutils` +## Format of the conformance test file + +Conformance tests consist of input CWL descriptions plus an input CWL object +and the expected outputs (or `should_fail: true` if the test is deliberately broken) + +They are stored in [`conformance_tests.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/conformance_tests.yaml) +(or a file `$import`ed into that one) + +You can examine [the formal schema of this file](https://github.com/common-workflow-language/cwltest/blob/main/cwltest/cwltest-schema.yml), +or just continue reading here for an explanation. + +The conformance test file is a YAML document: a list of key-value pairs. + +We will use this single entry to explain the format +``` yaml +- doc: Test command line with optional input (missing) + label: cl_optional_inputs_missing + tool: tests/cat1-testcli.cwl + job: tests/cat-job.json + output: + args: [cat, hello.txt] + tags: [ required, command_line_tool ] +``` +- `doc`: A sentence that explain what is being tested. Will be printed at test execution time. Should be unique. +- `label`: a short list of underscore (`_`) separated words that succinctly identifies and explains the test. +- `tool` the path to the CWL description to run +- `job`: the CWL input object in YAML/JSON format. If there are no inputs then use `tests/empty.json`. +- `output` [the CWL output object expected.](#output-matching) +- `tags`: a yaml list of tag names, see [the list of canonical tags below](#tags-for-conformance-tests). + +Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into separate files. + +Currently, the main file is too big (over 3400 lines); we are slowly re-organizing it. + +Eventually it would be good to organize the tests so that the test for each optional feature and other logical groups of tests are in their own separate file; +with the supporting CWL documents and their inputs in separate sub-folders of `tests` as well. + +Example: [`- $import: tests/string-interpolation/test-index.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/5f27e234b4ca88ed1280dedf9e3391a01de12912/conformance_tests.yaml#L3395) +adds all the entries in [`tests/string-interpolation/test-index.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/tests/string-interpolation/test-index.yaml) +as entries in the main conformance test file. + +## Output matching + +In each test entry there is an `output` field that contains a mapping of the expected outputs names and their values. + +If a particular value could vary and it doesn't matter to the proper functioning of the test, then it can be represented by the special token `Any`. + +At any level, if there is an extra field, then that will be considered an error. +An exception to this is `class: File` and `class: Directory` objects, the `cwl-runner` under test can add additional fields here. +Likewise, if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it. + +[According to the CWL standards](https://www.commonwl.org/v1.2/CommandLineTool.html#File), the format of the `location` field in +`class: File` and `class: Directory` is implementation specific and we should not be testing them. +Please remember to use `location: any` for them. + +Currently, we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930) +If you are editing those old tests, you may be interested in some special processing for `class: File` and `class: Directory` output objects: +any `location` value specified will succeed if there is either an exact match to the real output, or it matches the end of the real output. +Additionally, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison. + +## Writing a new conformance test + +To add a new conformance test: +1. Ensure the CWL description you have tests the desired feature or aspect. +2. Run your test using the CWL reference runner (`cwltool`) or another CWL runner + that shows the correct behavior to collect the output, or confirm that validation/execution fails as expected +3. Add the CWL description and output object to the subdirectory `tests` in this repository. +4. Fill out a new entry in [conformance_tests.yaml](conformance_tests.yaml) following the [format of the conformance test file](#format-of-the-conformance-test-file) +5. Send a pull request to [current staging branch for the next revision of the CWL standards](https://github.com/common-workflow-language/cwl-v1.2/tree/1.2.1_proposed) + with your changes + ## Tags for conformance tests Each test in the [conformance_tests.yaml](conformance_tests.yaml) should be tagged with one or more tags. From be84ca8ade8730bbeb74a1e563347ea71e0bad3e Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" <1330696+mr-c@users.noreply.github.com> Date: Tue, 5 Oct 2021 15:31:59 +0200 Subject: [PATCH 2/8] Add cwltest installation instructions No conda instructions for now, that package is out of date: https://github.com/bioconda/bioconda-recipes/pull/30871 --- CONFORMANCE_TESTS.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index a693a6e5..5313f619 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -3,6 +3,18 @@ The conformance tests are intended to test feature coverage of a CWL implementation. It uses the module cwltest from https://github.com/common-workflow-language/cwltest/. +## Pre-requisites + +You will need both the `cwltest` Python package and the CWL runner you would like to test installed. + +Installing the `cwltest` Python package using a virtualenv: + +``` +$ python3 -m venv cwltest_env +$ source cwltest_env/bin/activate +$ pip install cwltest +``` + ## Usage ```bash From dcb7a0e6133be9350f136697345a6ff907117051 Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" <1330696+mr-c@users.noreply.github.com> Date: Tue, 5 Oct 2021 17:39:48 +0200 Subject: [PATCH 3/8] harmonize references to "CWL document(s)" and other cleanups --- CONFORMANCE_TESTS.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index 5313f619..0ec8cc99 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -105,7 +105,7 @@ system some needed GNU-like tools like `greadlink`. ## Format of the conformance test file -Conformance tests consist of input CWL descriptions plus an input CWL object +A single conformance test consist of the path to an input CWL document plus an input CWL object and the expected outputs (or `should_fail: true` if the test is deliberately broken) They are stored in [`conformance_tests.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/conformance_tests.yaml) @@ -128,12 +128,13 @@ We will use this single entry to explain the format ``` - `doc`: A sentence that explain what is being tested. Will be printed at test execution time. Should be unique. - `label`: a short list of underscore (`_`) separated words that succinctly identifies and explains the test. -- `tool` the path to the CWL description to run +- `tool` the path to the CWL document to run - `job`: the CWL input object in YAML/JSON format. If there are no inputs then use `tests/empty.json`. - `output` [the CWL output object expected.](#output-matching) - `tags`: a yaml list of tag names, see [the list of canonical tags below](#tags-for-conformance-tests). -Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into separate files. +Because `conformance_tests.yaml` is a `schema-salad` processed document, [`$import`](https://www.commonwl.org/v1.2/SchemaSalad.html#Import) +can be used to organize the tests into separate files. Currently, the main file is too big (over 3400 lines); we are slowly re-organizing it. @@ -151,7 +152,7 @@ In each test entry there is an `output` field that contains a mapping of the exp If a particular value could vary and it doesn't matter to the proper functioning of the test, then it can be represented by the special token `Any`. At any level, if there is an extra field, then that will be considered an error. -An exception to this is `class: File` and `class: Directory` objects, the `cwl-runner` under test can add additional fields here. +An exception to this is `class: File` and `class: Directory` objects, the `cwl-runner` under test can add additional fields here without causing a test to fail. Likewise, if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it. [According to the CWL standards](https://www.commonwl.org/v1.2/CommandLineTool.html#File), the format of the `location` field in @@ -166,10 +167,10 @@ Additionally, for `class: Directory` the location reported by the actual executi ## Writing a new conformance test To add a new conformance test: -1. Ensure the CWL description you have tests the desired feature or aspect. +1. Ensure the CWL document you have tests the desired feature or aspect. 2. Run your test using the CWL reference runner (`cwltool`) or another CWL runner that shows the correct behavior to collect the output, or confirm that validation/execution fails as expected -3. Add the CWL description and output object to the subdirectory `tests` in this repository. +3. Add the CWL document and output object to the subdirectory `tests` in this repository. 4. Fill out a new entry in [conformance_tests.yaml](conformance_tests.yaml) following the [format of the conformance test file](#format-of-the-conformance-test-file) 5. Send a pull request to [current staging branch for the next revision of the CWL standards](https://github.com/common-workflow-language/cwl-v1.2/tree/1.2.1_proposed) with your changes From a6854aa9d455acee8965d3d1d6cfee737988bb6b Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" <1330696+mr-c@users.noreply.github.com> Date: Tue, 5 Oct 2021 18:02:39 +0200 Subject: [PATCH 4/8] add conda install --- CONFORMANCE_TESTS.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index 0ec8cc99..68cf90ce 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -15,6 +15,12 @@ $ source cwltest_env/bin/activate $ pip install cwltest ``` +or via `bioconda` + +``` +$ conda install -c bioconda cwltest +``` + ## Usage ```bash From ac6e8f0fb1d150246ec93eedd203c04f8b6fd5c6 Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" Date: Tue, 5 Oct 2021 18:04:37 +0200 Subject: [PATCH 5/8] capitalize Any --- CONFORMANCE_TESTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index 68cf90ce..b84f8244 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -163,7 +163,7 @@ Likewise, if you don't want to test some aspect of a `class: File` or `class: Di [According to the CWL standards](https://www.commonwl.org/v1.2/CommandLineTool.html#File), the format of the `location` field in `class: File` and `class: Directory` is implementation specific and we should not be testing them. -Please remember to use `location: any` for them. +Please remember to use `location: Any` for them. Currently, we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930) If you are editing those old tests, you may be interested in some special processing for `class: File` and `class: Directory` output objects: From acfc54210399148b4278de4d35c27193a50a30f6 Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" Date: Tue, 5 Oct 2021 18:06:51 +0200 Subject: [PATCH 6/8] document "id" field --- CONFORMANCE_TESTS.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index b84f8244..39d53a17 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -151,6 +151,10 @@ Example: [`- $import: tests/string-interpolation/test-index.yaml`](https://githu adds all the entries in [`tests/string-interpolation/test-index.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/tests/string-interpolation/test-index.yaml) as entries in the main conformance test file. +You may also see references to an `id` field. Don't add them for new tests, +[we are migrating off of the `id` field](common-workflow-language/cwltest#110) to +rely only on the `label`s. + ## Output matching In each test entry there is an `output` field that contains a mapping of the expected outputs names and their values. From 871961d3d3e556ce6dffc70ae5589040ac9c8c01 Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" Date: Tue, 5 Oct 2021 18:07:13 +0200 Subject: [PATCH 7/8] please don't write tests that use 'path' --- CONFORMANCE_TESTS.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index 39d53a17..d3a1f97d 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -174,6 +174,8 @@ If you are editing those old tests, you may be interested in some special proces any `location` value specified will succeed if there is either an exact match to the real output, or it matches the end of the real output. Additionally, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison. +Likewise, please do not test the `path` for `class: File` and `class: Directory`. + ## Writing a new conformance test To add a new conformance test: From 19cb68709a308438f79e93bff023607c91f82222 Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" <1330696+mr-c@users.noreply.github.com> Date: Tue, 5 Oct 2021 18:37:30 +0200 Subject: [PATCH 8/8] more clarifications --- CONFORMANCE_TESTS.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CONFORMANCE_TESTS.md b/CONFORMANCE_TESTS.md index d3a1f97d..31e54474 100644 --- a/CONFORMANCE_TESTS.md +++ b/CONFORMANCE_TESTS.md @@ -132,12 +132,16 @@ We will use this single entry to explain the format args: [cat, hello.txt] tags: [ required, command_line_tool ] ``` -- `doc`: A sentence that explain what is being tested. Will be printed at test execution time. Should be unique. +- `doc`: A unique, single-line sentence that explain what is being tested. + Will be printed at test execution time, so please don't make it too long! + Additional documentation can go as comments in the CWL document itself. - `label`: a short list of underscore (`_`) separated words that succinctly identifies and explains the test. - `tool` the path to the CWL document to run - `job`: the CWL input object in YAML/JSON format. If there are no inputs then use `tests/empty.json`. - `output` [the CWL output object expected.](#output-matching) - `tags`: a yaml list of tag names, see [the list of canonical tags below](#tags-for-conformance-tests). + Must include one or more of the following tags: `command_line_tool`, `expression_tool` or `workflow`. + If the test does not test any optional features, the tag `required` is required. Because `conformance_tests.yaml` is a `schema-salad` processed document, [`$import`](https://www.commonwl.org/v1.2/SchemaSalad.html#Import) can be used to organize the tests into separate files.