Skip to content

Commit 4ecf612

Browse files
authored
Update CONFORMANCE_TESTS.md
1 parent 04b9157 commit 4ecf612

File tree

1 file changed

+12
-11
lines changed

1 file changed

+12
-11
lines changed

CONFORMANCE_TESTS.md

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ The CWL implementation to be tested.
2323

2424
`--tags`
2525

26-
A comman separated list of [tags](#tags-for-conformance-tests); only tests with these tags will be tested.
26+
A comma separated list of [tags](#tags-for-conformance-tests); only tests with these tags will be tested.
2727
`--tags shell_command` will run all tests with `shell_command` in their `tags` list.
2828

2929
`-n{test_range}`
@@ -93,15 +93,16 @@ system some needed GNU-like tools like `greadlink`.
9393

9494
## Format of the conformance test file
9595

96-
Conformance tests consist of input CWL descriptions + a input CWL object and the expected outputs (or `should_fail: true` if the test is deliberately broken)
96+
Conformance tests consist of input CWL descriptions plus an input CWL object
97+
and the expected outputs (or `should_fail: true` if the test is deliberately broken)
9798

9899
They are stored in [`conformance_tests.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/conformance_tests.yaml)
99100
(or a file `$import`ed into that one)
100101

101-
You can examine [the formal schema of this file](https://github.com/common-workflow-language/cwltest/blob/main/cwltest/cwltest-schema.yml)
102+
You can examine [the formal schema of this file](https://github.com/common-workflow-language/cwltest/blob/main/cwltest/cwltest-schema.yml),
102103
or just continue reading here for an explanation.
103104

104-
The conformante test file is a YAML document: a list of key-value pairs.
105+
The conformance test file is a YAML document: a list of key-value pairs.
105106

106107
We will use this single entry to explain the format
107108
``` yaml
@@ -114,15 +115,15 @@ We will use this single entry to explain the format
114115
tags: [ required, command_line_tool ]
115116
```
116117
- `doc`: A sentence that explain what is being tested. Will be printed at test execution time. Should be unique.
117-
- `label`: a short list of underscore (`_`) seperated words that succintly identifies and explains the test.
118+
- `label`: a short list of underscore (`_`) separated words that succinctly identifies and explains the test.
118119
- `tool` the path to the CWL description to run
119120
- `job`: the CWL input object in YAML/JSON format. If there are no inputs then use `tests/empty.json`.
120121
- `output` [the CWL output object expected.](#output-matching)
121122
- `tags`: a yaml list of tag names, see [the list of canonical tags below](#tags-for-conformance-tests).
122123

123-
Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into seperate files.
124+
Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into separate files.
124125

125-
Currently the main file is too big (over 3400 lines); we are slowly re-organizing it.
126+
Currently, the main file is too big (over 3400 lines); we are slowly re-organizing it.
126127

127128
Eventually it would be good to organize the tests so that the test for each optional feature and other logical groups of tests are in their own separate file;
128129
with the supporting CWL documents and their inputs in separate sub-folders of `tests` as well.
@@ -135,20 +136,20 @@ as entries in the main conformance test file.
135136

136137
In each test entry there is an `output` field that contains a mapping of the expected outputs names and their values.
137138

138-
If a particular value could vary and it doesn't matter to the proper functioning of the test then it can be represented by the special token `Any`.
139+
If a particular value could vary and it doesn't matter to the proper functioning of the test, then it can be represented by the special token `Any`.
139140

140141
At any level, if there is an extra field, then that will be considered an error.
141142
An exception to this is `class: File` and `class: Directory` objects, the `cwl-runner` under test can add additional fields here.
142-
Likewise if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it.
143+
Likewise, if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it.
143144

144145
[According to the CWL standards](https://www.commonwl.org/v1.2/CommandLineTool.html#File), the format of the `location` field in
145146
`class: File` and `class: Directory` is implementation specific and we should not be testing them.
146147
Please remember to use `location: any` for them.
147148

148-
Currently we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930)
149+
Currently, we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930)
149150
If you are editing those old tests, you may be interested in some special processing for `class: File` and `class: Directory` output objects:
150151
any `location` value specified will succeed if there is either an exact match to the real output, or it matches the end of the real output.
151-
Additionall, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison.
152+
Additionally, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison.
152153

153154
## Writing a new conformance test
154155

0 commit comments

Comments
 (0)