You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: CONFORMANCE_TESTS.md
+12-11Lines changed: 12 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,7 +23,7 @@ The CWL implementation to be tested.
23
23
24
24
`--tags`
25
25
26
-
A comman separated list of [tags](#tags-for-conformance-tests); only tests with these tags will be tested.
26
+
A comma separated list of [tags](#tags-for-conformance-tests); only tests with these tags will be tested.
27
27
`--tags shell_command` will run all tests with `shell_command` in their `tags` list.
28
28
29
29
`-n{test_range}`
@@ -93,15 +93,16 @@ system some needed GNU-like tools like `greadlink`.
93
93
94
94
## Format of the conformance test file
95
95
96
-
Conformance tests consist of input CWL descriptions + a input CWL object and the expected outputs (or `should_fail: true` if the test is deliberately broken)
96
+
Conformance tests consist of input CWL descriptions plus an input CWL object
97
+
and the expected outputs (or `should_fail: true` if the test is deliberately broken)
97
98
98
99
They are stored in [`conformance_tests.yaml`](https://github.com/common-workflow-language/cwl-v1.2/blob/main/conformance_tests.yaml)
99
100
(or a file `$import`ed into that one)
100
101
101
-
You can examine [the formal schema of this file](https://github.com/common-workflow-language/cwltest/blob/main/cwltest/cwltest-schema.yml)
102
+
You can examine [the formal schema of this file](https://github.com/common-workflow-language/cwltest/blob/main/cwltest/cwltest-schema.yml),
102
103
or just continue reading here for an explanation.
103
104
104
-
The conformante test file is a YAML document: a list of key-value pairs.
105
+
The conformance test file is a YAML document: a list of key-value pairs.
105
106
106
107
We will use this single entry to explain the format
107
108
```yaml
@@ -114,15 +115,15 @@ We will use this single entry to explain the format
114
115
tags: [ required, command_line_tool ]
115
116
```
116
117
- `doc`: A sentence that explain what is being tested. Will be printed at test execution time. Should be unique.
117
-
- `label`: a short list of underscore (`_`) seperated words that succintly identifies and explains the test.
118
+
- `label`: a short list of underscore (`_`) separated words that succinctly identifies and explains the test.
118
119
- `tool`the path to the CWL description to run
119
120
- `job`: the CWL input object in YAML/JSON format. If there are no inputs then use `tests/empty.json`.
- `tags`: a yaml list of tag names, see [the list of canonical tags below](#tags-for-conformance-tests).
122
123
123
-
Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into seperate files.
124
+
Because this is a `schema-salad` processed document, `$import` can be used to organize the tests into separate files.
124
125
125
-
Currently the main file is too big (over 3400 lines); we are slowly re-organizing it.
126
+
Currently, the main file is too big (over 3400 lines); we are slowly re-organizing it.
126
127
127
128
Eventually it would be good to organize the tests so that the test for each optional feature and other logical groups of tests are in their own separate file;
128
129
with the supporting CWL documents and their inputs in separate sub-folders of `tests` as well.
@@ -135,20 +136,20 @@ as entries in the main conformance test file.
135
136
136
137
In each test entry there is an `output` field that contains a mapping of the expected outputs names and their values.
137
138
138
-
If a particular value could vary and it doesn't matter to the proper functioning of the test then it can be represented by the special token `Any`.
139
+
If a particular value could vary and it doesn't matter to the proper functioning of the test, then it can be represented by the special token `Any`.
139
140
140
141
At any level, if there is an extra field, then that will be considered an error.
141
142
An exception to this is `class: File` and `class: Directory` objects, the `cwl-runner` under test can add additional fields here.
142
-
Likewise if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it.
143
+
Likewise, if you don't want to test some aspect of a `class: File` or `class: Directory` object (like `nameext`) you can just omit it.
143
144
144
145
[According to the CWL standards](https://www.commonwl.org/v1.2/CommandLineTool.html#File), the format of the `location` field in
145
146
`class: File` and `class: Directory` is implementation specific and we should not be testing them.
146
147
Please remember to use `location: any` for them.
147
148
148
-
Currently we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930)
149
+
Currently, we do [test the contents of the location field in some older tests, but we should stop](https://github.com/common-workflow-language/common-workflow-language/issues/930)
149
150
If you are editing those old tests, you may be interested in some special processing for `class: File` and `class: Directory` output objects:
150
151
any `location` value specified will succeed if there is either an exact match to the real output, or it matches the end of the real output.
151
-
Additionall, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison.
152
+
Additionally, for `class: Directory` the location reported by the actual execution will have any trailing forward slash (`/`) trimmed off before comparison.
0 commit comments