From 7324b265506d8902359235ea192e816d451693ff Mon Sep 17 00:00:00 2001 From: "Bruno P. Kinoshita" Date: Tue, 6 Sep 2022 12:06:23 +1200 Subject: [PATCH 1/4] Highlight lines of interest in the command outputs --- src/topics/inputs.md | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index aba85daa..a5e1a31f 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -220,8 +220,11 @@ parameters together to describe these two conditions. ``` ```{code-block} console +:emphasize-lines: 6-7 $ cwl-runner record.cwl record-job1.yml -Workflow error, try again with --debug for more information: +INFO /home/kinow/Development/python/workspace/user_guide/venv/bin/cwl-runner 3.1.20220830195442 +INFO Resolved 'record.cwl' to 'file:///tmp/record.cwl' +ERROR Workflow error, try again with --debug for more information: Invalid job input record: record-job1.yml:1:1: the `dependent_parameters` field is not valid because missing required field `itemB` @@ -235,28 +238,30 @@ In the first example, you can't provide `itemA` without also providing `itemB`. :name: record-job2.yml ``` -```cwl -$ cwl-runner record.cwl record-job2.yml -record-job2.yml:6:3: invalid field `itemD`, expected one of: 'itemC' -[job record.cwl] /home/example$ echo \ +```{code-block} cwl +:emphasize-lines: 3, 9-10, 24 +INFO /home/kinow/Development/python/workspace/user_guide/venv/bin/cwl-runner 3.1.20220830195442 +INFO Resolved 'record.cwl' to 'file:///tmp/record.cwl' +record-job2.yml:6:3: Warning: invalid field `itemD`, expected one of: 'itemC' +INFO [job record.cwl] /tmp/r15et6qm$ echo \ -A \ one \ -B \ two \ -C \ - three > /home/example/output.txt -[job record.cwl] completed success + three > /tmp/r15et6qm/output.txt +INFO [job record.cwl] completed success { "example_out": { - "location": "file:///home/example/11-records/output.txt", + "location": "file:///tmp/output.txt", "basename": "output.txt", "class": "File", "checksum": "sha1$329fe3b598fed0dfd40f511522eaf386edb2d077", "size": 23, - "path": "/home/example/output.txt" + "path": "/tmp/output.txt" } } -Final process status is success +INFO Final process status is success $ cat output.txt -A one -B two -C three ``` @@ -271,26 +276,29 @@ is added to the command line and `itemD` is ignored. ``` ```{code-block} console +:emphasize-lines: 9-10, 24 $ cwl-runner record.cwl record-job3.yml -[job record.cwl] /home/example$ echo \ +INFO /home/kinow/Development/python/workspace/user_guide/venv/bin/cwl-runner 3.1.20220830195442 +INFO Resolved 'record.cwl' to 'file:///tmp/record.cwl' +INFO [job record.cwl] /tmp/6qz6choj$ echo \ -A \ one \ -B \ two \ -D \ - four > /home/example/output.txt -[job record.cwl] completed success + four > /tmp/6qz6choj/output.txt +INFO [job record.cwl] completed success { "example_out": { - "location": "file:///home/example/output.txt", + "location": "file:///tmp/output.txt", "basename": "output.txt", "class": "File", "checksum": "sha1$77f572b28e441240a5e30eb14f1d300bcc13a3b4", "size": 22, - "path": "/home/example/output.txt" + "path": "/tmp/output.txt" } } -Final process status is success +INFO Final process status is success $ cat output.txt -A one -B two -D four ``` From fc1c19fa1a66c909605d9ff538876f3431d87d20 Mon Sep 17 00:00:00 2001 From: "Bruno P. Kinoshita" Date: Tue, 6 Sep 2022 12:14:47 +1200 Subject: [PATCH 2/4] Rename "Advanced Inputs" to "Inclusive and Exclusive Inputs" --- src/topics/inputs.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index a5e1a31f..8b203b6e 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -200,7 +200,7 @@ This will be demonstrated in the next lesson and is discussed in more detail in the [YAML Guide](yaml-guide.md#arrays). You can specify arrays of arrays, arrays of records, and other complex types. -## Advanced Inputs +## Inclusive and Exclusive Inputs Sometimes an underlying tool has several arguments that must be provided together (they are dependent) or several arguments that cannot be provided @@ -306,10 +306,11 @@ $ cat output.txt In the third example, only `itemD` is provided, so it appears on the command line. +```{note}``` + % TODO % % - Explain its fields, such as default, valueFrom, etc. - https://github.com/common-workflow-language/common-workflow-language/issues/359 -% - Exclusive parameters https://github.com/common-workflow-language/user_guide/issues/162 % - Optional Inputs https://github.com/common-workflow-language/user_guide/issues/44 % - Several ways of defining inputs/arguments to tools and workflows - https://github.com/common-workflow-language/user_guide/issues/33 % - Using an input output in another input - https://github.com/common-workflow-language/user_guide/issues/90 From 04dd9447d3d96f9a0b55e04d486d846f140040d6 Mon Sep 17 00:00:00 2001 From: "Bruno P. Kinoshita" Date: Tue, 6 Sep 2022 13:22:50 +1200 Subject: [PATCH 3/4] Add text about the exclusive input parameters and expressions --- src/_includes/cwl/conformance-test.yml | 12 ++-- .../exclusive-parameter-expressions.cwl | 25 +++++++ .../{11-records => inputs}/record-job1.yml | 0 .../{11-records => inputs}/record-job2.yml | 0 .../{11-records => inputs}/record-job3.yml | 0 .../cwl/{11-records => inputs}/record.cwl | 0 src/topics/inputs.md | 70 +++++++++++++++++-- 7 files changed, 96 insertions(+), 11 deletions(-) create mode 100644 src/_includes/cwl/inputs/exclusive-parameter-expressions.cwl rename src/_includes/cwl/{11-records => inputs}/record-job1.yml (100%) rename src/_includes/cwl/{11-records => inputs}/record-job2.yml (100%) rename src/_includes/cwl/{11-records => inputs}/record-job3.yml (100%) rename src/_includes/cwl/{11-records => inputs}/record.cwl (100%) diff --git a/src/_includes/cwl/conformance-test.yml b/src/_includes/cwl/conformance-test.yml index acdab96d..dbb4e92a 100644 --- a/src/_includes/cwl/conformance-test.yml +++ b/src/_includes/cwl/conformance-test.yml @@ -103,13 +103,13 @@ # Section 11 depends on side-effects - doc: Test for section 11 (1st example) - job: 11-records/record-job1.yml - tool: 11-records/record.cwl + job: inputs/record-job1.yml + tool: inputs/record.cwl should_fail: true - doc: Test for section 11 (2nd example) - job: 11-records/record-job2.yml - tool: 11-records/record.cwl + job: inputs/record-job2.yml + tool: inputs/record.cwl output: example_out: class: File @@ -119,8 +119,8 @@ size: 23 - doc: Test for section 11 (3rd example) - job: 11-records/record-job3.yml - tool: 11-records/record.cwl + job: inputs/record-job3.yml + tool: inputs/record.cwl output: example_out: class: File diff --git a/src/_includes/cwl/inputs/exclusive-parameter-expressions.cwl b/src/_includes/cwl/inputs/exclusive-parameter-expressions.cwl new file mode 100644 index 00000000..6096de82 --- /dev/null +++ b/src/_includes/cwl/inputs/exclusive-parameter-expressions.cwl @@ -0,0 +1,25 @@ +cwlVersion: v1.2 +class: CommandLineTool + +inputs: + output_format: + type: + - 'null' + - name: output_format + type: enum + symbols: + - auto + - fasta + - fastq + - fasta.gz + - fastq.gz + inputBinding: + position: 0 + prefix: '--format' +outputs: + text_output: + type: string + outputBinding: + outputEval: $(inputs.output_format) + +baseCommand: 'true' diff --git a/src/_includes/cwl/11-records/record-job1.yml b/src/_includes/cwl/inputs/record-job1.yml similarity index 100% rename from src/_includes/cwl/11-records/record-job1.yml rename to src/_includes/cwl/inputs/record-job1.yml diff --git a/src/_includes/cwl/11-records/record-job2.yml b/src/_includes/cwl/inputs/record-job2.yml similarity index 100% rename from src/_includes/cwl/11-records/record-job2.yml rename to src/_includes/cwl/inputs/record-job2.yml diff --git a/src/_includes/cwl/11-records/record-job3.yml b/src/_includes/cwl/inputs/record-job3.yml similarity index 100% rename from src/_includes/cwl/11-records/record-job3.yml rename to src/_includes/cwl/inputs/record-job3.yml diff --git a/src/_includes/cwl/11-records/record.cwl b/src/_includes/cwl/inputs/record.cwl similarity index 100% rename from src/_includes/cwl/11-records/record.cwl rename to src/_includes/cwl/inputs/record.cwl diff --git a/src/topics/inputs.md b/src/topics/inputs.md index 8b203b6e..734d3960 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -207,13 +207,13 @@ together (they are dependent) or several arguments that cannot be provided together (they are exclusive). You can use records and type unions to group parameters together to describe these two conditions. -```{literalinclude} /_includes/cwl/11-records/record.cwl +```{literalinclude} /_includes/cwl/inputs/record.cwl :language: cwl :caption: "`record.cwl`" :name: record.cwl ``` -```{literalinclude} /_includes/cwl/11-records/record-job1.yml +```{literalinclude} /_includes/cwl/inputs/record-job1.yml :language: yaml :caption: "`record-job1.yml`" :name: record-job1.yml @@ -232,7 +232,7 @@ record-job1.yml:1:1: the `dependent_parameters` field is not valid because In the first example, you can't provide `itemA` without also providing `itemB`. -```{literalinclude} /_includes/cwl/11-records/record-job2.yml +```{literalinclude} /_includes/cwl/inputs/record-job2.yml :language: yaml :caption: "`record-job2.yml`" :name: record-job2.yml @@ -269,7 +269,7 @@ $ cat output.txt In the second example, `itemC` and `itemD` are exclusive, so only `itemC` is added to the command line and `itemD` is ignored. -```{literalinclude} /_includes/cwl/11-records/record-job3.yml +```{literalinclude} /_includes/cwl/inputs/record-job3.yml :language: yaml :caption: "`record-job3.yml`" :name: record-job3.yml @@ -306,7 +306,67 @@ $ cat output.txt In the third example, only `itemD` is provided, so it appears on the command line. -```{note}``` +### Exclusive Input Parameters with Expressions + +If you use exclusive input parameters combined with expressions, you need to be +aware that the `inputs` JavaScript object will contain one of the exclusive +input values. This means that you might need to use an **or** boolean operator +to check which values are present. + +Let's use an example that contains an exclusive `output_format` input parameter +that accepts `null` (i.e. no value provided), or any value from an enum. + +```{literalinclude} /_includes/cwl/inputs/exclusive-parameter-expressions.cwl +:language: cwl +:caption: "`exclusive-parameter-expressions.cwl`" +:name: exclusive-parameter-expressions.cwl +:emphasize-lines: 7, 21-23 +``` + +Note how the JavaScript expression uses the value of the exclusive input parameter +without taking into consideration a `null` value. If you provide a valid value, +such as “fasta” (one of the values of the enum), your command should execute +successfully: + +```{code-block} console +$ cwltool exclusive-parameter-expressions.cwl --output_format fasta +INFO /home/kinow/Development/python/workspace/user_guide/venv/bin/cwltool 3.1.20220830195442 +INFO Resolved 'exclusive-parameter-expressions.cwl' to 'file:///tmp/exclusive-parameter-expressions.cwl' +INFO [job exclusive-parameter-expressions.cwl] /tmp/j02lo8ky$ true \ + --format \ + fasta +INFO [job exclusive-parameter-expressions.cwl] completed success +{ + "text_output": "fasta" +} +INFO Final process status is success +``` + +However, if you do not provide any input value, then `output_format` will be +evaluated to a `null` value, which does not match the expected type for the +output field (a `string`), resulting in failure when running your workflow. + +```{code-block} console +:emphasize-lines: 6-10 +$ cwltool exclusive-parameter-expressions.cwl +INFO /home/kinow/Development/python/workspace/user_guide/venv/bin/cwltool 3.1.20220830195442 +INFO Resolved 'exclusive-parameter-expressions.cwl' to 'file:///tmp/exclusive-parameter-expressions.cwl' +INFO [job exclusive-parameter-expressions.cwl] /tmp/jmq3nqap$ true +ERROR [job exclusive-parameter-expressions.cwl] Job error: +Error validating output record. the `text_output` field is not valid because + the value is not string + in { + "text_output": null +} +WARNING [job exclusive-parameter-expressions.cwl] completed permanentFail +{} +WARNING Final process status is permanentFail +``` + +To correct it, you must remember to use an or operator in your JavaScript expression +when using exclusive parameters, or any parameter that allows `null`. For example, +the expression could be changed to `$(inputs.output_format || 'auto')`, to have +a default value if none was provided in the command line or job input file. % TODO % From ddd42064556399298fae74a1a78294e0413c89c7 Mon Sep 17 00:00:00 2001 From: "Michael R. Crusoe" Date: Mon, 10 Oct 2022 11:52:15 +0200 Subject: [PATCH 4/4] explain why itemC was chosen when both are provided --- src/topics/inputs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/topics/inputs.md b/src/topics/inputs.md index b944de65..b0b73eb7 100644 --- a/src/topics/inputs.md +++ b/src/topics/inputs.md @@ -230,8 +230,8 @@ $ cat output.txt -A one -B two -C three ``` -In the second example, `itemC` and `itemD` are exclusive, so only `itemC` -is added to the command line and `itemD` is ignored. +In the second example, `itemC` and `itemD` are exclusive, so only the first +matching item (`itemC`) is added to the command line and remaining item (`itemD`) is ignored. ```{literalinclude} /_includes/cwl/inputs/record-job3.yml :language: yaml