CWL tools with SchemaDefRequirement fail to run in workflows

[ThomasHickman]

test_workflow.cwl:

cwlVersion: v1.0
class: Workflow

inputs:
  - id: hello
    type: Any
outputs: []

steps:
  step:
    id: step
    run: ../common-workflow-language/v1.0/v1.0/schemadef-tool.cwl
    in:
      hello: hello
    out: []

test_inputs.yaml

hello:
  a: one
  b: two

$ cwltool --debug test_workflow.cwl test_inputs.yaml
/Users/th10/checkouts/cwltool/venv/bin/cwltool 1.0.20180626215339
Resolved 'test_workflow.cwl' to 'file:///Users/th10/checkouts/cwltool/test_workflow.cwl'
Validation failed at
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 343, in __init__
    raise SchemaParseException(fail_msg)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 772, in make_avsc_object
    def parse(json_string):
avro.schema.SchemaParseException: Could not make an Avro Schema object from HelloType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/process.py", line 535, in __init__
    schema.AvroSchemaFromJSONData(self.inputs_record_schema, self.names)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 744, in make_avsc_object
    elif type in VALID_TYPES:
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 665, in __init__
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 631, in make_field_objects
    fail_msg = 'Field name %s already in use.' % new_field.name
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/avro/schema.py", line 346, in __init__
    self.type = type_schema
avro.schema.SchemaParseException: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 509, in __init__
    loadingContext.prov_obj))
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 657, in __init__
    super(WorkflowStep, self).__init__(toolpath_object, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/process.py", line 535, in __init__
    schema.AvroSchemaFromJSONData(self.inputs_record_schema, self.names)
  File "/Users/th10/checkouts/cwltool/venv/lib/python3.6/site-packages/schema_salad/sourceline.py", line 157, in __exit__
    raise self.makeError(six.text_type(exc_value))
schema_salad.validate.ValidationException: test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
Tool definition failed validation:
test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.
Traceback (most recent call last):
  File "/Users/th10/checkouts/cwltool/cwltool/main.py", line 504, in main
    metadata, uri, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/load_tool.py", line 342, in make_tool
    tool = loadingContext.construct_tool_object(processobj, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 51, in default_make_tool
    return Workflow(toolpath_object, loadingContext)
  File "/Users/th10/checkouts/cwltool/cwltool/workflow.py", line 516, in __init__
    raise validate.ValidationException("\n".join(str(v) for v in validation_errors))
schema_salad.validate.ValidationException: test_workflow.cwl:11:5: Type property "HelloType" not a valid Avro schema: Could not make an Avro Schema object from HelloType.

$ cwltool --version
/Users/th10/checkouts/cwltool/venv/bin/cwltool 1.0.20180626215339

---

This is due to the type of hello in the tool being populated to the inputs of the workflow internally, while the SchemaDefRequirement isn't. A simple fix would be to propagate this requirement up into the workflow

[mr-c]

Hello @ThomasHickman

I agree that we should deliver a better error message and also allow this example to work.

A workaround is to import and use the type at the Workflow level; which is better as it removes the type: Any ambiguity.

Here is an example: http://www.commonwl.org/user_guide/19-custom-types/