RunLambdaTask with outputPath not working

[eikeon]

outputPath when used with RunLambdaTask is not working as documented. In the step functions overview under Lambda example

"The following snippet invokes a Lambda and sets the task output to only include the Lambda function response."

new sfn.Task(this, 'Invoke and set function response as task output', {
  task: new tasks.RunLambdaTask(myLambda, {
    payload: sfn.TaskInput.fromDataAt('$'),
  }),
  outputPath: '$.Payload',
});

Reproduction Steps

Starting from a working tasks.InvokeFunction:

<!--
minimal amount of code that causes the bug (if possible) or a reference:
-->

         const getObject = new sfn.Task(this, "Get Object", {
-            task: new tasks.RunLambdaTask(getObjectLambda, {}),
-            outputPath: "$.Payload",
+            task: new tasks.InvokeFunction(getObjectLambda, {}),
             resultPath: "$.GetObjectResult",
         });

Also tried outputPath: "$.output.Payload" with same result.

Error Log

TaskSucceeded with:

{
  "resourceType": "lambda",
  "resource": "invoke",
  "output": {
    "ExecutedVersion": "$LATEST",
    "Payload": {
      "Done": true,
      "Next": null,
      "BitsFetched": 0
    },
    "SdkHttpMetadata": {
      "HttpHeaders": {
        "Connection": "keep-alive",
        "Content-Length": "41",
        "Content-Type": "application/json",
        "Date": "Thu, 30 Apr 2020 00:36:36 GMT",
        "X-Amz-Executed-Version": "$LATEST",
        "x-amzn-Remapped-Content-Length": "0",
        "x-amzn-RequestId": "f1e1c5df-4740-4389-8230-2c495c5cd829",
        "X-Amzn-Trace-Id": "root=1-5eaa1d94-16cf39fcb6ea5d8bb11bd635;sampled=0"
      },
      "HttpStatusCode": 200
    },
    "SdkResponseMetadata": {
      "RequestId": "f1e1c5df-4740-4389-8230-2c495c5cd829"
    },
    "StatusCode": 200
  }
}

followed by `ExecutionFailed` with:

```json
{
  "error": "States.Runtime",
  "cause": "An error occurred while executing the state 'Get Object' (entered at the event id #2). Invalid path '$.Payload' : No results for path: $['Payload']"
}

{

### Environment

  - **CLI Version      :** 1.36.1
  - **Framework Version:** 1.36.1
  - **OS               :** macOS Catalina 10.15.5 Beta (19F62f)
  - **Language         :** Typescript

### Other

<!-- e.g. detailed explanation, stacktraces, related issues, suggestions on how to fix, links for us to have context, eg. associated pull-request, stackoverflow, gitter, etc -->




--- 

This is :bug: Bug Report

[shivlaks]

@eikeon I think the TaskSucceeded entry you're sharing there is from the Execution Event History, which uses the output field.

When you're setting resultPath to be $.GetObjectResult, you're putting the result of your Lambda into a node called GetObjectResult which is appended to the state input.

If you want the Lambda's result, you'd have to set your outputPath to be $.GetObjectResult.Payload

Let me know how it goes!

[eikeon]

I should have stared at https://docs.aws.amazon.com/step-functions/latest/dg/concepts-input-output-filtering.html longer. Had it in my head that the $outputPath happens before not after the $resultPath in that diagram.

But I'm still not able to replicate the TaskOutput I get with InvokeFunction when using RunLambdaTask. Attempt 1)

--- a/lib/getobject-statemachine.ts
+++ b/lib/getobject-statemachine.ts
@@ -48,6 +48,7 @@ export class GetObjectStateMachine extends cdk.Construct {

        const getObject = new sfn.Task(this, "Get Object", {
            task: new tasks.InvokeFunction(getObjectLambda, {}),
+            outputPath: "$.GetObjectResult.Payload",
            resultPath: "$.GetObjectResult",
        });

results in LambdaFunctionSucceeded with output:

{
  "output": {
    "Done": true,
    "Next": null,
    "BitsFetched": 0
  }
}

followed by ExecutionFailed with:

{
  "error": "States.Runtime",
  "cause": "An error occurred while executing the state 'Get Object' (entered at the event id #2). Invalid path '$.GetObjectResult.Payload' : No results for path: $['GetObjectResult']['Payload']"
}

Attempt 2)

--- a/lib/getobject-statemachine.ts
+++ b/lib/getobject-statemachine.ts
@@ -48,6 +48,7 @@ export class GetObjectStateMachine extends cdk.Construct {
 
         const getObject = new sfn.Task(this, "Get Object", {
             task: new tasks.InvokeFunction(getObjectLambda, {}),
+            outputPath: "$.GetObjectResult",
             resultPath: "$.GetObjectResult",
         });

followed by TaskStateExited with:

{
  "name": "Get Object",
  "output": {
    "Done": true,
    "Next": null,
    "BitsFetched": 0
  }
}

where as with the InvokeFunction the task state exited with:

{
  "name": "Get Object",
  "output": {
    "ParentContentID": "AQERFLelzPqbNGewZ3/CDNW/o/ZzuA4U",
    "Hash": "ERTg6jh13sV9G3ILH6qyChVfn4i2Bg==",
    "URL": "https://archive.org/download/snowyforluck00good/snowyforluck00good_metasource.xml",
    "Size": 0,
    "PartInfo": null,
    "GetObjectResult": {
      "Done": true,
      "Next": null,
      "BitsFetched": 0
    }
  }
}

[shivlaks]

@eikeon I see what you mean now. This is a limitation that's imposed by using RunLambda (or any service integration really since they all have the metadata).

There is a workaround, I'm giving it a try and will share if it works for me. Stay tuned!

[shivlaks]

@eikeon so, you cannot merge a part of the task output to the input, which is what you're going for when using resultPath.

The workaround from the Step Functions team if this behavior is still desired is to use a Pass state to merge the input and the part of the output you'd like.

The issue is also discussedhere

From the CDK side of things, here's a gist that shows it's usage of the workaround.

This applies to all service integrations that follow the service integration pattern (essentially all of them except the old InvokeFunction). I think we should at the very least update our README and documentation on paths to detail the limitation more clearly.

I'm not sure it's actionable from the CDK side since Step Functions doesn't allow it yet. I've also passed along the feedback to the team.

[eikeon]

@shivlaks Thank you. And agreed re: very least since InvokeFunction is being deprecated and others will likely be going down this path with the rake on the ground.

From the CDK side, where higher level constructs are being created, would it be possible to add an option to RunLambdaTask that would create the extra Pass state?

Alternatively, document how one can expect an additional layer of output and update the rest of their state machine to expect it. Going to find out what that looks like now ;)

Thanks again for your help! Hope my feedback has been helpful as well. Let me know is there's anything I can do to help. Else I should be set now.

[shivlaks]

@eikeon thanks for taking the time to provide this valuable feedback!

I'll circle back as to whether it would be a useful thing to try and generate the pass state when chaining Lambdas together or whether we would want to wait a bit before jumping into an implementation. My initial instinct is that It still feels a little clunky to encode a workaround into the API.

Let me know if you run into any other snags. At a minimum I'll start with documenting this since it applies to all service integrations, it'll likely be a FAQ

[eikeon]

Agreed re: a little clunky. Versus updating a few references elsewhere in the state machine to deal with the extra layer. I don't anticipate that's going to be a problem in my case.

[eikeon]

Yep; the addition of a few .Payloads elsewhere in my step functions and no need for workarounds. So maybe just a note along side the deprecated notice of InvokeFunction of what options lie ahead. If I would have started with RunLambdaFunction I would not have been trying to preserve that exact behavior so hard

[eikeon]

Doh, linked the above merge request accidentally to this one in a commit even though the title had the correct issue #

[shivlaks]

Doh, linked the above merge request accidentally to this one in a commit even though the title had the correct issue #

@eikeon no worries, it's not a big deal