Runtime timeout errors

[markosski]

I have lambda function deployed and experiencing some runtime errors. This happens pretty often on first invocation. Once it happens it's hard to replicate with consecutive invocations without waiting for a minute.

{
  "errorType": "Runtime.ExitError",
  "errorMessage": "RequestId: d922d798-3cc4-4a04-a492-acf5a3d69dc5 Error: Runtime exited with error: exit status 1"
}

START RequestId: d922d798-3cc4-4a04-a492-acf5a3d69dc5 Version: $LATEST
bootstrap: HttpExceptionRequest Request {
  host                 = "127.0.0.1"
  port                 = 9001
  secure               = False
  requestHeaders       = [("User-Agent","haskell wreq-0.5.2.1")]
  path                 = "/2018-06-01/runtime/invocation/next"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ResponseTimeout
END RequestId: d922d798-3cc4-4a04-a492-acf5a3d69dc5
REPORT RequestId: d922d798-3cc4-4a04-a492-acf5a3d69dc5	Duration: 51.78 ms	Billed Duration: 100 ms 	Memory Size: 512 MB	Max Memory Used: 57 MB	
RequestId: d922d798-3cc4-4a04-a492-acf5a3d69dc5 Error: Runtime exited with error: exit status 1
Runtime.ExitError

EDIT
aws-lambda-haskell-runtime-1.0.9
Lambda layer v2

EDIT
Also tested clean installation as per instructions in repo and experiencing the same issue.

[NickSeagull]

Hey, thanks for reporting this! What version are you using @markosski ?

[markosski]

@NickSeagull Thanks Nick for quick response. Updated issue with version.

[NickSeagull]

Hey @markosski #26 should have fixed this issue, can you try using the following?

aws-lambda-haskell-runtime-1.0.10
Lambda layer v4

[markosski]

@NickSeagull Thanks. I updated lambda with new binary and changed layer version, unfortunately still experiencing the same issue. Were you be able to replicate this issue? Let me know if I can help. Is there a way to test this locally without building and uploading? Also how did you build the layer, what's in it?

[NickSeagull]

Ah, what a bummer. I had issues with this yesterday, but after bumping my lambdas to the latest version, the problem went away.

Also, probably setting 512MB as the lambdas memory generally helps. The runtime won't use the memory, but it looks like it helps (see #18 ). As far as I can tell, I'm not getting the timeouts with 512MB.

Regarding the layer, its just the result of building this repo using stack build, zipping the bootstrap executable and uploading it. You can try doing so if you want. Actually we are doing so in CircleCI (see lines 21-23 and 82-88)

Unfortunately, I couldnt come with a better way of testing this, as the invocation/next endpoint is provided by AWS internally, and I suspect that this issue is related to AWS lambda container freezing.

[markosski]

@NickSeagull Thanks. I originally had it set to 512, also tried with 1024, same issue.
Pattern of this issue is interesting. Looks like it works fine on cold start and invocations after that made within couple seconds apart. After waiting for about 30 seconds before invoking again it will fail but next invocation made right after will work.

I can replicate this issue every time if I wait about 30 seconds before each invocation.

I don't understand how it is timing out on runtime/invocation/next when function responds immediately (~100ms).

[NickSeagull]

I'm trying some tests, and apparently, the function gets executed properly (I'm performing a putItem to a DynamoDB database and the new item appears). But the runtime doesn't notify AWS that it finished...

[fuzzy-id]

I ran into this too. I'm on version 1.0.10, increasing memory didn't hep.

Any way I can help to debug this?

[glammers1]

Hi @markosski @fuzzy-id a new version of the Lambda layer has deployed. Please try with the Lambda layer v5.

[markosski]

Hey @glammers1 I didn't test this exhaustively yet but it seems I can't replicate the issue I described earlier anymore. Could you explain what fixed the issue in the layer? Great work!

[glammers1]

Hi! We updated the version tag of the docker image that builds the bootstrap from stack-build:12.13 to stack-build:13.0 (see #27). The reason behind this was that we did some adjustments in order to isolate the problem (basically the user project acted like the runtime itself) and we was getting another kind of error related to a call to a C function (something related to file system). The exception was c_poll: permission denied. Therefore, we tried with another stack-build as a first attempt and everything works as expected. We are thinking about having some test in the project in order to detect that kind of errors among version updates for example.

Thanks a lot!

[fuzzy-id]

Wasn't able to test my original code again. But I developed a couple of other Lambdas since then and haven't experienced this issue anymore.

[NickSeagull]

Awesome, I guess we can close this now 😊